Simplify and backport the local environment

[pento]

The Docker-based local environment (#45165) is based on the Gutenberg local environment, which was an interesting experiment in automatically configuring a development environment, but it ultimately had several shortcomings:

• It only worked on MacOS, and kind of worked on Linux.
• It was able to recover from some issues, but was fairly opaque when it failed.
• It was extremely opinionated about how the environment should run.

For Core itself, I think we need something a lot simpler, for a few reasons:

• It should work on every platform, not just those with Bash support.
• It should be flexible for folks to be able to configure their own development environment.
• It should be simple to backport to old branches, so we can more easily set up test environments when backporting patches to old branches.

I think there's still a need for an opinionated, automated tool (ie, ​TestPress), but I don't think Core is the right place for it.

[pento]

47767.diff​ is a first pass at this. It requires Docker to be installed and running.

To get a local environment up and running, run this commands:

npm run env:start
npm run env:install

This will get a local environment running from src up and running on ​http://localhost:8889.

To change the port, you can set the LOCAL_PORT environment variable. To run from the build directory instead, you can set the LOCAL_DIR variable.

npm run env:stop
export LOCAL_PORT=8888
export LOCAL_DIR=build
npm run env:start
npm run env:install

You can also see in the docker-compose.yml and docker-compose.scripts.yml files that it depends on Docker images that are currently coming from my repo. These should be moved to a WordPress organisation on Docker Hub.

[pento]

In 47767.2.diff​:

• Copy the generated wp-config.php file to the root directory, so it can be used by src and build installs.
• Tidy up the docker-compose files.

[pento]

In 47767.3.diff​

• Fix npm run env:install not setting the correct port for the site URL on Windows.

Side note, if anyone is testing on Windows with Docker Toolbox, you need to run this command to be able to access the site:

"%VBOX_MSI_INSTALL_PATH%\VBoxManage" controlvm "VM name" natpf1 "tcp-port8889,tcp,127.0.0.1,8889,,8889"

If you change the LOCAL_PORT, you'll need to run this again with the new port number. No need to do this if you're using Docker for Windows, that will automatically do the port forwarding.

[pento]

@youknowriad, @gziolo, @herregroen, @mcsf: Does this fit in with the direction you'd imagined for running e2e tests?

[pento]

Is modern WordPress too bloated for you? Do you like your blogging platform fast and furious?

47767.4.diff​ is a proof of concept patch for the 3.7 branch, backporting the Docker config with some minor tweaks.

[noisysocks]

I love this! A useful development environment that works right out of the box is really going to help onboard new developers, especially Gutenberg contributors that might want to have a try at contributing upstream.

Everything I'd use for day-to-day development works great:

$ npm install
$ npm env:start
$ npm env:install
$ npx grunt watch --dev
$ npm run env:cli shell
wp> get_option( 'admin_email' )
=> string(13) "test@test.com"

I also checked that the E2E tests still run fine.

I have not tested this on Windows.

Some loose thoughts:

1. Documentation in the handbook for all of this functionality will go a long way.
2. It would be nice if getting unit tests set up weren't left as an exercise for the reader. Perhaps this is best addressed with some nice documentation, as above.
3. It feels awkward to me that npm run dev is an alias for grunt build --dev instead of grunt watch --dev, as I'm coming from Gutenberg where npm run dev watches for changes.
4. I'd like if we didn't need to define -next commands in the package.json, so that running npm run would display the commands that can be run and nothing else. I assume this is a dotenv-cli limitation, though?

[pento]

1. Documentation in the handbook for all of this functionality will go a long way.

Agreed, we can realistically replace all of the ​Handbook section on setting up a local environment.

2. It would be nice if getting unit tests set up weren't left as an exercise for the reader. Perhaps this is best addressed with some nice documentation, as above.

Yah, the unit test stuff is much more skeleton at the moment. It needs fleshing out, but it should be as simple as running npm run test:php.

3. It feels awkward to me that npm run dev is an alias for grunt build --dev instead of grunt watch --dev, as I'm coming from Gutenberg where npm run dev watches for changes.

Fair point. I don't think npm run dev is currently used anywhere, so we should be able to redefine it.

4. I'd like if we didn't need to define -next commands in the package.json, so that running npm run would display the commands that can be run and nothing else. I assume this is a dotenv-cli limitation, though?

Yah, that's a problem with dotenv-cli. It defines the environment variables in the child process that spawns. I'm happy to alter the pattern if anyone has better ideas, but I don't think we can avoid the concept of calling the -next scripts.

---

While I'm writing, here's a handful of issues I've run into while working on this that still need addressing:

• Docker caches the images it downloads pretty heavily, it won't check for new versions unless forced. We need an integrated-but-unobtrusive way to check for and download new images.
• Downgrading to WordPress 3.7/3.8 silently fails. They require MySQL 5.6, which uses a different data file format to MySQL 5.7. As the MySQL image uses a persistent volume, it'll fail to load if the data files were created in the newer MySQL version.
• PHP version and MySQL version should be defined through environment variables.
• There are no official FPM builds of PHP 5.2 and 5.3. There are third party options, though. Same with MySQL 5.0 and 5.1.

[mcsf]

4. I'd like if we didn't need to define -next commands in the package.json, so that running npm run would display the commands that can be run and nothing else. I assume this is a dotenv-cli limitation, though?

Yah, that's a problem with dotenv-cli. It defines the environment variables in the child process that spawns. I'm happy to alter the pattern if anyone has better ideas, but I don't think we can avoid the concept of calling the -next scripts.

This one also stuck out for me. What exactly is the limitation? Or, better phrased, when would one call the -next counterpart directly? From what I can see dotenv gives precedence to any preexisting environment variables:

$ cat my-cmd 
#!/bin/bash
echo "Hi, $NAME!"
$ cat .env
NAME="User"
$ npx dotenv ./my-cmd 
Hi, User!
$ NAME=Visitor npx dotenv ./my-cmd 
Hi, Visitor!

[pento]

You should never call the -next script directly, as the environment variable default values won't be loaded from the .env file.

Given an NPM script "test": "dotenv echo $FOO", a .env file with FOO=bar, and no FOO environment variable set, the following will happen:

• Run npm run test.
• NPM spawns a child process to execute dotenv echo $FOO, passing process.env (which currently doesn't have FOO defined).
• The child process replaces variables with their corresponding values, so the command to be run is now dotenv echo .
• dotenv runs, loads the .env file, and sets FOO=bar as an environment variable.
• dotenv spawns a child process to execute echo , passing process.env (which now includes FOO) to the child process.
• echo is run in the child process.

Given the same environment with a .env file with FOO=bar, no FOO environment variable set, but with two NPM scripts set, "test": "dotenv npm run test-next", "test-next": "echo $FOO", the following will happen:

• Run npm run test.
• NPM spawns a child process to execute dotenv npm run test-next, passing process.env (which currently doesn't have FOO defined).
• There are no variables to replace, so the command to be run is still dotenv npm run test-next.
• dotenv runs, loads the .env file, and sets FOO=bar as an environment variable.
• dotenv spawns a child process to execute npm run test-next, passing process.env (which now includes FOO) to the child process.
• NPM spawns a child process to execute echo $FOO, passing process.env (which still includes FOO) to the child process.
• The child process replaces variables with their corresponding values, so the command to be run is now echo bar.
• echo bar is run in the child process.

The problem that dotenv solves is allowing Core to set default values for config options, which a user can easily override, So, all of this this could be avoided if we had a different way to set default values for these environment variables. Here are some other options that I evaluated:

• Using Bash notation to specify a default value inline. For example, ${LOCAL_PORT-8889} would use $LOCAL_PORT if it was defined, or 8889 if it wasn't. There is no equivalent behaviour available when the shell is cmd.exe.
• Use NPM config settings. For example, adding "config": { "port": 8889 } to package.json would ensure the $npm_package_config_port variable was set to the default value of 8889 when running scripts. It can be overridden by running the command npm config set WordPress:port 7778 before running the NPM scripts. The downside of this is that this is set in your ~/.npmrc file, it isn't set on a per-project basis, so you can't have multiple WordPress installs running simultaneously. NPM supports a project .npmrc, which we could add to .gitignore to allow local overrides, if we weren't already using it.
• Write a custom package that combines the magic of dotvar, cross-var, and cross-env into something that works better. This is a valid option, but overkill for this ticket.

While dotenv has the downside of needing to have two scripts to do one thing, it's only a little bit of mess which can be cleaned up at a later date.

[mcsf]

Replying to pento:

Thanks for the detailed answer. My issue was the wrong assumption that the script split was an attempt to retain the possibility of invoking NPM scripts with custom environment variables, e.g. FOO=bar npm run test-next, hence my silly counterexample.

The problem that dotenv solves is allowing Core to set default values for config options, which a user can easily override, So, all of this this could be avoided if we had a different way to set default values for these environment variables. Here are some other options that I evaluated:

This is quite tricky. (Is it OK if I have a deep longing for Makefiles?)

Given the circumstances, I don't strongly oppose to the -next auxiliary scripts. I was going to suggest we use dotenv directly within Node script files. For example:

require('dotenv').config();
const url = `http://localhost:${process.env.LOCAL_PORT}`;

… but the reality is that many of the package-defined scripts are just inline arrangements of other scripts, so this would require adding a bunch of Node files just to run them. Which isn't wrong, just cumbersome.

For kicks, I played with the following:

$ cat if-def 
#!/usr/bin/env node

const variables = process.argv.slice(2);

for (const variable of variables) {
	if (process.env[variable] === undefined) {
		process.exit(1);
	}
}

process.exit(0);

  "scripts": {
    "hello": "./if-def NAME && echo \"Hi, $NAME\" || dotenv npm run hello"
  },

$ npm run hello

> npmtests@1.0.0 hello /private/tmp/npmtests
> ./if-def NAME && echo "Hi, $NAME" || dotenv npm run hello


> npmtests@1.0.0 hello /private/tmp/npmtests
> ./if-def NAME && echo "Hi, $NAME" || dotenv npm run hello

Hi, User

Is this horrible? Yes. Let's not do this ever. But it speaks to how NPM is ill-equipped for supporting this sort of customisable flow.

Moving past this, let's say that -next is fine for now. I'll review other pieces of the patch. :)

[pbearne]

Hi All

I am trying to test as "the dumb windows user"

I can't get to work
steps I have done

Download clean copy from SVN ​https://develop.svn.wordpress.org/trunk/

applied the patch (not sure it applied cleanly)

tried to run npm in the root and local-env folders

$ npm run env:start


> WordPress@5.3.0 env:start C:\Users\pbear\Documents\code\SVN-core-wp
> ./tools/local-env/start.sh

'.' is not recognized as an internal or external command,
operable program or batch file.

npm ERR! Windows_NT 10.0.17134
npm ERR! argv "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "run" "env:start"
npm ERR! node v6.9.4
npm ERR! npm  v3.10.10
npm ERR! code ELIFECYCLE
npm ERR! WordPress@5.3.0 env:start: `./tools/local-env/start.sh`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the WordPress@5.3.0 env:start script './tools/local-env/start.sh'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the WordPress package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     ./tools/local-env/start.sh
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs WordPress
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls WordPress
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     ...\SVN-core-wp\npm-debug.log

$ npm install
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@^1.2.7 (node_modules\chokidar\node_modules\fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.9: wanted {"os":"darwin","arch":"any"} (current: {"os":"win32","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@^1.2.7 (node_modules\jest-haste-map\node_modules\fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.9: wanted {"os":"darwin","arch":"any"} (current: {"os":"win32","arch":"x64"})
npm WARN WordPress@5.3.0 license should be a valid SPDX license expression

I am sure it just a case of not reading the doc's
Please point me to docs and let's add them to the into core so newbies can get this up and running

Thanks

[pento]

Replying to mcsf:

This is quite tricky. (Is it OK if I have a deep longing for Makefiles?)

If Makefiles were a reliable option for Windows, I'd be right there with you. 🙂

Replying to pbearne:

I am trying to test as "the dumb windows user"

I can't get to work
steps I have done

Download clean copy from SVN ​https://develop.svn.wordpress.org/trunk/

applied the patch (not sure it applied cleanly)

tried to run npm in the root and local-env folders

$ npm run env:start


> WordPress@5.3.0 env:start C:\Users\pbear\Documents\code\SVN-core-wp
> ./tools/local-env/start.sh

Thank you for testing, @pbearne! It looks like the patch didn't apply correctly, as the ./tools/local-env/start.sh script is removed by this patch.

Could you describe how you applied the patch, and any messages that showed when you applied it?

It's also worth noting that you should use 47767.3.diff​, not 47767.4.diff​. The former is correct for trunk, the latter is a proof of concept backport for the 3.7 branch.

[pento]

A few other requirements off the top of my head:

• You need to have Docker installed and running on your computer. If you don't have Docker installed, you can download it for Windows 10 Pro/Enterprise/Education ​here. For all other versions of Windows, here are ​installation instructions.
• After applying the patch, you'll need to run npm install before running npm run env:start.
• If you're not using Windows 10 Pro/Enterprise/Education, there's an additional command you'll need to run after starting the environment, in order to be able to visit the site in your browser.

Replying to pbearne:

I am sure it just a case of not reading the doc's
Please point me to docs and let's add them to the into core so newbies can get this up and running

You're absolutely right, documentation is an important part of this. Unfortunately, the documentation isn't written yet: my current focus has been on getting the tool working.

I greatly appreciate you jumping in and trying out this early version. Please post all issues you run into (no matter how minor), as that'll help improve both the tool and the documentation. 🙂

[pbearne]

@pento

reverted and updated SVN
applied .3 diff ( i used the TortoiseSVN to apply the patch ) seem to work this time

tools\local-env> npm install
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@^1.2.7 (node_modules\chokidar\node_modules\fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.9: wanted {"os":"darwin","arch":"any"} (current: {"os":"win32","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@^1.2.7 (node_modules\jest-haste-map\node_modules\fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.9: wanted {"os":"darwin","arch":"any"} (current: {"os":"win32","arch":"x64"})
npm WARN WordPress@5.3.0 license should be a valid SPDX license expression

I know this s waring but it would nice not to it
I then ran npm run env:start
that worked

so I go to ​http://localhost:8889/

and the npm install promote

in VVV it is run for us

 npm run dev

> WordPress@5.3.0 dev ...\code\SVN-core-wp
> grunt build --dev

Loading "Gruntfile.js" tasks...ERROR
>> SyntaxError: Unexpected token ...
Warning: Task "build" not found. Use --force to continue.

Aborted due to warnings.

npm ERR! Windows_NT 10.0.17134
npm ERR! argv "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "run" "dev"
npm ERR! node v6.9.4
npm ERR! npm  v3.10.10
npm ERR! code ELIFECYCLE
npm ERR! WordPress@5.3.0 dev: `grunt build --dev`
npm ERR! Exit status 3
npm ERR!
npm ERR! Failed at the WordPress@5.3.0 dev script 'grunt build --dev'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the WordPress package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     grunt build --dev
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs WordPress
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls WordPress
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     ...\SVN-core-wp\npm-debug.log

log

0 info it worked if it ends with ok
1 verbose cli [ 'C:\\Program Files\\nodejs\\node.exe',
1 verbose cli   'C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js',
1 verbose cli   'run',
1 verbose cli   'dev' ]
2 info using npm@3.10.10
3 info using node@v6.9.4
4 verbose run-script [ 'predev', 'dev', 'postdev' ]
5 info lifecycle WordPress@5.3.0~predev: WordPress@5.3.0
6 silly lifecycle WordPress@5.3.0~predev: no script for predev, continuing
7 info lifecycle WordPress@5.3.0~dev: WordPress@5.3.0
8 verbose lifecycle WordPress@5.3.0~dev: unsafe-perm in lifecycle true
9 verbose lifecycle WordPress@5.3.0~dev: PATH: C:\Program Files\nodejs\node_modules\npm\bin\node-gyp-bin;C:\Users\pbear\Documents\code\SVN-core-wp\node_modules\.bin;C:\ProgramData\DockerDesktop\version-bin;C:\Program Files\Docker\Docker\Resources\bin;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\ProgramData\Oracle\Java\javapath;C:\Program Files (x86)\Intel\iCLS Client\;C:\Program Files\Intel\iCLS Client\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files (x86)\Calibre2\;C:\Program Files\TortoiseSVN\bin;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\nodejs\;C:\php;C:\HashiCorp\Vagrant\bin;C:\Program Files\OpenVPN\bin;C:\Program Files\PuTTY\;C:\Program Files\Calibre2\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\Amazon\AWSCLI\;C:\Program Files\dotnet\;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files (x86)\Gpg4win\..\GnuPG\bin;C:\Program Files\TortoiseGit\bin;C:\Ruby24-x64\bin;C:\Users\pbear\AppData\Local\Microsoft\WindowsApps;C:\Users\pbear\AppData\Roaming\npm;;C:\Program Files\Lando\bin
10 verbose lifecycle WordPress@5.3.0~dev: CWD: C:\Users\pbear\Documents\code\SVN-core-wp
11 silly lifecycle WordPress@5.3.0~dev: Args: [ '/d /s /c', 'grunt build --dev' ]
12 silly lifecycle WordPress@5.3.0~dev: Returned: code: 3  signal: null
13 info lifecycle WordPress@5.3.0~dev: Failed to exec dev script
14 verbose stack Error: WordPress@5.3.0 dev: `grunt build --dev`
14 verbose stack Exit status 3
14 verbose stack     at EventEmitter.<anonymous> (C:\Program Files\nodejs\node_modules\npm\lib\utils\lifecycle.js:255:16)
14 verbose stack     at emitTwo (events.js:106:13)
14 verbose stack     at EventEmitter.emit (events.js:191:7)
14 verbose stack     at ChildProcess.<anonymous> (C:\Program Files\nodejs\node_modules\npm\lib\utils\spawn.js:40:14)
14 verbose stack     at emitTwo (events.js:106:13)
14 verbose stack     at ChildProcess.emit (events.js:191:7)
14 verbose stack     at maybeClose (internal/child_process.js:877:16)
14 verbose stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:226:5)
15 verbose pkgid WordPress@5.3.0
16 verbose cwd C:\Users\pbear\Documents\code\SVN-core-wp
17 error Windows_NT 10.0.17134
18 error argv "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "run" "dev"
19 error node v6.9.4
20 error npm  v3.10.10
21 error code ELIFECYCLE
22 error WordPress@5.3.0 dev: `grunt build --dev`
22 error Exit status 3
23 error Failed at the WordPress@5.3.0 dev script 'grunt build --dev'.
23 error Make sure you have the latest version of node.js and npm installed.
23 error If you do, this is most likely a problem with the WordPress package,
23 error not with npm itself.
23 error Tell the author that this fails on your system:
23 error     grunt build --dev
23 error You can get information on how to open an issue for this project with:
23 error     npm bugs WordPress
23 error Or if that isn't available, you can get their info via:
23 error     npm owner ls WordPress
23 error There is likely additional logging output above.
24 verbose exit [ 1, true ]

OK that is it for now

[pento]

We're making progress, thanks @pbearne!

Next up, could you download and install the ​latest LTS version of Node (currently 10.16.0).

Node 6.9 is quite old, I wouldn't be surprised if some of the packages we rely on don't work with it.

[pbearne]

ok that worked

I had to update the SASS npm rebuild node-sass

now I have the WP Installer running just need to work ou the DB details

[pbearne]

Can we setup up to auto-connect to the DB and maybe do to the 'npm run dev' as part of the Docker build npm run env:start

The fewer commands needed to get it up the better

Maybe we need some version checks in the script

I hadn't run env:install

that failed

\SVN-core-wp> npm run env:install

> WordPress@5.3.0 env:install C:\Users\....\Documents\code\SVN-core-wp
> dotenv npm run env:install-next


> WordPress@5.3.0 env:install-next C:\Users\....\Documents\code\SVN-core-wp
> npm run env:install-config && npm run env:reset-site && npm run env:install-site


> WordPress@5.3.0 env:install-config C:\Users\....\Documents\code\SVN-core-wp
> npm run env:cli config create -- --dbname=wordpress_develop --dbuser=root --dbpass=password --dbhost=mysql --path=/var/www/src --force && copyfiles -f src/wp-config.php .


> WordPress@5.3.0 env:cli C:\Users\....\Documents\code\SVN-core-wp
> docker-compose -f ./tools/local-env/docker-compose.yml -f ./tools/local-env/docker-compose.scripts.yml run --rm cli "config" "create" "--dbname=wordpress_develop" "--dbuser=root" "--dbpass=password" "--dbhost=mysql" "--path=/var/www/src" "--force"

Creating network "local-env_default" with the default driver
Creating volume "local-env_phpunit-uploads" with default driver
Pulling cli (garypendergast/wordpress-develop-cli:5.4-fpm)...
5.4-fpm: Pulling from garypendergast/wordpress-develop-cli
d4bce7fd68df: Pull complete
a3ed95caeb02: Pull complete
77a07a807b0b: Pull complete
8e9dd06f34b8: Pull complete
bdeb40d51bd7: Pull complete
2c8127992730: Pull complete
a5d836e0c192: Pull complete
f30196191d1a: Pull complete
dc7df7516240: Pull complete
a790cf986177: Pull complete
72cf88b6887f: Pull complete
c83f126e8f01: Pull complete
ERROR 2003 (HY000): Can't connect to MySQL server on 'mysql' (111)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! WordPress@5.3.0 env:cli: `docker-compose -f ./tools/local-env/docker-compose.yml -f ./tools/local-env/docker-compose.scripts.yml run --rm cli "config" "create" "--dbname=wordpress_develop" "--dbuser=root" "--dbpass=password" "--dbhost=mysql" "--path=/var/www/src" "--force"`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the WordPress@5.3.0 env:cli script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\pbear\AppData\Roaming\npm-cache\_logs\2019-07-28T20_01_45_807Z-debug.log
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! WordPress@5.3.0 env:install-config: `npm run env:cli config create -- --dbname=wordpress_develop --dbuser=root --dbpass=password --dbhost=mysql --path=/var/www/src --force && copyfiles -f src/wp-config.php .`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the WordPress@5.3.0 env:install-config script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\....\AppData\Roaming\npm-cache\_logs\2019-07-28T20_01_45_838Z-debug.log
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! WordPress@5.3.0 env:install-next: `npm run env:install-config && npm run env:reset-site && npm run env:install-site`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the WordPress@5.3.0 env:install-next script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\....\AppData\Roaming\npm-cache\_logs\2019-07-28T20_01_45_865Z-debug.log
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! WordPress@5.3.0 env:install: `dotenv npm run env:install-next`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the WordPress@5.3.0 env:install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\...\AppData\Roaming\npm-cache\_logs\2019-07-28T20_01_45_897Z-debug.log

[pento]

Replying to pbearne:

Can we setup up to auto-connect to the DB and maybe do to the 'npm run dev' as part of the Docker build npm run env:start

The fewer commands needed to get it up the better

Aye, I'll certainly look at where we can simplify it. It's worth noting that this way of doing this is intended for folks comfortable with the command line already: ​TestPress (which currently doesn't work with this patch) would be the method for everyone else.

Maybe we need some version checks in the script

That's possible, too.

I hadn't run env:install

that failed

ERROR 2003 (HY000): Can't connect to MySQL server on 'mysql' (111)

This seems to indicate that the MySQL server isn't running. Did you run npm env:start before running npm env:install?

If it's still failing after running npm env:start, please send the output of these commands:

docker-compose -f tools\local-env\docker-compose.yml ps
docker-compose -f tools\local-env\docker-compose.yml logs mysql

[mcsf]

@pento, I have no experience in configuring Travis, so bear with me, but a couple of questions looking at .travis.yml:

1. We are replacing Travis's preinstalled version of docker-compose with a specific version, 1.24.0. Is there a specific reason, is it about recency? Can we document in the file?

2. Should we turn the magic version number into a proper constant? I'm looking at Travis's ​own documentation on Docker.

Now onto docker-compose.yml:

3. With the switch to the nginx image, we drop the usage of WORDPRESS_DEBUG and WORDPRESS_CONFIG_EXTRA. What will the new recommended methods be to tweak WP runtime constants?

These questions aside, this looks pretty solid and tests well for me. It's nice to see some of these scripts "magically" gone!

[pento]

Replying to mcsf:

1. We are replacing Travis's preinstalled version of docker-compose with a specific version, 1.24.0. Is there a specific reason, is it about recency? Can we document in the file?

Strictly speaking, we can use 1.22.0 or newer. We need a version that supports docker-compose file format 3.7, for the init directive. This allows command line PHP tools (eg, phpunit, WP-CLI) to recognised Ctrl+C being pressed. (​ref)

I'll add comments to the docker-compose file, explaining what's going on.

2. Should we turn the magic version number into a proper constant? I'm looking at Travis's ​own documentation on Docker.

As it's only used in one place, I'm not too concerned about that. We have the inline version number with the wordpress-importer download in .travis.yml, too.

3. With the switch to the nginx image, we drop the usage of WORDPRESS_DEBUG and WORDPRESS_CONFIG_EXTRA. What will the new recommended methods be to tweak WP runtime constants?

Yah, I think we should re-add these. They were just a lower priority while I was working on getting it up and working.

Similarly, we also need:

• A way to add/remove PHP modules. Xdebug is the obvious one that comes to mind.
• Config variables for switching PHP and MySQL versions.

[pento]

In 47767.5.diff​:

• Added a bunch of documentation to the .env and docker-compose files.
• Enabled PHPUnit support. This can be run with npm run test:php.
• Enabled switching PHP versions with the LOCAL_PHP environment variable. (Note that currently, PHP 5.4 is the only alternative with Docker images built.)
• Enabled switching MySQL versions with the LOCAL_MYSQL environment variable.
• Added LOCAL_WP_DEBUG, LOCAL_WP_DEBUG_LOG, LOCAL_WP_DEBUG_DISPLAY, and LOCAL_SCRIPT_DEBUG environment variables, which set their corresponding wp-config.php debug setting.
• Added npm run env:clean, which stops the containers and removes the volumes. Particularly when switching MySQL versions.
• Added npm run env:reset, which stops the containers and removes all images. This will cause the next npm run env:start to download fresh images. Useful when you've broken everything, and want to start over from a known good state. 🔥

[pento]

In 47767.6.diff​:

• Docker images now exist for PHP 5.2-7.3.
• WP-CLI and PHPUnit images now obey the LOCAL_PHP environment variable.
• Include the docker-compose.scripts.yml file that was missing in the previous patch.
• Added a README.md file, with documentation of the NPM scripts.
• First pass at converting Travis to using Docker images (currently not working).

[pento]

In 47767.7.diff​:

• ​Travis now works. It uses Docker images where available, and local where it isn't. Running PHPUnit is a little slower in Docker than it is natively, but not overly so.
• Improved performance of PHPUnit when running in the build directory.

About the only thing remaining before this can go into trunk is to shift the Docker images over to the WordPress GitHub org.

If anyone feels like playing with how Docker handles updates to images, a good UX improvement for this would be to automatically recognise when there are updated images to pull from Docker Hub. At the moment, you can be stuck with old images, without realising it.

For now, running npm run env:reset will delete your local images, so new images will be downloaded when you start it up again.

[pento]

In 47767.8.diff​:

• Switch over to using ​wordpressdevelop images. The image definitions are generated and committed to the ​wpdev-docker-images GitHub repo, they're then built and deployed to Docker Hub by ​Travis.
• Ensure WP-CLI and PHPUnit obey the LOCAL_PHP_XDEBUG setting.

I'm pondering adding docker-compose pull to the npm run env:start command. This would add ~5 seconds to the start time when there are no updates (ie, nearly all of the time) but would ensure that the most up-to-date images have been downloaded.

[noisysocks]

This is looking good!

I received this error when running npm run env:reset with the containers running:

ERROR: Failed to remove image for service mysql: 409 Client Error: Conflict ("conflict: unable to remove repository reference "mysql:5.7" (must force) - container 015b4262f1c1 is using its referenced image f6509bac4980")

Other more minor notes:

• I still am wishing npm run dev ran grunt watch --dev. I understand the desire for consistency between npm and grunt, though.
• I mistakenly thought that npm run env:reset-site was a command that could be run, when it's actually an internal command used by npm run env:install. Not sure if a different name (e.g. env:install-reset) would help with this. Ideally we'd only have commands that the user is supposed to run in "scripts".
• It's awkward that npm run test:php, on first run, gives you an error about a missing wp-tests-config.php file with no clues on how to set one up using the built-in MySQL container.
• I received 'connection refused' errors when testing with PHP 5.3 and 5.4, and a WordPress error when using PHP 5.5.
• It would be neat if LOCAL_PHP=7.4-fpm worked so that we could test against the upcoming PHP release.
• I wasn't able to access WP Admin with MySQL set to use anything earlier than 5.6.
• How does one access the nginx error log? It might be useful to document this in e.g. a handbook page for this tool.
• The README is a nice addition, but I wonder if it goes into too much detail? Thinking it might be less confronting if we have README just outline how to start and stop the development environment, and then link to a handbook page which has more thorough documentation on local-env.
• Awkward phrasing in the README: I suggest "WordPress is a PHP/MySQL-based project. We have a basic development" instead of "WordPress is a PHP/MySQL-based project, we have a basic development".
• Typo in the README: Should say npm run env:cli, not npm run env-cli.

[pento]

Thanks for testing, @noisysocks!

Replying to noisysocks:

This is looking good!

I received this error when running npm run env:reset with the containers running:

ERROR: Failed to remove image for service mysql: 409 Client Error: Conflict ("conflict: unable to remove repository reference "mysql:5.7" (must force) - container 015b4262f1c1 is using its referenced image f6509bac4980")

Are you running the Gutenberg local-env, too? If that's running, it won't be able to delete the MySQL image that's in use.

• I still am wishing npm run dev ran grunt watch --dev. I understand the desire for consistency between npm and grunt, though.

I agree, I've changed it in 47767.11.diff​. This was only added when the e2e env landed, it's not a problem to change it.

• I mistakenly thought that npm run env:reset-site was a command that could be run, when it's actually an internal command used by npm run env:install. Not sure if a different name (e.g. env:install-reset) would help with this.

Fair point, 47767.11.diff​ prefixes the internal commands with __, which makes them a little more obviously not for external use.

Ideally we'd only have commands that the user is supposed to run in "scripts".

Agreed, and I'm fine with iterating on how this all works after we get it committed to trunk.

• It's awkward that npm run test:php, on first run, gives you an error about a missing wp-tests-config.php file with no clues on how to set one up using the built-in MySQL container.

Good catch! I don't think we have a simple method for generating wp-tests-config.php, the method in .travis.yml is ugly and not cross-platform compatible. A small script to generate the file might be an option.

• I received 'connection refused' errors when testing with PHP 5.3 and 5.4, and a WordPress error when using PHP 5.5.

For the 'connection refused' errors, it seems like nginx is redirecting to port 80 for requests that don't end in a /. That'll require further investigation.

• It would be neat if LOCAL_PHP=7.4-fpm worked so that we could test against the upcoming PHP release.

Yah, I thought about that (also for the nightly build), but decided to leave it for later, as it would also need a job to regenerate the build regularly.

• I wasn't able to access WP Admin with MySQL set to use anything earlier than 5.6.

MySQL 5.6 is currently the oldest version with ​official builds. We'll need to provide our own for older versions.

• How does one access the nginx error log? It might be useful to document this in e.g. a handbook page for this tool.

47767.11.diff​ adds a npm run env:logs command.

• The README is a nice addition, but I wonder if it goes into too much detail?

Yah, I agree. I've trimmed it down for now, we can add a link to the handbook later, once that handbook page exists.

[noisysocks]

Are you running the Gutenberg local-env, too?

I am not! I made sure docker ps was not listing anything before running npm run env:start; npm run env:reset.

[pento]

In 45745:

Build Tools: Add a WordPress Development Environment.

This commit adds the first iteration of a Docker-based config for setting up a local WordPress development environment.

Props pento, noisysocks, mcsf, pbearne, isabel_brison.
See #47767.

[pento]

In 45762:

Build Tools: Switch all Travis PHPUnit jobs over to Docker-based PHP.

Additionally, there are a handful of related minor changes in this commit:

• .travis.yml has been tidied up a little.
• [45745] was incorrectly marking Travis jobs as passed when some PHPUnit runs failed.
• Add the LOCAL_PHP_MEMCACHED environment variable, for enabling Memcached.
• Add the env:pull NPM script, for refreshing Docker images.

See #47767.

[pento]

47767.14.diff​ converts all of the NPM scripts into JS files.

I think this keeps it relatively clean: scripts only exist for commands that require extra logic in some way, otherwise, they can go through the generic docker.js.

@noisysocks, @mcsf, @youknowriad: You folks had opinions on this, how does this patch look to you?

[youknowriad]

@pento Thanks for the update, I think this looks fantastic.

Some quick thoughts:

• Should this tools/local-env/scripts/e2e.js live in the local-env folder? or maybe in the e2e tests folder? or a tools folder dedicated to tests?
• Is it still possible to pass extra arguments to the npm run test:e2e command to run a specific test or trigger the interactive mode with this new script?

[pento]

Replying to youknowriad:

• Should this tools/local-env/scripts/e2e.js live in the local-env folder? or maybe in the e2e tests folder? or a tools folder dedicated to tests?

That's true, it's not specific to local-env. I've moved it to tests/e2e/run-tests.js, instead.

• Is it still possible to pass extra arguments to the npm run test:e2e command to run a specific test or trigger the interactive mode with this new script?

Good catch, 47767.15.diff​ passes all extra arguments through to wp-scripts.

[youknowriad]

Changes look good.

While trying to test the new env. I had this error when running npm run env:install

> WordPress@5.3.0 env:install /Users/riad/Workspace/a8c/wordpress-trunk
> node ./tools/local-env/scripts/install.js

ERROR 1045 (28000): Access denied for user 'root'@'172.20.0.5' (using password: YES)
child_process.js:669
    throw err;
    ^

Error: Command failed: docker-compose -f ./tools/local-env/docker-compose.yml -f ./tools/local-env/docker-compose.scripts.yml run --rm cli config create --dbname=wordpress_develop --dbuser=root --dbpass=password --dbhost=mysql --path=/var/www/src --force
    at checkExecSyncError (child_process.js:629:11)
    at execSync (child_process.js:666:13)
    at wp_cli (/Users/riad/Workspace/a8c/wordpress-trunk/tools/local-env/scripts/install.js:28:2)
    at Object.<anonymous> (/Users/riad/Workspace/a8c/wordpress-trunk/tools/local-env/scripts/install.js:8:1)
    at Module._compile (internal/modules/cjs/loader.js:776:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:829:12)
npm ERR! code ELIFECYCLE

Probably unrelated with the last patch, it's more the former one.

[pento]

In 47767.16.diff​:

• Combine the docker-compose.yml and docker-compose.scripts.yml files, and move them to the root directory. This means we don't have to specify the filenames when running docker-compose, so can now use docker-compose.override.yml.
• Fix nginx redirecting to port 80 when the URL is a directory name, but doesn't end in a /. (eg, requesting /wp-admin instead of /wp-admin/.)

Note for committing: docker-compose.override.yml will need to be added to svn:ignore.

@youknowriad: I think there are issues when running the Gutenberg local environment and Core local environment at the same time, the MySQL instances try to mount the same internal data directory, or something of that nature.

I'm experimenting with how best to modify Gutenberg's local environment, so it can use Core, instead.

[pento]

Here's a sample docker-compose.override.yml to place in your WordPress checkout, which will load Gutenberg when you start the environment (assuming your Gutenberg directory is in a sibling directory called gutenberg, of course 🙂).

version: '3.7'

services:
  wordpress-develop:
    volumes:
      - ../gutenberg:/var/www/${LOCAL_DIR-src}/wp-content/plugins/gutenberg
  php:
    volumes:
      - ../gutenberg:/var/www/${LOCAL_DIR-src}/wp-content/plugins/gutenberg
  cli:
    volumes:
      - ../gutenberg:/var/www/${LOCAL_DIR-src}/wp-content/plugins/gutenberg
  phpunit:
    volumes:
      - ../gutenberg:/wordpress-develop/${LOCAL_DIR-src}/wp-content/plugins/gutenberg

[noisysocks]

This looks great! It's nice that we're able to avoid so much repetition. I applied 47767.16.diff​ and tested all of the commands on my macOS machine and it's working as before.

I noticed some places where the JavaScript doesn't quite follow our JavaScript coding standards and so ran the Gutenberg lint tool on these changes, see below. (Does Core have a similar tool? Could we get it to check these new files?)

$ cp ~/projects/gutenberg/.eslintrc.js .
$ npm install --no-save eslint-plugin-jest
$ npx eslint tools/local-env/scripts/*.js 

/Users/robert/projects/wordpress-develop/tools/local-env/scripts/install.js
   2:7    error  Identifier 'wait_on' is not in camel case  camelcase
  13:30   error  Expected space(s) after '${'               template-curly-spacing
  13:58   error  Expected space(s) before '}'               template-curly-spacing
  14:34   error  Expected space(s) after '${'               template-curly-spacing
  14:66   error  Expected space(s) before '}'               template-curly-spacing
  15:38   error  Expected space(s) after '${'               template-curly-spacing
  15:74   error  Expected space(s) before '}'               template-curly-spacing
  16:34   error  Expected space(s) after '${'               template-curly-spacing
  16:66   error  Expected space(s) before '}'               template-curly-spacing
  22:40   error  Expected space(s) after '${'               template-curly-spacing
  22:64   error  Expected space(s) before '}'               template-curly-spacing
  22:66   error  A space is required before ']'             array-bracket-spacing
  25:162  error  Expected space(s) after '${'               template-curly-spacing
  25:186  error  Expected space(s) before '}'               template-curly-spacing
  33:10   error  Identifier 'wp_cli' is not in camel case   camelcase
  34:41   error  Expected space(s) after '${'               template-curly-spacing
  34:46   error  Expected space(s) before '}'               template-curly-spacing

/Users/robert/projects/wordpress-develop/tools/local-env/scripts/start.js
  12:58   error  Expected space(s) after '${'  template-curly-spacing
  12:93   error  Expected space(s) before '}'  template-curly-spacing
  12:121  error  Missing semicolon             semi
  13:14   error  Expected space(s) after '${'  template-curly-spacing
  13:26   error  Expected space(s) before '}'  template-curly-spacing
  13:40   error  Expected space(s) after '${'  template-curly-spacing
  13:73   error  Expected space(s) before '}'  template-curly-spacing
  13:92   error  Expected space(s) after '${'  template-curly-spacing
  13:116  error  Expected space(s) before '}'  template-curly-spacing
  13:132  error  Expected space(s) after '${'  template-curly-spacing
  13:156  error  Expected space(s) before '}'  template-curly-spacing
  13:159  error  Expected space(s) after '${'  template-curly-spacing
  13:183  error  Expected space(s) before '}'  template-curly-spacing

✖ 30 problems (30 errors, 0 warnings)
  28 errors and 0 warnings potentially fixable with the `--fix` option.

[pento]

In 47767.18.diff​:

• npm run env:install now sets up a wp-tests-config.php.
• A bunch of unnecessary cruft is gone from .travis.yml.

I noticed some places where the JavaScript doesn't quite follow our JavaScript coding standards and so ran the Gutenberg lint tool on these changes, see below. (Does Core have a similar tool? Could we get it to check these new files?)

Core uses JSHint, but it's configured for ES5, not ES6. I don't know if anyone is working merging the different tools between Gutenberg and Core.

[pento]

In 45783:

Build Tools: Miscellaneous local-env improvements.

• Move the functionality for controlling local-env out of package.json, into JS scripts.
• Merge the docker-compose config files, and move it to the root directory. This allows docker-compose.override.yml to work for local overrides.
• Fix nginx redirecting to port 80 under some circumstances.
• npm run env:install now creates wp-tests.config.php for you.
• Cleaned up a bunch of cruft in .travis.yml.

See #47767.

[pento]

In 45784:

Build Tools: Add the docker-compose.yml file, missed in [45783].

See #47767.

[pento]

In 45785:

Build Tools: Remove the old docker-compose.yml file, now no longer used.

See #47767.

[pento]

In 45799:

Build Tools: Run composer commands in a Docker container on Travis.

This tidies up the .travis.yml file a little more.

See #47767.

[pento]

In 45800:

Build Tools: Change the PHPUnit Docker image's working directory to match the other images.

Also, pass the WP_MULTISITE environment variable along to the container.

See #47767.

[pento]

In 45819:

Build Tools: Improve local-env start behaviour under Docker Toolbox.

Docker Toolbox requires port forwarding to be configured, but generates error when trying to forward a port that's already been taken.

This change removes clashing port forwarding rules before adding our own.

See #47767.

[pento]

@anevins: Hey, let's move ​our Slack conversation here.

I haven't been able to reproduce the issue you reported, so I'd like to gather some more information:

What shell are you using? CMD, PowerShell, Git Bash, or something else?

What version of these tools do you have?

docker version
docker-compose -v
node -v
npm -v

Can you confirm that you've run npm install recently?

Could you post the full output of running these commands in your wordpress-develop directory?

git status
npm run env:reset
npm run env:start
npm run env:install

[pento]

In 45885:

Build Tools: Pass the LOCAL_DIR environment variable through to the phpunit Docker container.

This allows code running inside the container to know whether it's supposed to be running in build or src.

See #47767.

[noisysocks]

Core uses JSHint, but it's configured for ES5, not ES6. I don't know if anyone is working merging the different tools between Gutenberg and Core.

I wonder if maybe @nerrad has looked into it? Definitely a worthwhile endeavour if our goal is to one day unify the two codebases! 🙂

[nerrad]

I wonder if maybe @nerrad has looked into it?

I have not. I agree it'd be nice to unify the tools though (that wasn't me volunteering for the job 😉).

[anevins]

Hey @pento Sorry for the late reply.

$ docker version
Client:
 Version:           19.03.1
 API version:       1.40
 Go version:        go1.12.7
 Git commit:        74b1e89e8a
 Built:             Wed Jul 31 15:18:18 2019
 OS/Arch:           windows/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.1
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.5
  Git commit:       74b1e89e8a
  Built:            Thu Jul 25 21:27:55 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

$ docker-compose -v
docker-compose version 1.24.1, build 4667896b

$ node -v
v8.9.1

$ npm -v
5.5.1

I just removed node_modules and added them again via npm install and succeeded;

$ npm install
...

added 2357 packages in 832.337s

$ npm run env:reset

> WordPress@5.3.0 env:reset C:\Users\andre\Documents\git\wordpress-develop
> node ./tools/local-env/scripts/docker.js down --rmi all -v --remove-orphans

Removing network wordpress-develop_wpdevnet
WARNING: Network wordpress-develop_wpdevnet not found.
Removing volume wordpress-develop_mysql
WARNING: Volume wordpress-develop_mysql not found.
Removing volume wordpress-develop_phpunit-uploads
WARNING: Volume wordpress-develop_phpunit-uploads not found.
Removing image mysql:5.7
WARNING: Image mysql:5.7 not found.
Removing image wordpressdevelop/php:latest
WARNING: Image wordpressdevelop/php:latest not found.
Removing image nginx:alpine
WARNING: Image nginx:alpine not found.
Removing image wordpressdevelop/cli:latest
WARNING: Image wordpressdevelop/cli:latest not found.
Removing image wordpressdevelop/phpunit:latest
WARNING: Image wordpressdevelop/phpunit:latest not found.

$ npm run env:start

> WordPress@5.3.0 env:start C:\Users\andre\Documents\git\wordpress-develop
> node ./tools/local-env/scripts/start.js

Pulling mysql (mysql:5.7)...
ERROR: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 10.0.2.3:53: read udp 10.0.2.15:33686->10.0.2.3:53: i/o timeout
child_process.js:644
    throw err;
    ^

Error: Command failed: docker-compose up -d wordpress-develop
    at checkExecSyncError (child_process.js:601:13)
    at execSync (child_process.js:641:13)
    at Object.<anonymous> (C:\Users\andre\Documents\git\wordpress-develop\tools\local-env\scripts\start.js:7:1)
    at Module._compile (module.js:635:30)
    at Object.Module._extensions..js (module.js:646:10)
    at Module.load (module.js:554:32)
    at tryModuleLoad (module.js:497:12)
    at Function.Module._load (module.js:489:3)
    at Function.Module.runMain (module.js:676:10)
    at startup (bootstrap_node.js:187:16)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! WordPress@5.3.0 env:start: `node ./tools/local-env/scripts/start.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the WordPress@5.3.0 env:start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\andre\AppData\Roaming\npm-cache\_logs\2019-08-29T22_59_32_009Z-debug.log

$ npm run env:install

> WordPress@5.3.0 env:install C:\Users\andre\Documents\git\wordpress-develop
> node ./tools/local-env/scripts/install.js

Pulling cli (wordpressdevelop/cli:latest)...
ERROR: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 10.0.2.3:53: read udp 10.0.2.15:58902->10.0.2.3:53: i/o timeout
child_process.js:644
    throw err;
    ^

Error: Command failed: docker-compose run --rm cli config create --dbname=wordpress_develop --dbuser=root --dbpass=password --dbhost=mysql --path=/var/www/src --force
    at checkExecSyncError (child_process.js:601:13)
    at execSync (child_process.js:641:13)
    at wp_cli (C:\Users\andre\Documents\git\wordpress-develop\tools\local-env\scripts\install.js:44:2)
    at Object.<anonymous> (C:\Users\andre\Documents\git\wordpress-develop\tools\local-env\scripts\install.js:9:1)
    at Module._compile (module.js:635:30)
    at Object.Module._extensions..js (module.js:646:10)
    at Module.load (module.js:554:32)
    at tryModuleLoad (module.js:497:12)
    at Function.Module._load (module.js:489:3)
    at Function.Module.runMain (module.js:676:10)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! WordPress@5.3.0 env:install: `node ./tools/local-env/scripts/install.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the WordPress@5.3.0 env:install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\andre\AppData\Roaming\npm-cache\_logs\2019-08-29T23_00_36_602Z-debug.log

[pento]

Thanks for the info, @anevins!

There are a couple of things that jump out:

• Your versions of Node and NPM are pretty old, could you head over to ​https://nodejs.org/ and download/install the latest LTS version (currently 10.16.3).
• It looks like Docker is unable to connect to the Docker Registry, for downloading images. Do you need to use a proxy to connect to the internet, or are you using a VPN? Both of these can cause problems.

[Clorith]

47767.2.patch​ adds a LOCAL_URL entry to the .env file.

This value defaults to wpdev.localhost, which is the most readily available address for reaching services running on your local machine, due to how some browsers (in particular Edge) don't respect the local hosts file when doing address resolving so using a custom TLD won't work for us.

The reason for needing more than just localhost as the address is to make sure features requiring loopbacks continue to work even on the test environment, this can only be done by using a fully qualified domain that we can set up aliases for in the Docker environment.

[desrosj]

Is there anyway to have a custom .env file that takes precedence over the one included that can be ignored by version control?

Using export/SET/Set-Item commands work, but it does not persist. Changing the .env file works temporarily, but it's easy to revert those changes when refreshing your local environment. Lately I have been using 7.4-fpm, but forgetting to export that option before starting the environment has happened for me a lot.

Not sure if I missed this somewhere in this discussion or the Make/Core posts.

[anevins]

Heads up, I've updated node to v10.16.3.

When running $ npm run env:install, I get the following error:

wordpress-develop (master)
$ npm run env:install

> WordPress@5.3.0 env:install C:\Users\andre\Documents\git\wordpress-develop
> node ./tools/local-env/scripts/install.js

Pulling cli (wordpressdevelop/cli:latest)...
latest: Pulling from wordpressdevelop/cli
1ab2bdfe9778: Already exists
1448c64389e0: Already exists
4b8a4e62b444: Already exists
9eb9d1e8e241: Already exists
a7bf87822557: Already exists
ee32605e8bd1: Already exists
965457a3bcd9: Already exists
9960277f11c7: Already exists
20195d6057d9: Already exists
39ec1e7c5417: Already exists
1b6c868c5491: Already exists
173b6011791e: Already exists
46e4ebb49881: Already exists
169ed2354892: Already exists
0d469ce63501: Pull complete
bf97528fbd89: Pull complete
Digest: sha256:ec6868124182075d06c013b449a9eb3cd8fd39ac7539f79587b82f8379e0f397
Status: Downloaded newer image for wordpressdevelop/cli:latest
Error: This does not seem to be a WordPress installation.
Pass --path=`path/to/wordpress` or run `wp core download`.
child_process.js:669
    throw err;
    ^

Error: Command failed: docker-compose run --rm cli config create --dbname=wordpress_develop --dbuser=root --dbpass=password --dbhost=mysql --path=/var/www/src --force
    at checkExecSyncError (child_process.js:629:11)
    at execSync (child_process.js:666:13)
    at wp_cli (C:\Users\andre\Documents\git\wordpress-develop\tools\local-env\scripts\install.js:44:2)
    at Object.<anonymous> (C:\Users\andre\Documents\git\wordpress-develop\tools\local-env\scripts\install.js:9:1)
    at Module._compile (internal/modules/cjs/loader.js:778:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:831:12)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! WordPress@5.3.0 env:install: `node ./tools/local-env/scripts/install.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the WordPress@5.3.0 env:install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\andre\AppData\Roaming\npm-cache\_logs\2019-09-21T07_26_35_581Z-debug.log

[desrosj]

I'm going to convert this to a blessed task so that work can continue after beta 1 and be wrapped up.

[pento]

@desrosj: Unfortunately, there isn't a project-specific way to override .env permanently. There is a workaround, which is to define the variables in your global settings (eg, ~/.bashrc), but this will define it everywhere.

@anevins: Could you confirm that the WordPress files are properly checked out in the src subdirectory?

I'm not entirely sure what's causing this, we may need to organise a screen-sharing session so I can debug further.

[whyisjake]

As part of #48140, we committed .nvmrc files to older versions of WordPress.

[anevins]

Confirmed files are there in the src directory

[pento]

In 46320:

Build Tools: Pass the current uid/gid to Docker containers.

This prepares for an upcoming change to the Docker images, which improves file permission handling.

See #47767.

[netweb]

I'm presuming for the most part this will be finished for 5.3, I'll leave this in your Docker'ised hands @pento

[tusca95]

Hi all!

I'm trying to install wordpress-develop.git in a centos machine with the latest docker and npm.

I've encountered some errors during the setup.

First of all Phantomjs wasn't able to found, so i'll have to install it with phantomjs-prebuilt, then i found that npm needs the packet bzip2 to extract zip files, so it is necessary to have it in my same software configuration.

Solved this one, i have some problem with the run ​build:dev command. This is the output of the command

npm run ​build:dev

WordPress@5.3.0 ​build:dev /root/wordpress-develop
grunt build --dev

Loading "Gruntfile.js" tasks...ERROR

Error: ENOENT: no such file or directory, scandir '/root/wordpress-develop/node_modules/node-sass/vendor'

Warning: Task "build" not found. Use --force to continue.

Aborted due to warnings.
npm ERR! code ELIFECYCLE
npm ERR! errno 3
npm ERR! WordPress@5.3.0 ​build:dev: grunt build --dev
npm ERR! Exit status 3
npm ERR!
npm ERR! Failed at the WordPress@5.3.0 ​build:dev script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/2019-10-11T10_16_45_860Z-debug.log

Hope that someone can find a solution!!

Have a nice day

[tusca95]

Replying to tusca95:

Hi all!

I'm trying to install wordpress-develop.git in a centos machine with the latest docker and npm.

I've encountered some errors during the setup.

First of all Phantomjs wasn't able to found, so i'll have to install it with phantomjs-prebuilt, then i found that npm needs the packet bzip2 to extract zip files, so it is necessary to have it in my same software configuration.

Solved this one, i have some problem with the run ​build:dev command. This is the output of the command

npm run ​build:dev

WordPress@5.3.0 ​build:dev /root/wordpress-develop
grunt build --dev

Loading "Gruntfile.js" tasks...ERROR

Error: ENOENT: no such file or directory, scandir '/root/wordpress-develop/node_modules/node-sass/vendor'

Warning: Task "build" not found. Use --force to continue.

Aborted due to warnings.
npm ERR! code ELIFECYCLE
npm ERR! errno 3
npm ERR! WordPress@5.3.0 ​build:dev: grunt build --dev
npm ERR! Exit status 3
npm ERR!
npm ERR! Failed at the WordPress@5.3.0 ​build:dev script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/2019-10-11T10_16_45_860Z-debug.log

Hope that someone can find a solution!!

Have a nice day

So after a while i've found the issue, need to install grunt with

npm install grunt

and than reconfigure node-sass with

npm rebuild node-sass

[desrosj]

With 5.3 RC1 today, I'm going to close this ticket out. It has received a full release cycle of testing.

Moving forward, new tickets should be opened for bugs and enhancements.

In order to keep this ticket in the 5.3 milestone for historical record, I created #48301 to continue working on that.

[SergeyBiryukov]

In 46999:

Build/Test Tools: Pass the TRAVIS_BRANCH and TRAVIS_PULL_REQUEST environment variables along to the Docker container.

This ensures that WP_UnitTestCase::skipOnAutomatedBranches() has access to these variables.

See #49050, #47767.

[desrosj]

In 49267:

Build/Test Tools: Pass GitHub Action related environment variables to the Docker container.

This ensures that WP_UnitTestCase::skipOnAutomatedBranches() has access to these variables so that time sensitive tests can be skipped when appropriate.

This also updates that logic to be more clear.

Follow up to [49264].

Props ocean90, johnbillion.
See #50401, #49050, #47767.

[desrosj]

In 49527:

Build/Test Tools: Backport the local Docker environment to the 5.2 branch.

This commit introduces the Docker-based local WordPress development environment to the 5.2 branch and converts the Travis test jobs to utilize this environment for easier and more consistent testing.

Merges [45745,45762,45783-45784,45800,45819,45885,46320,46999,47225,47912,48121,49335,49358,49360,49362] to the 5.2 branch.
See #48301, #47767.

[desrosj]

In 49528:

Build/Test Tools: Backport the local Docker environment to the 5.2 branch.

This commit introduces the Docker-based local WordPress development environment to the 5.2 branch and converts the Travis test jobs to utilize this environment for easier and more consistent testing.

Merges [45745,45762,45783-45784,45800,45819,45885,46320,46999,47225,47912,48121,49335,49358,49360,49362] to the 5.1 branch.
See #48301, #47767.

[desrosj]

In 49529:

Build/Test Tools: Backport the local Docker environment to the 5.0 branch.

This commit introduces the Docker-based local WordPress development environment to the 5.0 branch and converts the Travis test jobs to utilize this environment for easier and more consistent testing.

Until existing blockers with the PHP 5.2 Docker container can be solved, the PHP 5.2 test job will remain using the Travis precise image.

Merges [45745,45762,45783-45784,45800,45819,45885,46320,46999,47225,47912,48121,49335,49358,49360,49362] to the 5.0 branch.
See #48301, #47767.

[desrosj]

In 49530:

Build/Test Tools: Backport the local Docker environment to the 4.9 branch.

This commit introduces the Docker-based local WordPress development environment to the 4.9 branch and converts the Travis test jobs to utilize this environment for easier and more consistent testing.

Until existing blockers with the PHP 5.2 Docker container can be solved, the PHP 5.2 test job will remain using the Travis precise image.

Merges [45745,45762,45783-45784,45800,45819,45885,46320,46999,47225,47912,48121,49335,49358,49360,49362] to the 4.9 branch.
See #48301, #47767.

[desrosj]

In 49531:

Build/Test Tools: Backport the local Docker environment to the 4.8 branch.

This commit introduces the Docker-based local WordPress development environment to the 4.8 branch and converts the Travis test jobs to utilize this environment for easier and more consistent testing.

Until existing blockers with the PHP 5.2 Docker container can be solved, the PHP 5.2 test job will remain using the Travis precise image.

Merges [45745,45762,45783-45784,45800,45819,45885,46320,46999,47225,47912,48121,49335,49358,49360,49362] to the 4.8 branch.
See #48301, #47767.

[desrosj]

In 49532:

Build/Test Tools: Backport the local Docker environment to the 4.7 branch.

This commit introduces the Docker-based local WordPress development environment to the 4.7 branch and converts the Travis test jobs to utilize this environment for easier and more consistent testing.

Until existing blockers with the PHP 5.2 Docker container can be solved, the PHP 5.2 test job will remain using the Travis precise image.

Merges [45745,45762,45783-45784,45800,45819,45885,46320,46999,47225,47912,48121,49335,49358,49360,49362] to the 4.7 branch.
See #48301, #47767.

[desrosj]

In 49533:

Build/Test Tools: Backport the local Docker environment to the 4.6 branch.

This commit introduces the Docker-based local WordPress development environment to the 4.6 branch and converts the Travis test jobs to utilize this environment for easier and more consistent testing.

Until existing blockers with the PHP 5.2 Docker container can be solved, the PHP 5.2 test job will remain using the Travis precise image.

Merges [45745,45762,45783-45784,45800,45819,45885,46320,46999,47225,47912,48121,49335,49358,49360,49362] to the 4.6 branch.
See #48301, #47767.

[desrosj]

In 50239:

Build/Test Tools: Correct svn:mergeinfo for the 4.9 branch.

[49530] also merged [44176], [45445], and [49267], but these were missed in the commit message.

See #48301, #47767.

[desrosj]

In 50240:

Build/Test Tools: Correct svn:mergeinfo for the 4.8 branch.

[49531] also merged [44176], [45445], and [49267], but these were missed in the commit message.

See #48301, #47767.

[desrosj]

In 50241:

Build/Test Tools: Correct svn:mergeinfo for the 4.7 branch.

[49532] also merged [44176], [45445], and [49267], but these were missed in the commit message.

See #48301, #47767.

[desrosj]

In 50242:

Build/Test Tools: Correct svn:mergeinfo for the 4.6 branch.

[49533] also merged [44176], [45445], and [49267], but these were missed in the commit message.

See #48301, #47767.

[desrosj]

In 50243:

Build/Test Tools: Backport the local Docker environment to the 4.5 branch.

This commit introduces the Docker-based local WordPress development environment to the 4.5 branch and converts the Travis test jobs to utilize this environment for easier and more consistent testing.

Merges [44176,45445,45745,45762,45783-45784,45800,45819,45885,46320,46999,47225,47912,48121,49267,49335,49358,49360,49362] to the 4.5 branch.
See #48301, #47767.

[desrosj]

In 50244:

Build/Test Tools: Backport the local Docker environment to the 4.4 branch.

This commit introduces the Docker-based local WordPress development environment to the 4.4 branch.

Merges [44176,45445,45745,45762,45783-45784,45800,45819,45885,46320,46999,47225,47912,48121,49267,49335,49358,49360,49362] to the 4.4 branch.
See #48301, #47767.

[desrosj]

In 50245:

Build/Test Tools: Backport the local Docker environment to the 4.3 branch.

This commit introduces the Docker-based local WordPress development environment to the 4.3 branch.

Merges [44176,45445,45745,45762,45783-45784,45800,45819,45885,46320,46999,47225,47912,48121,49267,49335,49358,49360,49362] to the 4.3 branch.
See #48301, #47767.

[desrosj]

In 50246:

Build/Test Tools: Backport the local Docker environment to the 4.2 branch.

This commit introduces the Docker-based local WordPress development environment to the 4.2 branch.

Merges [44176,45445,45745,45762,45783-45784,45800,45819,45885,46320,46999,47225,47912,48121,49267,49335,49358,49360,49362] to the 4.2 branch.
See #48301, #47767.

[desrosj]

In 50247:

Build/Test Tools: Backport the local Docker environment to the 4.1 branch.

This commit introduces the Docker-based local WordPress development environment to the 4.1 branch.

Merges [44176,45445,45745,45762,45783-45784,45800,45819,45885,46320,46999,47225,47912,48121,49267,49335,49358,49360,49362] to the 4.1 branch.
See #48301, #47767.

[desrosj]

In 50248:

Build/Test Tools: Backport the local Docker environment to the 4.0 branch.

This commit introduces the Docker-based local WordPress development environment to the 4.0 branch.

Merges [44176,45445,45745,45762,45783-45784,45800,45819,45885,46320,46999,47225,47912,48121,49267,49335,49358,49360,49362] to the 4.0 branch.
See #48301, #47767.

[desrosj]

In 50249:

Build/Test Tools: Backport the local Docker environment to the 3.9 branch.

This commit introduces the Docker-based local WordPress development environment to the 3.9 branch.

Merges [44176,45445,45745,45762,45783-45784,45800,45819,45885,46320,46999,47225,47912,48121,49267,49335,49358,49360,49362] to the 3.9 branch.
See #48301, #47767.

[desrosj]

In 50250:

Build/Test Tools: Backport the local Docker environment to the 3.8 branch.

This commit introduces the Docker-based local WordPress development environment to the 3.8 branch.

Merges [44176,45445,45745,45762,45783-45784,45800,45819,45885,46320,46999,47225,47912,48121,49267,49335,49358,49360,49362] to the 3.8 branch.
See #48301, #47767.

[desrosj]

In 50251:

Build/Test Tools: Backport the local Docker environment to the 3.7 branch.

This commit introduces the Docker-based local WordPress development environment to the 3.7 branch.

Merges [44176,45445,45745,45762,45783-45784,45800,45819,45885,46320,46999,47225,47912,48121,49267,49335,49358,49360,49362] to the 3.7 branch.
See #47767.
Fixes #48301.

[spacedmonkey]

#43711 was marked as a duplicate.