Let's create a standard development setup for WordPress core.

[omarreiss]

^{This ticket was triggered by previous discussion in https://core.trac.wordpress.org/ticket/43055}
The current default for developing WordPress core seems to be VVV. While VVV is doing a great job in making it easier to run WordPress locally, I think we could do better, especially for WordPress core.

With this ticket I would like to set a goal for improvement and discuss ways to achieve it.

The goal: a simple, two-step process for setting up a WordPress core development environment

The simplest development setup workflow I can think of consists just of two simple steps:

Download and boot

That's how simple it should (eventually) be. WordPress should now be running on some local address and every developer should be able to dive in the code and make changes.

Integrating watch and build behavior

As we plan to introduce a build step to the development workflow, it becomes more important that building is not an afterthought but something that is integrated into the development workflow.

The desirable state is one in which we just boot and can trust all changes are processed to whatever's being served in the browser.

This unfortunately is not the case at the moment. If we want to do better, we need a more unified approach.

Requirements

We need something that is:

• the easiest possible development workflow: just download and boot.
• fast: reduce build and boot times to a minimum.
• reliable: it should just work.

@pento noted that eventually he'd like this thing to be configurable and installable through a GUI in order to reduce the barrier of entry for new contributors to the absolute minimum.

Possible solutions

Docker install script

@pento built automated development environment setup scripts for Gutenberg based on Docker.

The Gutenberg setup script walks the user through setting up their development environment. It would obviously need to be written to work on Windows, too, before it was a valid option for Core. That will be fun. 🙂

There are a few benefits to using a Docker based environment.

• It's fast
• It's composable
• It only requires Docker

Aside: If we get good and well maintained Docker images for WordPress, this could also benefit the web hosting community, will make WordPress more optimized for cloud native hosting infrastructures, could also help optimize WordPress unit testing in container based CI solutions like Travis etc.

VVV install script

@TJNowell noted similar advantages might also be achieved with VVV

I see no reason the docker call in the GB script couldn't be a git clone ... cd vvv.. vagrant up --provision

Of course VVV and the community behind it include a great deal of experience with regard to the intricacies of creating the development environment for WordPress.

If we can make it easy, fast and reliable enough, that might just be what we need!

[youknowriad]

Thanks for opening this issue

Two things that I think are important to address as part of this:

• Ideally, the integration and e2e tests (even unit tests since PHP unit tests require a full setup) should use the same environment
• A dev environment even if it's hidden behind magical scripts should stay simple. When bugs happen developers of any level should be able to understand, debug and tweak how the environment works.

I admit having a preference for docker setups because it's light compared to a real Virtual Machine.
I think the current docker setup is Gutenberg has grown to be slightly more complex to what I want it to be. (two many scripts calling each other), I believe we can achieve the same goals using a simple docker-compose file.

[TJNowell]

With note to VVV:

Currently VVV uses the NFS option to map folders, which is why grunt watch is slow/unreliable, but there are mechanisms for mitigating this. Such as reducing the caching time to 1ms, or disabling caching at all, or even using a completely different mechanism. These options were set in VVV a ​long time ago, and so the exact reasons it's set up that way may be lost to the sands of time ( depends how sleepy Jeremy was at the time )

But since the option of a setup script is present, we could setup node etc locally on the host bypassing any performance costs of mounted/shared folders in Vagrant/Docker

There are also other Vagrant options to be explored, such as changes to guard, or using watchr.

I also had an idea of how to mitigate the problem of all users having a grunt watch in the background even when they aren't doing work on core, make it an option turned off by default

Re: Docker

Outside linux, docker still runs inside a VM, so a docker container may have different performance, but it still has a performance cost. It appears that with caching eliminated, the tradeoff is speed

See here for more details:

​https://docs.docker.com/docker-for-mac/osxfs/#performance-issues-solutions-and-roadmap

[Mte90]

I am a huge fan of VVV, so I will try to be not from a side and be impartial.

One of the problems that has VVV right now is that require a lot of time on first start because of installation (and download of a lot of things) but the plan for version 3 (​https://github.com/Varying-Vagrant-Vagrants/VVV/issues/1469) with a box that contain everything and also the parallel for the provision will improve a lot this point.

The other problems that often at contributor day we have is Windows. Install virtualbox, vagrant, git on Windows is a big step because people do't know anything of them.
Often the problems on VVV is because there are different versions of powershell on the system (don't ask me why) and Vagrant or Virtualbox works with issue based on the version of ps.

With docker we will have more problems on Windows because we need a virtual machine so (virtualbox and git will be required in any case) so the only advantage that we don't have anymore the easy vagrant to use but docker (that is not so entry level).

Sure, there is the VVV USB Generator that prepare all the executables for download (maybe also the latest image) but we need to think that at contributor day not everyone has an OS dev friendly for *nix stuff.
We have already a lot of issues with vagrant I can image with Docker virtualized...

From the other side VVV has the cool part to be used also for not contribute and very easy with vagrant up/halt and db backup so usually people are interested in that then contribute.

So probably the better way is to start to work on VVV at least for the GUI side, so we can gather feedback because the technical problems are from both the side.

[TJNowell]

Don't forget that we'll need to instruct users on how to setup Hyper-V to get Docker working, and docker has a lot of the same file performance issues Vagrant has, since it runs in a VM on all platforms except Linux. Users may even have to upgrade to Windows 10 Professional. No amount of USB drives will save us from Netbooks and cheap Windows machine trying to perform large scale downloads over conference wifi via Microsoft update servers

VVV 3 should resolve the provisioning time, and there are options for improving file system performance, such as vagrant-bindfs, or changing cache timings

As an aside, I see some of the reasoning for docker is that travis doesn't provide PHP 5.2 and you can't run a VM in travis. Given that we power 30% of the web, has nobody thought to spend $2 a month and set up CI with GitLab/Jenkin/Circle CI/Teamcity etc and just have a dedicated build server? We set up a Jenkins server to build boxes for VVV 3, and VVV already supports running multiple versions of PHP side by side, so it's not a huge task by any stretch

[netweb]

Integrating watch and build behavior: As we plan to introduce a build step to the development workflow, it becomes more important that building is not an afterthought but something that is integrated into the development workflow.

• A make/core post here has just been posted on the ​above subject and it is an important aspect to take into consideration, having access to a ​https://en.wikipedia.org/wiki/Command-line_interface (CLI) to run grunt watch and grunt build tasks will be an important part of the WordPress development workflow.

• WordPress will also soon begin moving away from Grunt to npm scripts and webpack, see #43731

• WordPress' CLI commands and scripts will need to be compatible with Mac, Windows, and *nix operating systems

---

Here are a few of the wishlist requirements I see whilst wearing my "Build Tools Component Maintainer Hat"

• ​Composer is required as this was recently added in #43558 for ​PHPCS support

• Support multiple versions of MySQL and MariaDB for testing #30462 #42811

• Support multiple versions of PHPUnit for testing (Legacy PHP versions required legacy PHPUnit versions) #43218

• Support multiple versions of PHP for testing, Travis CI ended support of PHP 5.2 and PHP 5.3 in September 2017 and WordPress is only able to use Travis CI builds on Ubuntu 12.04 LTS servers that no longer receive updates #41292.

This last item here is the most important here, there are a not many options available to us apart from a using a custom Docker image with Travis CI, at the moment Gutenberg is using ​these scripts to manually install PHP 5.2 & 5.3 versions for Travis CI though they are quite fragile.

Also in #41292 @miyauchi created a Docker image supporting PHP 5.2 - 7.1 and PHPUnit, this now supports PHP 7.2 as can be seen in the GitHub repo ​here, it was a great first step into exploring the potential of a Docker solution for Travis CI.

---

• Docker can be run on Mac, Windows, and *nix platforms

• WordPress can have a purpose-built Travis CI environment

• The same purpose-built Travis CI environment can be used locally

• A standardised "base" WordPress development environment

• Existing Vagrant based solutions can use the Docker for ​provisioning images instead of VirtualBox VMs

• Existing Docker solutions can use the images as their base images

[TJNowell]

Keep in mind that when using docker with Vagrant actually runs inside a Virtualbox VM. Under MacOS native docker uses an xhyve VM running Alpine Linux, and under Windows it's an Alpine Linux VM under a Hyper-V managed VM.

I'm mindful that in the future, we'll be told that docker containers are the only option, and that other methods will suffer from this. I'm also disappointed that Travis limitations are dictating local dev environment decisions. If docker is the solution there, then fine, but that's not the same issue as a standardised development environment.

Before we blindly charge down this route, I'd like to see a solution to making contributer day attendees pay for and download a Windows upgrade since Hyper V is required for Docker on Windows, and it requires Windows Enterprise/Professional. Docker is not as inclusive or available as it appears, and it's being touted as a way to make contributing for new users easier, which isn't necessarily true.

Any solution would require a provisioning script to set up the database user and the relevant files, as well as perform checks that relevant tools are installed. This would seem to be a more portable approach that needs to be taken anyway regardless of what devops gets done.

[pento]

Thank you for your thoughts, @TJNowell. I wasn't aware of Docker's requirement for either Hyper V or Virtualbox when run on Windows, which certain changes the ease of use for a large portion of contributors.

One of the most valuable things that Docker has done is to popularise the "container" concept: small, disposable, task-oriented VMs that are designed to spin up quickly, perform a single task well, and can be cheaply reset. This encourages tasks to be performed in the environment they most make sense: if something needs disk performance (eg, watch-type tasks), it can be run on the host. If something requires specialised software or configuration (eg, a MySQL database), it can be run inside of a container.

This directly benefits the contributor experience: isolating the magic to little boxes that can't be easily broken (and can be easily reset in the event they are) reduces the cost of experimenting, knowing that you have a fast and reliable reset mechanism to fall back on. Drawing from my personal experience, VVV has the opposite experience: provisioning is a slow process, which discourages resetting, which makes experimenting a risky proposition.

Thinking about a feature set I would want to see in the "VM" part of this project, it'd probably look something like this:

• Lightweight. Download size should be small, startup time should be fast.
• Disposable. Upgrades shouldn't run internally in the VM, downloading a fresh copy should be a good experience, encouraging people to take risks, knowing that they have a simple safety net.
• Centrally manageable. We should be able to upgrade the VM images and make them available to download. As a contributor, I generally don't care what specific version of PHP is being used, just that it works. Ultimately, our contributor tool should be able to download and install updates in the background, preferably without the contributor even needing to be prompted.
• Swappable components. Occasionally, I do care about the version of PHP: if I could easily click a button to switch to PHP 5.2 to test something, I would be a very happy contributor. I don't want to have to spend forever hacking PHP 5.2 support into a modern Ubuntu image: it should just work.

@TJNowell: As our resident VVV expert, do you think that this is achievable with the Vagrant/Virtualbox stack? You mentioned that VVV3 should solve the provisioning time, do you have an idea of how fast provisioning would be? Instant? Seconds? Minutes?

VVV has a reputation for being fairly fragile, it's easy to break your development environment, and hard to fix it. Can we address this perception by making it more robust, and easier to reset?

It bears repeating that I have no attachment to any particular technology or service, when it comes to solving this problem. To take the Travis example, Travis is a service that was easy to get up and running, and it's served us well for years. But, should we find ourselves better served by a solution that requires switching to a different CI service (or even self hosting!), we can absolutely do that.

[TJNowell]

We already prebuild the box for contributor days, the script generates:

• a box with an already installed Ubuntu running Nginx/MariaDB etc so that no package installation happens on contributor day wifi
• A copy of a preprovisioned VVV folder so that it doesn't download WP
• A folder with all the installers needed for Windows and Mac users

The only thing contributor day USB drives need to do on provision is setup the DB, which happens automatically on the first vagrant up. For an end user, it's:

• run the installers
• copy and extract vvv.zip
• open the new folder in a command prompt and type vagrant up
• wait for it to finish

Most of the issue and time spent is telling users how to work it, they're unfamiliar with local dev environments, and might have strange issues. For example one users I helped had turned off scrollbars because she didn't like how they looked, and a user at a local user group had not set up trackpad drivers and couldn't right click or 2 finger scroll.

VVV 3 goes one step further. We're planning to provision a prebuilt box on a CI server so that the only thing that happens on the users computer is setting up the sites. No package installation or configuring, just a box download, and updates arriving by pulling in new boxes. This also eliminates the need for the vagrant-vbguest plugin, and allows the provisioning scripts to be dramatically cut down in size.

VVV 2.2.1 which is released either today or tomorrow also brings Vagrant 2.1 support, which removes the need for the vagrant triggers plugin. Bundling Vagrant hosts updater is also being considered, which

Lightweight. Download size should be small, startup time should be fast.

VVV 3 should provision faster than 2, the main determiner of time at the moment is conference wifi stability and the ability of users to follow instructions. Efforts to reduce the first issue, and eliminate the second are in progress ( e.g. an electron based installer that verifies you have the stuff needed installed, then does all the vagrant commands for you with a progress bar ​https://github.com/tomjn/vvv-cd-installer )

Disposable. Upgrades shouldn't run internally in the VM, downloading a fresh copy should be a good experience, encouraging people to take risks, knowing that they have a simple safety net.

Technically VVV 2 already has this, we backup the DB on halt/destroy, and restore the DB if it's lost, so a vagrant destroy && vagrant up --provision will recreate the VM fresh

Centrally manageable. We should be able to upgrade the VM images and make them available to download. As a contributor, I generally don't care what specific version of PHP is being used, just that it works. Ultimately, our contributor tool should be able to download and install updates in the background, preferably without the contributor even needing to be prompted.

VVV 3 accomplishes this, security updates, guest additions, are applied on the CI server. If they fail for whatever reason then an update isn't pushed so no end users are impacted.

Swappable components. Occasionally, I do care about the version of PHP: if I could easily click a button to switch to PHP 5.2 to test something, I would be a very happy contributor. I don't want to have to spend forever hacking PHP 5.2 support into a modern Ubuntu image: it should just work.

VVV lets you swap out PHP versions and install additional versions at the moment. We use the Ondrej packages, but adding in PHP 5.2 would be easy, utilities are literally a bash script in a folder that gets run on provisioning so you can install software that isn't a website.

​https://varyingvagrantvagrants.org/docs/en-US/adding-a-new-site/changing-php-version/

VVV has a reputation for being fairly fragile, it's easy to break your development environment, and hard to fix it. Can we address this perception by making it more robust, and easier to reset?

We made a lot of changes when VVV 2 came out that improved on this front, reducing the support burden of VVV has been a top goal, as well as lots of other things planned. VVV 3 means large swathes of the provisioning will happen elsewhere, and updates to Vagrant itself have eliminated some issues that were endemic several years ago.

For that reason VVV 2.2 requires Vagrant 2.1, which also ensures a more modern version of VirtualBox. Most issues these days are fixed by upgrading Vagrant and VirtualBox. The longer term hope is to have a point and click UI

---

As a side note, something the VVV project lacks is feedback. We made a lot of changes to try and improve things, such as putting a colourful splash logo with the version number and branch and links when you try to provision, or a dashboard that lists all the users, with a search box and tips/hints, or the rewrite of the docs, and sometimes we get helpful comments on those, mostly that the colours are pretty and welcoming

But on some fronts, there's very little feedback at all. E.g. Lorelei built a custom-site-template-develop site template, making WP core dev provisioning more robust, adding in features, and allowing you to have multiple dev sites installed rather than the single site the old template allowed. But we've have almost no feedback of any kind for it, and that makes it difficult to know what blips need to go on the radar. Coupled with backwards compatibility, it's not an easy task. My fear is people who are afraid to update and run multiple versions side by side, and so there's been an emphasis on reducing maintenance cost and making everything friendlier and easier

[nbachiyski]

If our end goal is to make it really easy, the same way WordPress made website create easy 😜 we should aim higher – installing docker/vvv/waiting for any installation/setup seems both slow and error-prone (I’ve been to my fair share of contributor days). Two, closely related ideas:

• ​https://janitor.technology/ – a project build by Mozilla to easy open-source project contributions. Everything is a click away, mostly aimed at desktop software (VNC), but might simplify web projects, too.

• ​https://c9.io/ – a web IDE now run by Amazon AWS (Janitor uses it).

This might require some more research and maybe even infrastructure on our side, but will save hundreds of hours and will probably have immense long-term effects 🚀

[Mte90]

I agree about Janitor, infact I tried to make a docker solution for that: ​https://github.com/Mte90/wordpress-janitor
I know the developer/creator of the project so he helped me but I have no time to work on that but as I can remember my image was complete (based on VVV).

[pento]

@nbachiyski: I'm absolutely in favour of supporting web-based development environments, as well as investigating their use for new contributors. They're a long way from being full solutions, however, for a few reasons:

• They require a continuous, reasonably stable internet connection.
• Adding config for popular editors is a good idea. Mandating the editor is not.
• Storing GitHub/SVN/whatever credentials in the cloud is... less than ideal. 🙃 Requiring re-authentication for every commit is a terrible UX.

There are other services that are more focussed on providing the shared development stack (eg, ​Koding), but they have their own set of issues.

So as to not get too far ahead of ourselves, the focus of this ticket is to create a development environment that:

• Fits existing uses. Primarily, core development, Gutenberg development (as a model for future core development), and can be easily setup at contributor days.
• Is built with future uses in mind. Either container or full VM configurations can be relatively easily translated to online development environments, or to testing environments like calypso.live.

[MikeSchinkel]

Hi everyone,

I'd like to throw a new contender into the ring; (currently named) ​WPLib Box ​0.17.0.

We originally decided to build it because of frustrations with VVV. The fact VVV was provisioned locally using Puppet, which often failed and required sysadmin level skills to debug meant that it was not a solution that developers with few sysadmin skills could trust. Instead we wanted a box that “just worked,” so we built one that was pre-provisioned. And as of 0.10.2 it was really solid. That took us about a year, though not full time.

Then we set our sites higher and decided to build a box that not only just worked but what capable of running any stack needed while still ensuring the most common use-cases were dead simple. And that lead us to planning the use of a set of orchestrated Docker containers.

But what we found over the past year is that Docker for Mac has significant limitations, such as ​only being able to use the host’s network on Docker for Mac. This issue alone makes a pure implementation of orchestrated Docker containers a non-starter, assuming you want to make it “just work” for the developer installing it.

That lead us back to using a Vagrant and VirtualBox with Docker containers. Thus the current version of WPLib Box is 0.16.2 (with 0.16.3 coming later this week, hopefully) and the current version is based on Ubunutu 14.04 LTS + Vagrant + VirtualBox + individual Docker containers inside.

But in reality Vagrant is very problematic, constantly breaking working boxes on updates and just persnickety when it comes to installation. Further, Ubuntu is really memory intensive.

So our plans are to release 0.17.0 in the near future, and it will be based on either Tiny Core Linux or Alpine, VirtualBox and ​many Docker containers. It will also come with both a Mac and a Windows installer where the goal is for it to be as easy to get working as any application you might download and install on your Mac or Windows computer.

Looking back, we worked on 0.16.0 for over a year and it was a huge release which changed the box from a LEMP VM to a VM designed to orchestrate Docker containers and it positions us for a relatively quick release for 0.17.0. In order to release sooner or later we will probably not implement all of our ​current backlog of issues tagged for 0.17.0 and instead implement a lot of them in 0.17.x.

Longer term, we even plan to get rid of VirtualBox, but that will require more than a short-term engineering effort.

Now on to the requirements mentioned by everyone in this thread:

Replying to omarreiss:

Download and boot

• the easiest possible development workflow: just download and boot.
• reliable: it should just work.

From day 1 that has been a mantra for WPLib Box, that is “just works.”

Integrating watch and build behavior

As we plan to introduce a build step to the development workflow, it becomes more important that building is not an afterthought but something that is integrated into the development workflow.

A key aspect of WPLIb Box is its extensible architecture. With that architecture we will be able to easily implement and incorporate the build solution from #43055.

WPLib’s extensibility features as based on two concepts: “Stacks” and “Installables.” Stacks allow implementing any conceivable set of services you need with the default one being “WordPress.” But there could be a “WordPress Develop” stack, a “Tide” stack, or hell, even a “Drupal” or Node Express” stack. Stacks can extend other stacks and WPLib Box will be able to run multiple stacks.

Stacks are comprised of one of more “Installables”, and an Installable can be anything that either a “Project” needs or that the Box needs globally. Further, Installables can a Docker container, a WordPress plugin, a Theme, a SQL data file, or even a collection of those things. or a collections of other Installables. An Installable can even be a Stack.

Stacks and/or Installables required by a Project get written to a project.json file that can be version controlled and shared so that when a team member, or another WordPress core contributor brings up a project they will automatically see the exact environment that was configured for them by the developer who committed the project.json file.

Note WPLib Box 0.17.0 will support multiple “Projects,” each being identified by a local domain, e.g. example.local (we don’t call is a “multi-site” box for hopefully obvious reasons.)

• fast: reduce build and boot times to a minimum.

You should download and try the current version to see for yourself. We’ve really made it to be very performant, and 0.17.0 should only improve that.

@pento noted that eventually he'd like this thing to be configurable and installable through a GUI in order to reduce the barrier of entry for new contributors to the absolute minimum.

Part of our architecture will be to have an Electron-based admin console. It will run based on the API we are building into the box. Assuming something very similar to Local by Flywheel.

Replying to youknowriad:

• Ideally, the integration and e2e tests (even unit tests since PHP unit tests require a full setup) should use the same environment

We are ​adding PHPUnit to the box by implementing a ​general-purpose PHP script container approach that will make adding PHP-script based solutions as easy as specifying the details in an installable.json file.

From that we should be able to create an Installable adds the integration and e2e test and then include them in the “WordPress Develop” stack.

• A dev environment even if it's hidden behind magical scripts should stay simple. When bugs happen developers of any level should be able to understand, debug and tweak how the environment works.

Saying this a different way, our goal with WPLib Box is that the common case is dead simple, and the edge cases are possible. We are focused on hiding Docker containers completely from everyone except those that want to interact with them and/or build their own Docker containers for WPLib Box.

We will be segmenting our UX to multiple levels:

1. End-user developer: All they will need to ever see and are the installer and Electron-based admin console.
2. Power-user developer: In addition to the end-user developer they will also be interested in the box command line tool that will work both in the box and also on the host. They will also be able to build simple installables such as a starting point for a website; e.g. WordPress core, a collection of plugins and a theme.
3. Front-end extension developer: Beyond end- and power-user developer they will be able to extend the Electron-based admin using Vue.js and the Box API.
4. Box extension developer: Beyond end- and power-user developer these people will be able to extend the box itself such as Installable containers and even Stacks, as well as adding commands to the box CLI.

From the above segmentation we should be able to directly support hiding all complexity and make the end-user developer’s use very simple.

Replying to Mte90:

One of the problems that has VVV right now is that require a lot of time on first start because of installation (and download of a lot of things) but the plan for version 3 (​https://github.com/Varying-Vagrant-Vagrants/VVV/issues/1469) with a box that contain everything and also the parallel for the provision will improve a lot this point.

The current WPLib Box (0.16.2) contains everything and boots in about 30 seconds. We expect the next major version (0.17.0) will boot in around 5 seconds.

The other problems that often at contributor day we have is Windows. Install virtualbox, vagrant, git on Windows is a big step because people do't know anything of them.

We attend to hide the installation of VirtualBox via our installers. We can also look into installing Git, or we can provide them Git access via box git … which could be wrapped in a batch or Powershell file as just git.

Often the problems on VVV is because there are different versions of powershell on the system (don't ask me why) and Vagrant or Virtualbox works with issue based on the version of ps.

For the most part this just won’t be an issue in 0.17.0 given our architecture. And our box command will be implemented in GoLang so that should resolve the cross platform issue.

With docker we will have more problems on Windows because we need a virtual machine so (virtualbox and git will be required in any case) so the only advantage that we don't have anymore the easy vagrant to use but docker (that is not so entry level).

Again, we are addressing all of this with an installer, a VM to contain and orchestrate the Docker containers, and a simple admin console to make things easy.

Sure, there is the VVV USB Generator that prepare all the executables for download (maybe also the latest image) but we need to think that at contributor day not everyone has an OS dev friendly for *nix stuff.

We will support Windows day one.

From the other side VVV has the cool part to be used also for not contribute and very easy with vagrant up/halt and db backup so usually people are interested in that then contribute.

I would like to have is so that when someone adopts WPLib Box because of how easy it is that we make it just as easy for them to contribute to WordPress by giving them a menu option to just get started.

Integrating watch and build behavior: As we plan to introduce a build step to the development workflow, it becomes more important that building is not an afterthought but something that is integrated into the development workflow.

We plan to add watching/building as a default installable for our WordPress stack.

We have another mantra that is

"If we can simplify things for the end user and make it run in the box, we should make it run in the box."

So our vision is to minimize the number of things developers have to (figure out how to) install and get working on their host machine, and that we instead deliver a default solution that "just works," but that is also configurable and extensible for their needs.

having access to a ​https://en.wikipedia.org/wiki/Command-line_interface (CLI) to run grunt watch and grunt build tasks will be an important part of the WordPress development workflow.

We will certainly have installables for Grunt and Gulp though we plan to have WebPack + NPM scripts be the default build solution for WordPress (and WordPress Develop.)

• WordPress' CLI commands and scripts will need to be compatible with Mac, Windows, and *nix operating systems

The box CLI will be compatible on Mac, Windows and *nix, and the build systems will be able to run in the box so the scripts we use will only need to run within our purpose-built containers (which is the whole benefit of containers, right?)

• ​Composer is required as this was recently added in #43558 for ​PHPCS support

We already include support for ​several versions of Composer in WPLib Box 0.16.2.

• Support multiple versions of MySQL and MariaDB for testing #30462 #42811

We already include support for ​multiple versions of MySQL and ​multiple versions of MariaDB in WPLib Box 0.16.2.

• Support multiple versions of PHPUnit for testing (Legacy PHP versions required legacy PHPUnit versions) #43218

As noted, we have plans to ​support all necessary versions of PHPUnit in 0.17.0.

• Support multiple versions of PHP for testing,

​Check, for 0.17.0.

at the moment Gutenberg is using ​these scripts to manually install PHP 5.2 & 5.3 versions for Travis CI though they are quite fragile.

I don't fully understand what you are asking for, but I expect we can easily address it given our architecture.

Replying to TJNowell:

Any solution would require a provisioning script to set up the database user and the relevant files, as well as perform checks that relevant tools are installed. This would seem to be a more portable approach that needs to be taken anyway regardless of what devops gets done.

WPLib Box currently provisions the database on first load, and a package of Installables should support all of this.

Replying to pento:

One of the most valuable things that Docker has done is to popularise the "container" concept: small, disposable, task-oriented VMs that are designed to spin up quickly, perform a single task well, and can be cheaply reset. This encourages tasks to be performed in the environment they most make sense: if something needs disk performance (eg, watch-type tasks), it can be run on the host. If something requires specialised software or configuration (eg, a MySQL database), it can be run inside of a container.
This directly benefits the contributor experience: isolating the magic to little boxes that can't be easily broken (and can be easily reset in the event they are) reduces the cost of experimenting, knowing that you have a fast and reliable reset mechanism to fall back on. Drawing from my personal experience,

I cannot agree more. That is exactly why we chose to architect WPLib Box as a collection of orchestrated containers rather than try and create one big behemoth Docker container containing everything.

VVV has the opposite experience: provisioning is a slow process, which discourages resetting, which makes experimenting a risky proposition.

Yet again, agreed. That opposite experience is exactly why we first started to build WPLib Box.

• Lightweight.

Check. See above.

• Disposable.

WPLib Box is currently designed to be disposable. In 0.17.0 we plan to continue that, if for no other reason than that will make upgrades trivial.

• Centrally manageable.
• Swappable components (such as for PHP)

Check. We will be managing containers centrally and will have the open to auto-update; currently you can run box self-update in version 0.16.2 and it does essentially that. And swapping is just changing the name of the PHP container in the projects.json file

Further, given our architecture the WordPress core team could manage their own "WordPress Develop" Stack (basically a list of Installable containers and other installables) and those containers could either extend ours (to make it quick and simple to get the this Stack started) or they could replace ours if the core team's needs are more urgent than we can respond and/or if the core team needs something that makes no sense to include in our standard WPLib Box containers.

Replying to nbachiyski:

If our end goal is to make it really easy, the same way WordPress made website create easy 😜 we should aim higher

Absolutely. WPLib Box’s core mantra is “It just works” and everything we do is filtered with the lens of getting us there by version 1.0

• ​https://c9.io/ – a web IDE now run by Amazon AWS (Janitor uses it).

I really hope we don’t decide to go there. I have tried using C9, and it cannot compare to a solid local experience with PHPStorm+XDEBUG. I don’t know about others, but if contributing to WordPress required using c9, I would never even consider it.

That said, if c9 were an optional approach for newbies, that might make sense. More on this "optional" idea below.

Replying to pento:

web-based development environments require a continuous, reasonably stable internet connection.

Absolutely. A local solution that requires no internet connection is ideal. Which is how we've configured WPLib Box.

• Adding config for popular editors is a good idea. Mandating the editor is not.

Plus+1 for a WPLib Box Installable.

Storing GitHub/SVN/whatever credentials in the cloud is... less than ideal. 🙃 Requiring re-authentication for every commit is a terrible UX.

Agreed. SSH passthru from host is better. Auto-managing SSH in the box for those who do not already have set up is an even more ideal approach (which we want to implement with a Installable container.)

• Fits existing uses. Primarily, core development, Gutenberg development (as a model for future core development), and can be easily setup at contributor days.

Check: 0.16.2

• Is built with future uses in mind. Either container or full VM configurations can be relatively easily translated to online development environments, or to testing environments like calypso.live.

Check: 0.17.0

All that said, if you made it to this point I greatly appreciate it.

What I do notice on this ticket is there are now three (3) solutions vying for adoption: VVV, some abstract Docker solution, and now WPLib Box.

So my proposal is this: Rather than make a hard and fast decision on one solution, why not leverage the competition inherent in the market and instead establish a set of requirements that any recommended solution will need to provide, and then provide a recommended list of solution based on which one meets all (or most) of those objectives? That way you won't be shutting the door on a solution that might end up being significantly better simply by agreeing to choose one early one?

And if the core team later chooses to adopt just one as the "official" version then you will have the option of picking the one that evolved to be the best and not just the one that was anointed early on?

[jeremyfelt]

Replying to MikeSchinkel:

The fact VVV was provisioned locally using Puppet

Never with Puppet, though we did give it a try on a branch 5 years ago. :)

[MikeSchinkel]

Replying to jeremyfelt:

Replying to MikeSchinkel:

The fact VVV was provisioned locally using Puppet

Never with Puppet, though we did give it a try on a branch 5 years ago. :)

Weird, maybe I am confusing VIP QuickStart. That said, I found the 1000+ line Bash provision script almost as problematic.

[TJNowell]

Weird, maybe I am confusing VIP QuickStart.

That's correct, VIP Quickstart provisioned via a puppet setup, though VIP Quickstart is quite dead

As a sidenote:

• VVV 2 is intended to be disposable, you configure what you want and provision, and a VM gets created, with the intention that you can throw the VM away and recreate it at any time
• We have Jenkins pre-building base boxes with Ubuntu 14, and are in the testing phase prior to merge
• VVV 2.2.1 now only requires a single vagrant plugin, and updates to the Contributor day USB generator now bundle that plugin with the USB drive, eliminating all network dependencies
• It seems Pento has his own solution that installs Node and grunt watchers on the host then fires up a docker based WP setup via an electron app ( ​https://github.com/pento/wordpress-development-environment ). The irony being that the code could be simplified by swapping the commands for vagrant commands as only vagrant up, provision, and halt would be necessary

I appreciate the desire to make getting started with contribution easier, but it's clear the existing projects for local dev already have this in mind as VVV demonstrates, and as Mike indicated with new versions of WP Lib Box.

What we have here is a conflation of the Travis + PHP 5.2 problem, and ​this Xkcd comic. I don't expect that any contender will get picked here, either things will remain as they are, or something new will be built then used by a minority of people, for which we'll have a new support and maintenance burden.

In the meantime, consider provisioning a more recent build of VVV 2 with Vagrant 2.1, VVV is a very different beast internally these days, and a lot of issues people remember were resolved years ago

[jeremyfelt]

From the original ticket description:

• the easiest possible development workflow: just download and boot.
• fast: reduce build and boot times to a minimum.
• reliable: it should just work.

and

@pento noted that eventually he'd like this thing to be configurable and installable through a GUI in order to reduce the barrier of entry for new contributors to the absolute minimum.

I think these are the right objectives for the future. I don't think we can consider these complete while also requiring contributors to install Vagrant, VirtualBox, Docker, NPM, PHPUnit, and other command line tools to boot the box and get started.

I think this happens when somebody creates a native app with a GUI that may run Docker or something else in the background, but shows no sign of that on the front. If we know people interested in tackling native app experiences in OSX and Windows, that would be super interesting.

My (maybe slightly pessimistic) expectation of other solutions—new Vagrant VM, new Docker config, etc…—is that they'll all follow a similar path to VVV adoption over the last several years. More and more people use it, more and more people find bugs, a year or two later we have a bunch of the bugs figured out but it's still a pain for people to use in Windows.

In the meantime, those who are frustrated with the new approach will filter off and install other tools that work for them (MAMP, Local, WPLib Box, etc…).

One benefit that VVV has is that we've already ran into so many bugs at so many contributor days. And, as @TJNowell mentioned, VVV 3 will deliver a pre-provisioned Vagrant box that requires no time consuming process other than an initial download and future vagrant box updates. We've reached that point as a result of years of investment. We probably could have reached it sooner, but yeah - provisioning with bash scripts is ugly. :)

Aside: I ​really wanted this as part of core 5 years ago, but maintaining an open source dev environment project has altered my view quite a bit.

And that all said, whichever route we go, I believe it is even more important that we continue to improve our documentation around contributing code. What the build process is, how to manage basic tasks with NPM, Grunt, Git, etc... I think that will have a larger impact overall.

[spacedmonkey]

Fixed in #47767