WordPress.org

Make WordPress Core

Opened 13 days ago

Last modified 6 days ago

#43055 new enhancement

Reorganize Core JS / introduce build step

Reported by: omarreiss Owned by:
Milestone: 5.0 Priority: normal
Severity: normal Version: trunk
Component: Build/Test Tools Keywords: dev-feedback
Focuses: javascript Cc:

Description

This ticket aims to introduce a build step and better code management to WordPress core JavaScript. The current way the JavaScript is organized raises some concerns and needs an update to bring it up to par with current day processes for JS development. The last few months I've been busy figuring out what the concerns are with regard to JS code organization in WordPress, what a good process for modernizing it would be and what we need to change in order to facilitate such a process and be ready to include ie. Gutenberg into WordPress. In this project I've consulted with Gary Pendergast, Adam Silverstein, Riad Benguella and Andrew Duthie in #core-js and on WCUS and Anton Timmermans and Herre Groen at Yoast HQ.

A build step for WordPress core developent, what would change?

This patch will introduce a build step to WordPress core development. Once this gets committed, it will no longer be possible to run WordPress from the src directory. To make this transition easier for developers, we've added a special page that people running WordPress from source will start seeing:

https://user-images.githubusercontent.com/1488816/34698565-0a2cf5d0-f4d9-11e7-981f-6f64b15a0c79.png

As the screen says, npm install && grunt build will be enough to trigger a build. The JavaScript in the source has been organized to meet the following concerns.

Code organization concerns

1. Need to maintain backwards compatibility while providing flexibility.

We've discussed we want all enqueued scripts to still be built and present in the respective wp-admin/js and wp-includes/js. As they are there right now, they are really hard to manage. The way the code is organized is very hard to understand and it is very hard to refactor anything. To break out of the current "mess", it would be nice to have all JS source in one directory, from which it is copied or bundled to its legacy location. It's fully backwards compatible but adds an important first layer of flexibility that'll help advance JS development in core.

2. Need to separate scripts that end up in WordPress from source code and provide freedom in working with the source and organizing it in the best way.

Now that we have all the JS together in one directory, it will still be a mess to change things or split things up. It would not be transparent what would end up being enqueued and what is the source for those enqueues. In an ideal world, enqueued scripts are nothing more than a sort of manifests that load source code, initialize objects and perhaps set a few hooks. They would ideally not contain actual behavior. As we currently don't have a way to distinguish between enqueues and source code, basically everything right now is an enqueue. With the media library we've tried to demonstrate how this could work when you separate the source away. The media library then doesn't assign itself to the wp global (see src/js/media/models.js example). Instead this is then done in the enqueue (src/js/_enqueues/wp/media/models.js example). In fact It's practically all the enqueued script does.

3. Need to organize the enqueued scripts in a better way.

In the current setup, vendor scripts are mixed with self contained utility or lib scripts, deprecated scripts that are no longer in use and "other" scripts that are so all over the place that they are hard to classify. It's very hard to understand what's going on from looking at the directory structure. A large part of scripts simply assign something to the wp namespace. Why not have a directory structure for scripts which just follows the namespace? It also immediately becomes clear what should be the responsibility of those files, namely to assign modules to the namespace. The modules themselves could be moved to the src/js root and split up easily if deemed useful. This way it will be easy to explain the upgrade path towards a more modern code base. Things can simply be extracted / abstracted in multiple iterations.

Not every script assigns something to the wp namespace. We've come up with a directory structure that seems useful to us, but anything is possible really. A very nice advantage is that deprecated and vendor scripts become much easier identifiable. This what it could look like:

https://user-images.githubusercontent.com/1488816/34159855-fa860c66-e4ca-11e7-82f4-aace92ce1918.png

4. Need to manage vendor scripts better

In the current setup. Vendor scripts that exist as NPM packages are actually shipped with WordPress and managed with a sort of copy/paste system. Managing these with NPM and having a copy task or Webpack move them into the right place makes so much more sense. Most of the NPM packages simply ship with the dist files, so we only have to copy them to the right location. But now we actually manage those with NPM and they are listed in the package.json, which to me seems like a huge improvement.

Sometimes vendor scripts need to be built after install. This holds true for maybe 2 or three dependencies. This can simply be part of the build task.

Some vendor scripts cannot be managed with NPM. I've moved them to a separate src/js/_enqueues/vendor directory and added a copy task to the Gruntfile to copy these files over to their original location.

https://user-images.githubusercontent.com/1488816/34160143-044282e2-e4cc-11e7-9b06-8868784f8237.png

5. Need to install WordPress packages and assign them to a global.

WordPress Packages is an initiative to extract general purpose WP libraries to separate NPM packages, so they can also be used outside of the scope of WordPress. The first example of this is wp-includes/wp-a11y.js. It's easy to configure Webpack to automatically generate that file based on the @wordpress/a11y package. This was the chore that lead to this undertaking. Technically it's not a very hard thing to do, but I'd like to be as forward compatible as possible, so that's why I decided to zoom out a bit.

Webpack simply provides the library option for this kind of thing.

entry: {
        'src/wp-includes/js/wp-a11y.js': ['@wordpress/a11y']
},
output: {
        filename: 'src/wp-includes/js/wp-a11y.js',
        library: [ 'wp', 'a11y', 'speak' ],
}

6. Allowing a build step.

At WCUS, we pretty much agreed a build step is a necessity in modern JS development. While we can do much to help onboard new developers to get used to this, it simply makes sense to run WP from the build directory, no longer directly from source. We definitely intend to create good documentation for this that is also referenced in the notice showed earlier.

7. Managing packages under WP source.

At WCUS, it was pretty much decided WP packages should be managed in WP core after all. This setup can easily facilitate that. I'd propose to simply have a packages directory in src/js where we can continue managing them with lerna. This will also make it incredibly easy to publish packages from WordPress core to NPM, which could help us increase our impact in open source even further.


Testing

I ran a diff to compare the build that this thing generates with the current build on master. There are still a few minor differences:

➜  wp diff -qr compare/build mirror/build
Files compare/build/wp-admin/css/common-rtl.css and mirror/build/wp-admin/css/common-rtl.css differ
Files compare/build/wp-admin/css/common-rtl.min.css and mirror/build/wp-admin/css/common-rtl.min.css differ
Files compare/build/wp-includes/js/backbone.min.js and mirror/build/wp-includes/js/backbone.min.js differ
Files compare/build/wp-includes/js/hoverIntent.js and mirror/build/wp-includes/js/hoverIntent.js differ
Files compare/build/wp-includes/js/hoverIntent.min.js and mirror/build/wp-includes/js/hoverIntent.min.js differ
Files compare/build/wp-includes/js/imagesloaded.min.js and mirror/build/wp-includes/js/imagesloaded.min.js differ
Files compare/build/wp-includes/js/jquery/jquery-migrate.js and mirror/build/wp-includes/js/jquery/jquery-migrate.js differ
Files compare/build/wp-includes/js/jquery/jquery.color.min.js and mirror/build/wp-includes/js/jquery/jquery.color.min.js differ
Files compare/build/wp-includes/js/jquery/jquery.js and mirror/build/wp-includes/js/jquery/jquery.js differ
Files compare/build/wp-includes/js/masonry.min.js and mirror/build/wp-includes/js/masonry.min.js differ
Files compare/build/wp-includes/js/media-audiovideo.js and mirror/build/wp-includes/js/media-audiovideo.js differ
Files compare/build/wp-includes/js/media-audiovideo.min.js and mirror/build/wp-includes/js/media-audiovideo.min.js differ
Files compare/build/wp-includes/js/media-grid.js and mirror/build/wp-includes/js/media-grid.js differ
Files compare/build/wp-includes/js/media-grid.min.js and mirror/build/wp-includes/js/media-grid.min.js differ
Files compare/build/wp-includes/js/media-models.js and mirror/build/wp-includes/js/media-models.js differ
Files compare/build/wp-includes/js/media-models.min.js and mirror/build/wp-includes/js/media-models.min.js differ
Files compare/build/wp-includes/js/media-views.js and mirror/build/wp-includes/js/media-views.js differ
Files compare/build/wp-includes/js/media-views.min.js and mirror/build/wp-includes/js/media-views.min.js differ
Files compare/build/wp-includes/js/twemoji.js and mirror/build/wp-includes/js/twemoji.js differ
Files compare/build/wp-includes/js/underscore.min.js and mirror/build/wp-includes/js/underscore.min.js differ
Files compare/build/wp-includes/js/zxcvbn.min.js and mirror/build/wp-includes/js/zxcvbn.min.js differ
Files compare/build/wp-includes/version.php and mirror/build/wp-includes/version.php differ

For most of those there is a valid explanation. The media files have been slightly altered to fit the new setup. Some files are now copied over from node_modules and don't end in a newline. I still have to do a little bit of investigation here.

Attachments (1)

js-organization.patch (2.6 MB) - added by omarreiss 13 days ago.

Change History (16)

#1 @omarreiss
13 days ago

If you wish to checkout the branch for yourself and play around with it, this is what I based the patch on: https://github.com/Yoast/wordpress-develop-mirror/tree/fresh-js-src-move

#2 @netweb
13 days ago

  • Keywords dev-feedback added

#3 @omarreiss
13 days ago

This ticket was discussed today in #core-js on WordPress Slack, see https://wordpress.slack.com/archives/C5UNMSU4R/p1515505132000043

The main concern that was raised was the transition problems that might occur in not being able to run WordPress from the src directory anymore. I took 3 options from the :

1) Stop allowing running from /src and only build to /build.
2) Allow running from /src after a build step but .gitignore the built files.
3) Allow running from /src and commit the latest stable version of the built files in /src.

The general consensus seems to be that we need to have a transition period in which we start out with option 3 and after some time transition to option 2 and/or option 1. I will do another iteration to make this possible.

I would personally be in favor of gitignoring the built files from the start, because of the confusion that comes with having a set version of build files in the source. This can lead to a myriad of issues that I'd like to avoid. If we choose option 2 as the initial approach, the configuration / dev workflow impact would be minimal, while the gain would still be complete.

Last edited 13 days ago by omarreiss (previous) (diff)

#4 @omarreiss
13 days ago

To share a little more context. Here's an overview of the directory structure I've devised and the idea behind it.

src/js				| All the JavaScript source.
├── _enqueues			| Any script that ends up being enqueued.
│   ├── admin 			| Procedural scripts ran in the admin.
│   ├── deprecated		| All deprecated scripts.
│   ├── lib			| All standalone lib scripts.
│   ├── vendor			| All 3rd party deps that can't be managed with NPM.
│   └── wp			| All scripts that assign something to the wp namespace.
│       ├── customize		| Anything under wp.customize.
│       ├── editor		| Anything under wp.editor.
│       ├── media		| Anything under wp.media.
│       ├── utils		| Anything under wp.utils.
│       └── widgets		| jQuery widgets on the wp namespace.
└── media			| The media library.

As I see it, most of the future source will live outside of the _enqueues directory. One of the things this would enable us to do is to extract all of the business logic from the customizer into a modular library. This is something that can be done very easily in this structure and will make it much easier for us to get started once we move to the customizer / Gutenberg focus. The only thing that would remain in the _enqueues would be things like global assignments. An alternative approach is to not have these at all anymore and just configure Webpack to do that, but that is something we can also decide in a later stage. The important thing is that this structure makes it possible.

#5 @netweb
13 days ago

  • Component changed from General to Build/Test Tools
  • Milestone changed from Awaiting Review to 5.0

Moving to 5.0 for higher visibility

#6 @jorbin
12 days ago

Thanks for working on this @omarreiss! I fully support us doing this, but have a handful of thoughts and questions before I think it should be committed to core. I still need to review the actual code, but here are some thoughts on the idea and process for proceeding.

1) We should add file mapping to grunt-patch-wordpress so that we don't invalidate every single patch on trac that has JS in it.

2) Should we also modify the organization of qunit tests? The standard that we have used there is for test file names to match the path and name of files being tested.

3) We should come up with a communication plan for this. I think something along the lines of:

  • announcing it in dev chat the week this will land
  • Have a make/core post ready to go live within a few minutes of this landing in core

4) We should have some documentation about how to patch libraries that we load and also how to test those patches. npm link can be a new barrier to testing changes for many developers.

5) If we start relying on more and more external JS, I would love to come up with some philosophy points around it. In general, we should be relying on projects that have a strong commitment to backward compatibility.

6) as far as transitioning, I lean towards going all in and communicating it loud. This may also require working with maintainers of things such as VVV and other common dev platforms so that they are ready for the change.

#7 @omarreiss
12 days ago

@jorbin thanks a lot for these points!

1) Absolutely, will look into it!
2) Yes, was planning on this, but forgot about it again. I think we should. Will do the work.
3) I can definitely prepare a make/core post based on everything I've written up so far. I think @pento or @adamsilverstein would be likely candidates to eventually commit this.
4) Absolutely, I will work on docs explaining the process for managing dependencies, the build process, the directory structure and a process for extracting code from existing _enqueues scripts for reuse (some kind of boyscout principle). Then I think it would be good to have a README.md in src/js referencing this documentation.
5) I can add a draft for this to the dependency management documentation. We can always figure out later where this should end up eventually.
6) I would be in favor of that. When I discussed this with @pento he was also thinking along those lines. The most compelling argument I read against only running from build is that PHP changes will also need to be built in order to take effect. I can imagine that to be an undesireable barrier. There is also something to say for /build not really being a development build, but more of a distribution artifact. To me it seems like a good balance to allow building the JS to src as well but just adding the JS build dirs to .gitignore so that they don't get included in the source. That's still pretty much all-in if you ask me and makes the impact on dev platforms like VVV minimal. They basically only need to add npm install && grunt build to their provision scripts. In other words: maximum gain with almost zero cost.

#8 @pento
12 days ago

Nice work on this, @omarreiss. You've done a great job on this reorg, I think it has the potential to set WordPress up for a much nicer experience for contributors.

I'd like to chat about why I think this change is a a good idea.

The Problems

I've been reflecting on this issue since our discussions at WCUS, and I think it comes down to three issues that we're facing.

  1. The experience of contributing to WordPress is awful. There's a high barrier of entry just to get some sort of a local development install running, it takes way too much effort to maintain, and it's really hard to collaborate. Even if you manage to produce a patch, it feels like you're throwing it into the void.
  1. Adding a build step to the WordPress development flow is unavoidable. As we move towards our JS future, it's going to be necessary to make that move, Gutenberg is proving to be a useful testing ground for playing with build processes.
  1. JS build tools are nearly as bad as WordPress, in terms of the ease of someone being able to contribute to a JS project. They certainly don't tackle any of the hard problems, like setting up complete local dev environment. (If you're talking to an API to store your data, and not setting up a local copy of that API server and corresponding database, your dev environment is not complete.)

The Solutions

With those problems in mind, I think there are a few promising options to solve them.

On the topic of where to build things to, I find myself leaning towards option 1, though I can't entirely dismiss option 2. Option 3 is completely out, I don't believe there's anything but pain and maintenance headaches down the road of committing built files to the source repo. Option 2 appeals, as the idea of allowing PHP changes without a build step is nice. That said, I also think it opens us up to a potentially confusing development experience, as we'll have committable PHP files living next to uncommittable JS files. Option 1 requires a build step for PHP changes, but it gives us the cleanest separation between development code and built code, even if the PHP is never really "built", as such.

I have a love/hate relationship with VVV. I love it when it works, I hate whenever I have to setup a new computer, because the VVV part is so often the most painful. I believe we can do a lot better than that, and a lot better than what the current JS process looks like. Gutenberg currently makes limited use of Docker to provide a simple testing environment, I've been playing around with expanding that to include running PHPUnit tests, and generally having a full development environment created automatically. That PR is obviously a little rough around the edges, but I think it points in a good direction - we can easily automate setting up the entire development environment, and it's not difficult for us to detect when something is wrong, and give useful, contextual information for a contributor to get their development environment running again.

Finally, (and this is more of a long term goal), I would be exceedingly happy if my development experience never involved opening a terminal window. App IDEs have been working on making their development experience not suck for many years, and they're a long way ahead of where the JS world is now. But there's a lot we can learn from them, and I think one of the most important lessons is that developing entirely in a GUI is an inherently better experience than typing a command into a terminal.

What's Next?

Some of the stuff above is beyond this ticket, but it gives some context for why I think this is a necessary change, that on the face of it adds some complexity to the WordPress development experience, but actually sets us up for making the experience awesome.

As @jorbin has mentioned, there are certainly some housekeeping issues to sort out before this can be committed. I agree that a make/core post is important, I'd be inclined to post it before the change is committed, not only so that people are prepared, but so we can get a wider audience on the proposal. This is a big change for WordPress development, it's appropriate that we should be able to make a convincing argument to the wider WordPress community before it lands.

If build tools are relevant to your interest, I would very much like to see more activity on the Gutenberg tools. WordPress' future tools will be heavily influenced by what works in Gutenberg, so get in early if you want to shape what that's going to look like.

#9 @jeremyfelt
12 days ago

This is great work so far @omarreiss, thank you.

2) Allow running from /src after a build step but .gitignore the built files.

This seems like the option least likely to impact existing workflows. If you work on JS, you'll already be used to the existing process and have likely used grunt watch as part of your workflow. If you work on PHP, then nothing really changes.

1) Stop allowing running from /src and only build to /build

That said, option 1 is much cleaner and I don't think it will be too impactful to users of VVV if we switch to a "must build" workflow. We already have a build.wordpress-develop.test that's intended to use with the build/ directory and that is built with grunt by default during provisioning. We'll probably need to dust that off and make sure the docs are good.

Personally - as long as the builds triggered by grunt watch (or whatever is doing the instant builds) are quick, especially when working on files that don't require a build, then +1.

This ticket was mentioned in Slack in #core by jorbin. View the logs.


12 days ago

#11 @dingo_bastard
11 days ago

I never had any issues with writing patches or testing stuff on VVV, and adding one small extra command line is not that complicated.

Somehow I get the feeling that we are worried that people will be thrown off of developing or contributing because they need to use the command line (which I've grown quite fond of). Even Windows has bash built into it (granted only win10 but still) :D
We should be looking at this as an opportunity to educate people how to use new and exciting tools. Sure, npm can be fidgety, sometimes some js projects work with one version of npm & node, other with different one. But I've had same experience with ruby (where one project works on 1.9.1 and not on 2.2.2 etc), and the development on those projects when compared to WordPress is just scary awful (the builds fall apart because hey, somebody removed a gem that existed n years ago and was used in the project). With WordPress I didn't have that experience. And I've learned a lot since I've started to work with it (and still do).

Point I'm trying to make is that we should use this opportunity to create a good documentation and show people that it's not hard to use these tools. I mean everybody in JS world is using either yarn or npm. WordPress is often the first touch people have with development (usually when they have to modify themes). If we show them that they can learn to use modern tools that might inspire them to learn more by themselves and be a better developer.

#12 @gziolo
10 days ago

As the screen says, npm install && grunt build will be enough to trigger a build. The JavaScript in the source has been organized to meet the following concerns.

It can be as simple as npm install. You just need to put grunt build as postinstall npm run-script. See how it is configured in WordPress/packages: https://github.com/WordPress/packages/blob/master/package.json#L35.

#13 @gziolo
10 days ago

Bonus question, would it be a very bad idea to call npm install from PHP using shell_exec function when Node.js is installed on the machine?

Last edited 10 days ago by gziolo (previous) (diff)

#14 @Frank Klein
7 days ago

Thanks for working on this patch @omarreiss. The proposed reorganisation makes sense, and I look forward to seeing this land in Core.

On the subject of the ease of contribution, I consider that this can be solved by creating better documentation. Web development has become more complex, and there's no way to get rid of that complexity, as it is there for a reason. Let's not remain stuck in the "good old days" of "hacking" on WordPress using MAMP.

As far as Gutenberg is concerned, I do not necessarily see it as being the determining factor for how Core should handle JavaScript. There is no technical requirement for having to keep the build system that Gutenberg uses. And considering the small numbers of Gutenberg contributors, familiarity with the plugin's build process is not an argument.

This ticket was mentioned in Slack in #core-js by aduth. View the logs.


6 days ago

Note: See TracTickets for help on using tickets.