Keep `@wordpress` packages up to date

[atimmer]

For the 5.0 release we have to make sure we have the latest version of the packages that needs to be included in 5.0. I will be regularly updating the packages, this ticket will track those commits.

[atimmer]

In 43789:

Build tools: Upgrade @wordpress packages.

Generated using npm update @wordpress/api-fetch @wordpress/blob @wordpress/block-library @wordpress/blocks @wordpress/components @wordpress/compose @wordpress/core-data @wordpress/data @wordpress/date @wordpress/dom @wordpress/edit-post @wordpress/editor @wordpress/element @wordpress/escape-html @wordpress/list-reusable-blocks @wordpress/nux @wordpress/plugins @wordpress/redux-routine @wordpress/rich-text @wordpress/url @wordpress/viewport.

See #45145.

[pento]

@atimmer: You need to run grunt webpack:dev after upgrading these packages, particularly block-library, which copies the PHP files for core dynamic blocks.

Also, remember to run unit tests after this, too, as I think one of the blocks changed their structure a little, which will break the block fixture tests.

[pento]

In 43801:

Build: Update the @wordpress/editor package.

The @wordpress/editor wasn't upgraded to the correct version in [43789].

This commit includes changes from the @wordpress/block-library upgrade in [43789], that weren't copied to the correct location in core.

This also regenerates the package-lock.json in Linux, which the build server requires.

See #45145.

[atimmer]

Didn't realize that the JavaScript packages could also contain PHP after [43751]. I'll keep that in mind.

[atimmer]

In 43824:

Build tools: Clone and build from packages repo.

Instead of requiring the packages individually this commit changes it so the packages are used from the packages repository (gutenberg). This is a solution that makes updating the packages easier, as long as we don't have a monorepo to manage both WordPress & the JavaScript packages together.

Props youknowriad, gziolo, omarreiss, herregroen.
See #45145.

[omarreiss]

Fix looks good to me!

[atimmer]

In 43826:

Build tools: Fix build after [43824].

Props omarreiss.
See #45145.

[ocean90]

Can we avoid installing the submodules and all the mobile stuff? Asking because it now requires to have yarn installed which is currently not available on w.org and thus broke the build.

[pento]

@ocean90: Are you sure it was yarn? That gets installed into packages/node_modules/.bin, the build should be using that yarn (that's what it does when I run it on my sandbox).

I think it's broken because it looks like the package-lock.json changes were generated on MacOS, instead of Linux.

[ocean90]

@pento Ah, maybe it’s that. There’s definitely going on too much during the npm install these days. :)

[omarreiss]

For clarity, I’ve signed off on @atimmer keeping the packages up-to-date. So any commit in that regard by him is fine by me.

[pento]

@atimmer: Please generate the package-lock.json file on Linux for this commit.

[jorbin]

With this method, how do we test packages before they are a part of a tagged Gutenberg release?

[azaozz]

Building of the 5.0 branch currently fails on Windows. Caused because now we download and build Gutenberg while building WP and the mobile module there fails.

Seems this has to be fixed "upstream", opened ​https://github.com/WordPress/gutenberg/issues/11144.

[azaozz]

Also, I may be missing something but shouldn't we be including pre-build (release) versions of the external packages when building WordPress? When updated packages (that have been tested and released "upstream") are available we can update them in package.json, even if that means we need to update it quite regularly at the moment.

This is the proper way to include external packages :)

[johnjamesjacoby]

Please generate ... on Linux

Is someone working on that? Building WordPress never required a specific operating system before, and shouldn’t now.

[netweb]

Replying to azaozz:

When updated packages (that have been tested and released "upstream") are available we can update them in package.json, even if that means we need to update it quite regularly at the moment.

This is the proper way to include external packages :)

Replying to johnjamesjacoby:

Is someone working on that? Building WordPress never required a specific operating system before, and shouldn’t now.

100% in agreement here, [43824] and [43826] should be reverted restoring the previous processes

[netweb]

Related: #45201 Exclude JavaScript source maps from build

The above also appears to be a side affect of [43284]

[pento]

In 43835:

Build Tools: Revert [43826] and [43824].

Grabbing the packages directly from the Gutenberg repo was an interesting experiment, but it causes significant workflow issues, particular for folks using Windows.

See #45145.

[pento]

Replying to johnjamesjacoby:

Is someone working on that? Building WordPress never required a specific operating system before, and shouldn’t now.

WordPress (in particular, the build server) has always required that package-lock.json be generated on Linux, the difference is that we used to only create that file right before release. 🙂

It needs to be fixed, but it's extremely low priority, when the workaround is for the occasional package.json updates to be done from a Linux box.

[johnjamesjacoby]

when the workaround is for the occasional package.json updates to be done from a Linux box

This raises the barrier to entry significantly for some folks, and I’m uncomfortable with that. This seems like a workflow regression, and has caused issues here already. How was this decided? What alternatives are there?

[pento]

Maybe @nacin will remember why the build server requires a special package-lock.json, I don't recall off the top of my head. I suspect it's just that it doesn't like when running npm install causes the package-lock.json file to change (in which case, the fix is pretty easy), but I haven't confirmed that.

The build server needs to be fixed, but there are only a handful of people with the ability to do that, and it's a low priority for all of them.

This doesn't really raise the barrier of entry at the moment, given that the only packages being updated are the @wordpress/ packages, which only a handful of people can do. It will raise the barrier if left unfixed, but no-one is arguing in favour of doing that.

[netweb]

Replying to pento:

WordPress (in particular, the build server) has always required that package-lock.json be generated on Linux, the difference is that we used to only create that file right before release. 🙂

Adding to this, it was only ever committed to the release branch and never /trunk.

Now that there is a package-lock.json file in the primary working branch this requires committers to ensure that the npm lock file is committed is a Linux safe state for the build server to not crash

See also: #45088 - Update package-lock.json for Mac, Linux, and Windows cross-platform compatibility

[netweb]

p.s. The build issue was originally raised in #38657, and fixed in [39368]:

Build: Remove fsevents from npm-shrinkwrap.json

fsevents is macOS only, and npm-shrinkwrap tries to force it onto other architectures, causing build failures. Upstream: ​https://github.com/npm/npm/issues/2679.

[pento]

45145.diff​ updates to the latest version of the @wordpress/ packages, and adds the new @wordpress/format-library and @wordpress/notices packages.

It currently causes a JS error: ​https://github.com/WordPress/gutenberg/pull/11122

[pento]

In 43840:

Build Tools: Upgrade @wordpress packages to the latest version.

This also includes the new @wordpress/format-library and @wordpress/notices packages.

package-lock.json has been completely regenerated in this commit.

See #45145.

[pento]

After [43840], 45145.4.diff​ are the changes I get when running npm install on MacOS. Applying that patch on Linux, then running npm install just reverts them.

[pento]

In 43841:

Tests: Fix failing unit tests caused by [43840].

See #45145.

[netweb]

Replying to pento:

After [43840], 45145.4.diff​ are the changes I get when running npm install on MacOS. Applying that patch on Linux, then running npm install just reverts them.

I got the same results in my testing, of interest though was the only changes were related to the https:// / https:// URLs, none of the optional changes occured this time /shrug

[azaozz]

@pento thanks for taking care of this. Can confirm building of the 5.0 branch works properly on Windows 10 with the default installations of git, npm, etc. no tweaks needed :)

[azaozz]

Replying to johnjamesjacoby:

This raises the barrier to entry significantly for some folks...

Not exactly :) This generally concerns only the committers. package-lock.json is an auto-generated file and currently there is a bug in npm that rewrites optional dependencies when generating it from different OS.

It is the committers' "job" to ensure we have the proper package-lock.json and not introduce OS related changes there. As @netweb mentioned above, this is tracked in #45088.

Replying to netweb:
Building on Win10 still added ~10 "optional": true apart from the http:// to https:// fixes. Can upload a diff to #45088 just as example.

[netweb]

Replying to azaozz:

Replying to netweb:
Building on Win10 still added ~10 "optional": true apart from the http:// to https:// fixes. Can upload a diff to #45088 just as example.

Thanks, no need for a sample diff 👌

[pento]

In 43843:

Build Tools: Update @wordpress packages.

See #45145.

[pento]

In 43847:

Build Tools: Update the @wordpress packages.

Trunk now matches Gutenberg 4.2 RC1.

See #45145.

[pento]

In 43861:

Build Tools: Update the @wordpress packages.

This brings support for the custom fields meta box to the block editor.

The webpack and copy-webpack-plugin packages have also been updated.

See #45145.
Fixes #45257.

[pento]

In 43884:

Block Editor: Update @wordpress dependencies to the latest version.

Changes of note:

• Includes the new Annotations API package.
• wp-polyfill-ecmascript.js is renamed to wp-polyfill.js.
• strip_dynamic_blocks() has been removed in favour of excerpt_remove_blocks().
• The PHP block parser is now syncing from the block-serialization-default-parser package.
• do_blocks() uses the new parser.
• The do_block filter has been removed from do_blocks(), in favour of a render_block filter in render_block().

See #45145, #45190, #45264, #45282.

[azaozz]

Looking at [43884], the render_block() in wp-includes/blocks.php in particular. Couple of nitpicks:

• Thinking the normalization of the block attrs should be done in all cases, not just for dynamic blocks (it's better for the 'render_block' filter).
• As we do is_array( $block['attrs'] ) we don't need to cast it to array right after.

See 45145-blocks.php.diff.

[pento]

Good catch, thanks @azaozz!

45145-blocks.php.diff​ is good to commit.

[azaozz]

In 43888:

Block Editor: A little cleanup to render_block(). Always normalize $block['attrs'] to array in 'render_block' filter.

See #45145.

[youknowriad]

Latest update to the WordPress packages (based on 4.3.0)

[youknowriad]

In 43891:

Block Editor: Update @wordpress dependencies to the latest version matching Gutenberg 4.3.0

• Update the editor,edit-post,components,block-library,nux and list-reusable blocks packages
• Also add a missing wp-date dependency to the editor script

Props atimmer
See #45145.

[desrosj]

Updates reflecting Gutenberg 4.4 release.

[desrosj]

45145.5.diff​ reflects the package updates in Gutenberg 4.4.0.

Installed the new versions and compiled and the new editor works great for me.

[aduth]

Packages update for Gutenberg plugin v4.4.0

[desrosj]

@aduth Ah, your patch also has the updates to the script-loader.php. I was working on a separate patch for PHP changes. Including them here makes sense though!

[aduth]

Whoops, guess we overlapped work!

My patch includes changes to script-loader.php. Notably this also includes the addition of a dependency between React and wp-polyfill. Heads-up to @atimmer since it was mentioned at ​the relevant pull request to be on your "list".

Some workflow shortcuts which helped here:

1. Use ​npm-check-updates to automate the version bumps
   • ncu '/^@wordpress\//' -a
2. Use git diff since the last plugin version tag to find differences in script-loader.php
   • git diff v4.3.0 master -U10 -- lib/client-assets.php

[aduth]

Worth pointing out that neither of these patches include the version bump for React 16.6.3 which was included in the v4.4.0 plugin:

​https://github.com/WordPress/gutenberg/pull/11840

I'm unsure if there's a more appropriate ticket for tracking these vendor dependencies, but raising as relevant to the 4.4.0 upgrade.

[pento]

Thanks for doing this update, @desrosj and @aduth!

packages-4-4-0.diff​ is looking good, lets bump the React version as well, though.

[aduth]

Packages update for Gutenberg plugin v4.4.0 (incl. React 16.6.3)

[pento]

Thanks @aduth! packages-4-4-0.2.diff​ is good to commit.

As a side note, the build server now ignores package-lock.json changes, so there's no need to use Linux to generate it.

[aduth]

In 43903:

Block Editor: Update @wordpress dependencies to match Gutenberg 4.4.0

Updates packages: annotations, api-fetch, block-library, blocks, components, compose, core-data, data, date, edit-post, editor, element, format-library, i18n, list-reusable-blocks, notices, nux, plugins, rich-text, viewport

Updates changed dependencies in script-loader.php

Upgrades React from 16.5.2 to 16.6.3

See #45145

[mcsf]

First pass at update for Gutenberg 4.5 / WordPress 5.0 RC

[mcsf]

45145.6.diff constitutes a first pass, including:

• An updated package.json thanks to the ncu tool as pointed out by Andrew in #comment:47.
• A manually updated dependency tree in script-loader.php built from Gutenberg's packages-dependencies.php.
• Manually merged changes from Gutenberg's client-assets.php.
• A fix in tools/webpack/packages.js based on ​gb-12006.

As it's the middle of the night here, it's time for me to pass the baton. Please properly check the diff, in particular where it concerns TinyMCE (the line for a new plugin, tinymce-latest-lists, was blindly copied to $vendor_scripts), and there's a discrepancy where it reads wp_adv (though it's probably Gutenberg the plugin which is wrong on this one).

[mcsf]

I realise that my previous comment may not be super clear on the fact that it is not a full diff of the PHP changes, so that still needs to be performed — just as an example, see git diff v4.4.0 master -U10 -- lib/compat.php

[noisysocks]

We're 90% there. I uploaded 45145.7.diff which continues on from 45145.6.diff and:

• Applies patches generated from git diff v4.4.0 master -- **/*.php to these files:
  • lib/class-wp-block-type.php
  • lib/class-wp-rest-block-renderer-controller.php
  • lib/class-wp-rest-blocks-controller.php
  • phpunit/class-block-type-test.php
  • phpunit/class-rest-block-renderer-controller-test.php
  • phpunit/class-rest-blocks-controller-test.php
• Includes the artefacts generated by running npm install
• Adds global $wp_locale to the top of wp_default_packages_vendor()
• Replaces the tinymce-latest-lists dependency with wp-tinymce-lists which is already added by wp_register_tinymce_scripts()

There are some unit test failures that will be fixed when we update block-library to include ​gutenberg#12149.

There is also still the wp_adv discrepancy mentioned by @mcsf to address.

[iseulde]

Ideally we use wp_adv in the plugin, but we couldn't because we use inline mode and the toolbars work differently. If wp_adv has been updated, then it would be fine for WordPress 5.0, but it still wouldn't work with WordPress 4.9.8 and the plugin.

[youknowriad]

Gutenberg 4.5.1 patch

[youknowriad]

Updated the patch

In addition to the last patch from Robert, this fixes an issue where the font styles were not loaded properly and updates the packages to the 4.5.1 version.

[azaozz]

there's a discrepancy where it reads wp_adv...

Right, as @iseulde points out that's not a discrepancy :)

Core uses the (old) wp_adv for the "toggle toolbars" button, Gutenberg still uses kitchensink. The TinyMCE wordpress plugin (that handles toggling of toolbars) was fixed to make wp_adv work properly in the Classic Block, but the Gutenberg's kitchensink was kept as a back-compat for running Gutenberg on 4.9 (where the core fix is not present).

We should switch to using wp_adv in Gutenberg as soon as it starts to require WP 5.0.

[iseulde]

Replaces the tinymce-latest-lists dependency with wp-tinymce-lists which is already added by wp_register_tinymce_scripts()

It might be worth to include ​https://github.com/WordPress/gutenberg/pull/12170 then.

[azaozz]

Right, seems that separately adding the lists TinyMCE plugin is not needed. May be missing something but the included lists plugin in core works as expected? Also couldn't find why it was added separately in Gutenberg. Lest remove tinymce-latest-lists.

[danielbachhuber]

In 43917:

REST API: Always include title.raw/content.raw for Blocks in context=view.

Demarcations for reusable blocks are always expected to be accessible by clients.

Props noisysocks, youknowriad.
See #45145 for the patch, #45098 for the original ticket.

[danielbachhuber]

In 43918:

REST API: Preserve unknown, respect null in server-side block rendering.

• Skips validation where there is no attribute definition, but keeps the attribute value. Previously, the attribute would be omitted from the attributes passed to render_callback. Notably, this resolves an issue where render_callback cannot receive a block's align and customClassName attribute values, since these are defined as a client-side filter.
• Validates null as a proper value in its own right. Previously, a client implementation of a block could track {"attribute":null} as an explicitly empty value, and the server would wrongly initiate defaulting behavior. The new behavior will now only populate a default value if the attribute is not defined at all, including when unset in its being invalid per the attribute schema.

Props aduth, noisysocks, youknowriad.
See #45145 for the patch, #45098 for the original ticket.

[danielbachhuber]

In 43919:

Block Editor: Update @WordPress dependencies to match Gutenberg 4.5.1.

Updates packages: annotations,
api-fetch,
block-library,
blocks,
components,
core-data,
data,
dom, edit-post,
editor,
format-library,
jest-console,
jest-preset-default,
keycodes,
html-entities,
list-reusable-blocks,
notices,
nux,
rich-text,
scripts,
token-list, url,
viewport

Fixes undefined notices in some blocks. Removes incorrect 'gutenberg' textdomain too.

See #45145

[azaozz]

In remove-wp-tinymce-lists.diff: remove separate enqueuing and loading of the lists TinyMCE plugin.

That plugin doesn't need to be added/loaded separately. In production we load wp-tinymce.js.gz, lists is already included there. When SCRIPT_DEBUG is true or when running from /src TinyMCE loads its plugins that are in wp-includes/js/tinymce/plugins dir. There is no need or any advantages to load lists separately.

[noisysocks]

In 43935:

Block Editor: Update @wordpress package dependencies

Updates packages:

• block-library
• components
• edit-post
• editor
• format-library
• list-reusable-blocks
• nux

See #45145

[pento]

In 45145.8.diff​:

Update packages:

• @wordpress/block-library@2.2.8
• @wordpress/components@7.0.3
• @wordpress/edit-post@3.1.3
• @wordpress/editor@9.0.3
• @wordpress/format-library@1.2.6
• @wordpress/list-reusable-blocks@1.1.16
• @wordpress/nux@3.0.4

Other changes:

• Fix a translator comment in edit-form-blocks.php
• Rename the gutenberg_ functions in blocks/latest-comments.php. This is a manual port of ​GB12326.

[youknowriad]

@pento I tested the patch and everything looks good to me.

[youknowriad]

In 43949:

Block Editor: Update @wordpress package dependencies.

Update packages:

• @wordpress/block-library@2.2.8
• @wordpress/components@7.0.3
• @wordpress/edit-post@3.1.3
• @wordpress/editor@9.0.3
• @wordpress/format-library@1.2.6
• @wordpress/list-reusable-blocks@1.1.16
• @wordpress/nux@3.0.4

Other changes:

Fix a translator comment in edit-form-blocks.php.
Rename the gutenberg_ functions in blocks/latest-comments.php.

Props pento.
Fixes #45145.

[youknowriad]

In 43950:

Block Editor: Rename the gutenberg_ functions in blocks/latest-comments.php.

This was ported manually from ​https://github.com/WordPress/gutenberg/pull/12326

Props pento, swissspidy.
See #45145.

[youknowriad]

In 43951:

Block Editor: Remove the latest "gutenberg" mention in the latest-comments.php file

Props swissspidy.
See #45145.

[youknowriad]

Reopening because we need a small package update to backport this PR ​https://github.com/WordPress/gutenberg/pull/12342

[youknowriad]

Updated packages

• @wordpress/block-library@2.2.9
  • @wordpress/block-serialization-default-parser@2.0.1
  • @wordpress/block-serialization-spec-parser@2.0.1
  • @wordpress/blocks@6.0.3
  • @wordpress/edit-post@3.1.4
  • @wordpress/editor@9.0.4
  • @wordpress/format-library@1.2.7

Other changes:

• Backport the parser changes.

[matveb]

Looks good.

[youknowriad]

In 43955:

Block Editor: Update @wordpress package dependencies.

Updated packages

• @wordpress/block-library@2.2.9
• @wordpress/block-serialization-default-parser@2.0.1
• @wordpress/block-serialization-spec-parser@2.0.1
• @wordpress/blocks@6.0.3
• @wordpress/edit-post@3.1.4
• @wordpress/editor@9.0.4
• @wordpress/format-library@1.2.7

Other changes:

• Backport the parser changes.

Fixes #45145.

[youknowriad]

Patch for the Gutenberg 4.7 release (aka WP 5.0.2)

[pento]

In 44163:

Build Tools: Update the @wordpress/* packages.

Merges [43801,43803] from the 5.0 branch to trunk.

See #45145.

[atimmer]

In 44177:

Build Tools: Upgrade @wordpress packages to the latest version.

This also includes the new @wordpress/format-library and @wordpress/notices packages.

package-lock.json has been completely regenerated in this commit.

Props pento.
Merges [43840] to trunk.
See #45145.

[atimmer]

In 44179:

Tests: Fix failing unit tests caused by [43840].

Props pento.
Merges [43841] to trunk.
See #45145.

[atimmer]

In 44180:

Build Tools: Update @wordpress packages.

Props pento.
Merges [43843] to trunk.
See #45145.

[atimmer]

In 44182:

Build Tools: Update the @wordpress packages.

Trunk now matches Gutenberg 4.2 RC1.

Props pento.
Merges [43847] to trunk.
See #45145.

[desrosj]

In 44260:

Editor: Add custom fields meta box support in the block editor.

This brings support for the custom fields meta box into the new block editor.

The webpack and copy-webpack-plugin packages have also been updated.

This does not bump the @wordpress packages like in [43861] because of conflicts with package versions already installed in trunk. The packages will be brought up to date in a subsequent merge.

Merges [43861] and [43863] into trunk.

See #45145.
Fixes #45257.

[desrosj]

In 44261:

Block Editor: Update @wordpress dependencies.

Changes of note:

• Includes the new Annotations API package.
• wp-polyfill-ecmascript.js is renamed to wp-polyfill.js.
• strip_dynamic_blocks() has been removed in favor of excerpt_remove_blocks().
• The PHP block parser is now syncing from the block-serialization-default-parser package.
• do_blocks() uses the new parser.
• The do_block filter has been removed from do_blocks(), in favor of a render_block filter in render_block().

Also, a little cleanup to render_block(). Always normalize $block['attrs’] to array in ’render_block’ filter.
Props pento, azaozz.

Merges [43884] and [43888] to trunk.

See #45145, #45190, #45264, #45282.

[desrosj]

In 44262:

Block Editor: Update @wordpress dependencies to match Gutenberg 4.5.1.

• Update the annotations, api-fetch, block-library, blocks, components, compose, core-data, data, date, dom, edit-post, editor, element, format-library, html-entities, i18n, jest-console, jest-preset-default, keycodes, list-reusable-blocks, notices, nux, plugins, rich-text, scripts, token-lists, url, viewport packages.
• Upgrades React from 16.5.2 to 16.6.3.
• Adds a missing wp-date dependency to the editor script.
• Updates changed dependencies in script-loader.php.
• Fixes undefined notices in some blocks.
• Removes incorrect gutenberg textdomain.

Merges [43891], [43903], and [43919] to trunk.

Props atimmer, aduth, youknowriad, danielbachhuber.
See #45145.

[desrosj]

In 44268:

REST API: Always include title.raw/content.raw for Blocks in context=view.

Demarcations for reusable blocks are always expected to be accessible by clients.

Props noisysocks, youknowriad.

Merges [43917] to trunk.

See #45145 for the patch, #45098 for the original ticket.

[desrosj]

In 44269:

REST API: Preserve unknown, respect null in server-side block rendering.

• Skips validation where there is no attribute definition, but keeps the attribute value. Previously, the attribute would be omitted from the attributes passed to render_callback. Notably, this resolves an issue where render_callback cannot receive a block's align and customClassName attribute values, since these are defined as a client-side filter.
• Validates null as a proper value in its own right. Previously, a client implementation of a block could track {“attribute":null} as an explicitly empty value, and the server would wrongly initiate defaulting behavior. The new behavior will now only populate a default value if the attribute is not defined at all, including when unset in its being invalid per the attribute schema.

Props aduth, noisysocks, youknowriad, danielbachhuber.

Merges [43918] to trunk.

See #45145 for the patch, #45098 for the original ticket.

[desrosj]

In 44273:

Block Editor: Update @wordpress package dependencies.

Update packages include:

• block-library
• components
• edit-post
• editor
• format-library
• list-reusable-blocks
• nux

Other changes:

• Fix a translator comment in edit-form-blocks.php.
• Rename the gutenberg_ functions in blocks/latest-comments.php.

Props noisy socks, youknowriad, pinto, swissspidy.

Merges [43935] and [43949-43951] into trunk.

See #45145.

[desrosj]

In 44281:

Block Editor: Update @wordpress package dependencies.

Updates the following packages:

block-library, block-serialization-default-parser, block-serialization-spec-parser, blocks, edit-post, editor, format-library.

Merges [43955] to trunk.

Props youknowriad.

Fixes #45145.