Extend Block Metadata PHP Cache to Third-Party Blocks

[mreishus]

Summary

WordPress 6.1 introduced a performance improvement for core blocks by reading block metadata from a single PHP file instead of parsing block.json files on each request. This change proposes extending this optimization to third-party blocks, allowing them to register their block metadata in a centralized location and benefit from the same performance improvements.

Description

In WordPress 6.1, the core team implemented a performance enhancement for core blocks by compiling block metadata into a single PHP file (wp-includes/blocks/blocks-json.php) to avoid parsing multiple block.json files on each request. This change notably improved the block registration process's performance, as detailed in the ​developer note.

However, this optimization is currently only available for core blocks. Third-party blocks, especially those from large plugins with numerous blocks (for example, Jetpack or WooCommerce), could also benefit from this performance improvement.

This proposal introduces a new class, WP_Block_Metadata_Registry, which allows core and third-party plugins to register their block metadata in a centralized location. The register_block_type_from_metadata function is modified to accept an optional $metadata_source argument, which is a string in the format namespace/source. If this argument is provided, the function will attempt to retrieve the block's metadata from the registry before falling back to reading from the block.json file, maintaining backward compatibility with the current behavior.

Key points:

• Introduce a new function, wp_register_block_metadata, for core and third-party plugins to register their block metadata.
• Modify register_block_type_from_metadata to accept an optional $metadata_source argument to retrieve metadata from the registry.
• Fall back to reading from the block.json file if metadata is not found in the registry, ensuring backward compatibility.

By extending the block metadata PHP cache to third-party blocks, we can provide a notable performance benefit to sites using large block libraries, like those from WooCommerce and Jetpack. This change also encourages plugin developers to optimize their block registration process, leading to better overall performance for WordPress sites.

The implementation is based on the original optimization for core blocks introduced in WordPress 6.1, as described in the ​developer note.

Related Tickets

#55005: Improve PHP performance for block.json files. This ticket is where the idea originally came from.
changeset 54276: Editor: Improve block loading PHP performance. This commit implemented the original $core_blocks_meta for core only.

Alternative Approach Considered

An alternative approach was considered that didn't require making core changes. It involved modifying a third-party plugin to keep its own compiled block file and use that to generate $args to pass to register_block_type_from_metadata(), taking advantage of the change introduced in changeset 57026.

However, this approach had some drawbacks:

When registering blocks through the $args function without passing a path to a block.json file, attempting to register blocks with styles did not work. This is because the linking is done to the style from looking at $metadata['name'], but when registering via $args, the block name is not present in $metadata. It only exists in $args/$settings at this point. This issue has been reported in a new bug: #61910.

The need for additional processing from the block.json file, which needs to be kept in sync with core. For example, the field named apiVersion in the block.json file needs to be transformed to api_version before passing through to $args. While this isn't too problematic, it's concerning that if core changes the fields, this extra processing from the third-party plugin will have to be kept in sync with core's list of fields.

In contrast, the mechanism developed in changeset 54276, which works for core only, does not require this processing, making it ideal. The build script only needs to be '<?php return ' + json2php( blocks ) + ';'.

Patch

The proposed patch includes:

• Introduction of the WP_Block_Metadata_Registry class to manage block metadata from various sources.
• Addition of the wp_register_block_metadata function for registering block metadata.
• Modifications to register_block_type_from_metadata to use the metadata registry and accept an optional $metadata_source argument.
• Additional unit tests to cover the new functionality.

See the attached patch for the complete implementation.

Backward Compatibility

The proposed change is designed to be backward compatible with existing third-party blocks. If a block's metadata is not found in the registry, or the $metadata_source argument is not provided, register_block_type_from_metadata will fall back to reading from the block.json file, preserving the current behavior.

Trac ticket: https://core.trac.wordpress.org/ticket/62002

[mreishus]

Attached Github PR (first time trying this in conjunction w/ trac)

• ​https://github.com/WordPress/wordpress-develop/pull/7303/files

[gziolo]

It's also something we could use in the Gutenberg plugin and avoid the performance penalty of deregistering all core block types and their assets before registering them again from scratch. I'm very much in favor of offering a way to feed some registry with block metadata before the block registration happens. I'm not entirely sure how much work that would required on the Gutenberg plugin side, but at least all static blocks that aren't initialized from the PHP file could easily get migrated to this approach.

[azaozz]

It's also something we could use in the Gutenberg plugin and avoid the performance penalty

Was just thinking that while reading the ticket description :)

Moving this to the 6.7 milestone for consideration. It has a patch that looks pretty good (may need couple small enhancements). Seems this only needs a review by the performance team.

The recommended and most popular usage for block registration is through register_block_type, which uses register_block_type_from_metadata internally when the path to block.json is passed as the first argument. Example:

function my_plugin_my_block_block_init() {
        register_block_type( __DIR__ . '/build' );
}
add_action( 'init', my_plugin_my_block_block_init' );

There were some refactorings in the past that I'm not deeply familiar with, but that made the implementation more flexible and more performant. In the world where we have the proposed cache registry for metadata, it would get to another level. Although, it also raises the question of whether we can simplify everything. I took a look at how it all works today.

This is an example of core block registration:

​https://github.com/WordPress/wordpress-develop/blob/a78540b0881a195ec8488bb6bdf878114d75f8b8/src/wp-includes/blocks/index.php#L149-L157

There are multiple steps here in WP core:

1. Use webpack to generate ​require-static-blocks.php file
2. Use also webpack to generate the cache ​blocks-json.php with all block.json files loaded
3. Register the block type by providing the path to metadata based on paths generated with (1). Metadata is consumed from file (2), which has everything bundled, with keys being core block names.

Some aspects I was thinking about, as register_block_type_from_metadata has the following steps after many iterations:

1. Load the block.json file from the disk or from the cache from core blocks
2. Translate the metadata to block settings, which means either mapping the properties or processing them so some additional work happens, like registering scripts and styles, callbacks, etc
3. Register the block type in the registry:

​https://github.com/WordPress/wordpress-develop/blob/a78540b0881a195ec8488bb6bdf878114d75f8b8/src/wp-includes/blocks.php#L707-L710

---

What I'm currently thinking about is whether we can keep the existing public API. In effect, we would need a single global registry where the metadata gets stored. $file_or_folder could become the key for setting and getting the entry from the registry:

function register_block_type_from_metadata( $file_or_folder_or_cache_key, $args = array() ) {
        // Try to get metadata from the static cache for core blocks.
        $metadata = array();    
        $registry = WP_Block_Metadata_Registry::get_instance();
        if ( $registry->has( $file_or_folder_or_cache_key ) {
                $metadata = $registry->get( $file_or_folder_or_cache_key );
        } else {
                // Load the file from the disk as before.
        }
        // The rest of the logic.
}

I assumed that the file no longer needs to exist on the disk and instead anything can be put as a cache key so I renamed the first param to $file_or_folder_or_cache_key.

The registry itself could have some helper method to register multiple block type metadata entries in one call, so for WP core that would be the following:

$registry = WP_Block_Metadata_Registry::get_instance();
$registry->addCollection( require ABSPATH . WPINC . '/blocks/blocks-json.php' );

I'm not entirely sure how that would work with the Gutenberg plugin, but we can figure it out later. Let me know what do you think about all of the above. I'm curious about the rationale for having namespaces in the metadata registry that in effect would trigger adding the 3rd param. I'm happy to discuss it further to align on the path forward.

Thank you for the feedback, this is very helpful in refining the approach.

Simplifying the API
I agree with your suggestion to simplify the API. The idea of using a single global registry with $file_or_folder_or_cache_key as the first parameter makes sense and would allow us to remove the third parameter I initially proposed. This approach seems cleaner and more in line with existing WordPress conventions. I have modified the code to give this a try.

Namespaces in the metadata registry
I initially introduced namespaces due to concerns about potential conflicts, particularly noticing that both core and the Gutenberg plugin can have differing metadata for the same blocks. However, I'm open to removing namespaces if we can address the potential overwriting issues in another way. Perhaps we can ensure that Gutenberg always registers its metadata after core.

Third-party plugin usage
I did imagine third party plugins having to add their own build step to create a compiled php file with all of the block data. I have a prospective approach for WooCommerce here: ​https://github.com/woocommerce/woocommerce/pull/51184 _(In the next hour or so, I will update it to match the latest changes in this PR)._

register_block_type()
I did not know this was the main way to register blocks. In accordance with the API changes above, I have also changed this to feed through and use the same mechanism. If you check the code, you'll see something strange about checking the type of $args - I noticed the unit test test_should_return_a_default_fallback_navigation_menu_with_no_blocks_if_page_list_block_is_not_registered is passing the undocumented type of WP_Block_Type to $args (instead of an array with the same properties as WP_Block_Type). In this case, WP_Block_Type_Registry::get_instance()->register() can handle an $args as WP_Block_Type here, but register_block_type_from_metadata() cannot, so I had to add the safeguard. However, this feels a bit strange and might warrant further inspection.

register all helper
Didn't get a chance to look at this yet.

I did not know this was the main way to register blocks. In accordance with the API changes above, I have also changed this to feed through and use the same mechanism.

The nice part is that we will be able to avoid all the file system checks if the metadata is cached. We definitely will have to adjust register_block_type in the way you did it. I'll have a closer look at the edge case you highlighted later. It sounds like the test might need some adjustments. In my opinion, everything is on the right track.

I did not know this was the main way to register blocks. In accordance with the API changes above, I have also changed this to feed through and use the same mechanism.

The nice part is that we will be able to avoid all the file system checks if the metadata is cached. We definitely will have to adjust register_block_type in the way you did it. I'll have a closer look at the edge case you highlighted later. It sounds like the test might need some adjustments. In my opinion, everything is on the right track.

[flixos90]

I just reviewed the pull request, and I like where it's going, however there are some foundational things to still figure out on the approach. Together with the beta being in a week from now, this most likely will mean we won't be able to get this ready in time for 6.7.

If we can prioritize this and continue the discussion on definition and implementation in the next few days, maybe we can still make it happen. I'll definitely keep an eye out to any new replies. But if this isn't ready to commit by next Monday, we'll have to punt it to 6.8.

A lot of changes here from the initial implementation! The register_block_type_from_metadata() changes are looking a lot cleaner now. The core specific code is removed and now uses the new registry. I was also able to get WooCommerce working with it (​draft PR here).

I tried to beef up the comments to explain what's happening, but essentially you first register the file with:

WP_Block_Metadata_Registry::register_collection(
  '/srv/http/wordpress/wp-content/plugins/myplugin/blocks/',
  '/srv/http/wordpress/wp-content/plugins/myplugin/blocks/blocks-json.php'
)

That php file ("manifest file" in comments - should this change?) should contain something like:

<?php return array(
  'timeline' => array(
    'name' => 'myplugin/timeline',
    'title' => 'Timeline',
    // ..properties omitted..
  ),
  'profile-card' => array(
    'name' => 'myplugin/profile-card',
    'title' => 'Profile Card',
    // ..properties omitted..
  ),
  // ..other unique blocks..
);

Now, if you

register_block_type_from_metadata( '/srv/http/wordpress/wp-content/plugins/myplugin/blocks/timeline' );

or

register_block_type_from_metadata( '/srv/http/wordpress/wp-content/plugins/myplugin/blocks/timeline/block.json' )

It will look in the timeline key of the registered data to find the metadata for that block. The mapping we do from $file_or_folder to manifest key is done by taking the directory name of where the block.json file lives. Right now, you can provide an optional callback to change this mapping, but I didn't need it for WooCommerce. I wonder if other large plugins like Gutenberg or Jetpack would need this customization available or not? I haven't researched that yet.

@gziolo Would you be able to give this another review as well? Is there anything that Gutenberg would need to change to cater for this once it lands in Core?

One other idea I had was to automatically unset() any metadata after fetching it from the registry, since presumably it would only be needed once, and it would free up a small amount of memory. Not sure about this though.

One other idea I had was to automatically unset() any metadata after fetching it from the registry, since presumably it would only be needed once, and it would free up a small amount of memory. Not sure about this though.

Probably overkill :)

Also you never know how plugins will use this function, so best to keep the data in place.

Would you be able to give this another review as well? Is there anything that Gutenberg would need to change to cater for this once it lands in Core?

I’m looking at it right now. I need to read the communication to better understand how the metadata collection gets registered and consumed. I still need to see how that relates to Gutenberg and whether there is an easy path to override the metadata cache and avoid block deregistration and registration. However, that is advanced usage so it shouldn’t block this one.

I’m looking at it right now. I need to read the communication to better understand how the metadata collection gets registered and consumed. I still need to see how that relates to Gutenberg and whether there is an easy path to override the metadata cache and avoid block deregistration and registration. However, that is advanced usage so it shouldn’t block this one.

Looking at the gutenberg_reregister_core_block_types() function, I don't think this PR would really change anything. The Gutenberg code would work the same as before. So the core blocks would still need to be unregistered to then re-register the Gutenberg equivalents, but as you're saying it's advanced usage anyway.

The one enhancement this would bring is that Gutenberg could now register its own collection file, to get the same performance optimization when re-registering its own blocks. So at least there's that benefit even for this scenario. Still, it's optional, so while we should probably add this to Gutenberg, it's not urgent, and there shouldn't be any breakage from landing this PR in Core.

[flixos90]

IMO this is good to commit. Will give a bit of time in case other folks want to chime in.

Note that I'm also adding needs-dev-note, as the new API functionality is primarily relevant to plugin developers so that they can improve performance of how their plugin registers blocks.

@felixarntz, the test case you described is what I was contemplating, as we could remove many file checks that aren’t necessary in the case we have everything in the metadata cache. There is some nuance to it with assets that define paths as they are relative to block.json for developers convienience.

@mreishus this looks great. @felixarntz, great feedback shared. I'm very happy about all the improvements that give me hope we might be able to use it as is in Gutenberg, too. I don't have any blocking for feedback. I left several notes for my own understanding of the changes and thought about future considerations when it's time to wire it in the Gutenberg plugin.

If you can, it would be great if you could add 1-2 tests that cover the new logic in register_block_type_from_metadata(). For example, you could register a theoretical block collection and then register a theoretical block from that collection without actually having the block.json file exist, but ensure the block gets still registered because the function relies on the data from the block collection registry.

I added two tests here: ​https://github.com/WordPress/wordpress-develop/blob/db067b0b770c6f65425c864e4a3809046d39b23f/tests/phpunit/tests/blocks/registerBlockTypeFromMetadataWithRegistry.php First time adding a new test file, so not sure I got everything right

[flixos90]

@gziolo @mreishus Is there already tooling available to generate the kind of block metadata manifest file needed for this API? In other words, where does this happen for the Core blocks?

Ideally, whatever is used for the Core blocks would be a reusable solution that plugins could rely on to, so that using the API that this ticket and PR would introduce is in fact easy to use without every plugin developer having to copy a lot of code - for example as part of @wordpress/scripts?

[mreishus]

@flixos90 Core has set it up in this grunt task, which the new implementation is compatible with: ​https://github.com/WordPress/wordpress-develop/blob/66afbbdb519d4b21a2ed486de64e32eda3f5adde/Gruntfile.js#L1448-L1463

As a test, I have a prospective implementation of the task for WooCommerce here: ​https://github.com/woocommerce/woocommerce/pull/51184/files#diff-76bcd2fcf3b7e61b77f7a53f939ccb1bdad57676121cc392fa1102d227f509c0

There isn't too much to it if you let json2php do most of the work. Personally, I suspect the most difficult part will be navigating the custom build tooling of each repository to add a new step.

Seeing mysql Error Head "https://registry-1.docker.io/v2/library/mysql/manifests/8.0": unauthorized: incorrect username or password. I don't think that's related to this PR, I can try rebasing..

Seeing mysql Error Head "https://registry-1.docker.io/v2/library/mysql/manifests/8.0": unauthorized: incorrect username or password. I don't think that's related to this PR, I can try rebasing..

Edit: Doesn't seem to have worked :(

Definitely seems unrelated. I wonder if something else in Core is causing this, or it's just some temporary outage or instability somewhere?

[gziolo]

Is there already tooling available to generate the kind of block metadata manifest file needed for this API? In other words, where does this happen for the Core blocks?

It isn't part of @wordpress/scripts. So far, it has been implemented only in WordPress core, but I don't have enough insights on whether it's integrated on the webpack or grunt level. That's said, I would be game of providing a way to offer a simple and performant way to register multiple blocks with one function call that internally combines the new API for registering the metadata cache collection and using it to generate all individual blocks listed.

[flixos90]

In 59132:

Editor: Allow registering PHP manifest file for block metadata collections for enhanced performance.

Typically, when registering a new block type, its metadata is read from the provided block.json file. The more block types are registered on a site, the more costly becomes this process, as it involves filesystem reads and parsing JSON.

WordPress Core's built-in blocks have in the past worked around that by having a auto-generated PHP manifest file that includes the already parsed JSON data for all blocks. This changeset effectively allows plugins to do the same, by introducing a new API function wp_register_block_metadata_collection(). The WordPress Core block manifest is now handled using this API as well, rather than custom logic baked into register_block_type_from_metadata().

The wp_register_block_metadata_collection() function requires two parameters:

• $path: The base path in which block files for the collection reside.
• $manifest: The path to the manifest file for the collection.

Every block.json file that is supposed to be part of the collection must reside within the provided $path, within its own block-specific directory matching the block name (without the block namespace). For example, for a collection $path of /wp-content/plugins/test-plugin and a block test-plugin/testimonial, the block file could be /wp-content/plugins/test-plugins/blocks/testimonial/block.json.

It is recommended that plugins use the new API function for enhanced performance, especially if they register several block types. However, the use of the function is entirely optional. Not using it will not result in any difference in user-facing behavior.

Props mreishus, flixos90, gziolo, spacedmonkey, azaozz, mukesh27.
Fixes #62002.

Committed in https://core.trac.wordpress.org/changeset/59132

[flixos90]

Replying to gziolo:

Is there already tooling available to generate the kind of block metadata manifest file needed for this API? In other words, where does this happen for the Core blocks?

It isn't part of @wordpress/scripts. So far, it has been implemented only in WordPress core, but I don't have enough insights on whether it's integrated on the webpack or grunt level. That's said, I would be game of providing a way to offer a simple and performant way to register multiple blocks with one function call that internally combines the new API for registering the metadata cache collection and using it to generate all individual blocks listed.

I opened a Gutenberg issue to get this added to @wordpress/scripts: ​https://github.com/WordPress/gutenberg/issues/65758

[flixos90]

For reference: @mreishus already opened a Gutenberg PR ​https://github.com/WordPress/gutenberg/pull/65866 to add a command for this to @wordpress/scripts.

[fabiankaegy]

Hey @flixos90 👋

Are you planning to work on a dev note for this in the next week? Would love to get this included in the Field Guide

[flixos90]

Replying to fabiankaegy:

Hey @flixos90 👋

Are you planning to work on a dev note for this in the next week? Would love to get this included in the Field Guide

Yes it's on my list. Will try to get it done this week.

[flixos90]

@fabiankaegy Dev note public draft: ​https://make.wordpress.org/core/?p=115864&preview=1&_ppp=848634d99d

Only awaiting reviews, but otherwise would be ready to publish.

[flixos90]

Dev note published: ​https://make.wordpress.org/core/2024/10/17/new-block-type-registration-apis-to-improve-performance-in-wordpress-6-7/