Make WordPress Core

Opened 7 weeks ago

Last modified 5 days ago

#60647 assigned feature request

Script Modules: Allow modules to depend on existing WordPress scripts

Reported by: jonsurrell's profile jonsurrell Owned by: jonsurrell's profile jonsurrell
Milestone: 6.6 Priority: normal
Severity: normal Version: 6.5
Component: Script Loader Keywords: has-patch
Focuses: javascript Cc:

Description

Script modules cannot depend on existing scripts such as wp-url, wp-i18n, or wp-api-fetch.

Script modules should be able to leverage the functionality available in existing scripts.

Change History (16)

#1 @johnbillion
7 weeks ago

  • Version set to trunk

#2 @jonsurrell
6 weeks ago

I've spent a good amount of considering this and I'll share my ideas. First some concepts and constraints followed by ideas for implementation. I welcome questions, challenges, and feedback.


Backwards compatibility is very important. Therefore, all of the currently available scripts need to remain available for the foreseeable future.

Many existing scripts have side effects when they're loaded, such as initialization of a store. They may also have state or singletons that do not behave as expected when duplicated. See Gutenberg issue #8981 for details.

Scripts share functionality by attaching variables to the window object to expose them, they're accessible through the global namespace. For example. wp-api-fetch is accessible through window.wp.apiFetch.

Scripts can use modules through "dynamic import" (`import()`). import() is an async function (returns a Promise), implying that scripts using modules necessarily become async.

Modules can trivially access script functionality as long as the order of evaluation is correct (window.wp.apiFetch cannot be accessed before the wp-api-fetch script has been fetched and evaluated).


Given

  • Scripts need to remain.
  • Scripts should not be duplicated.
  • Modules can use scripts easily via globals.
  • Scripts using modules requires promises.

Some possible approaches:

First, modules could depend on scripts. Modules can access scripts through globals, so all we need is for dependencies to correctly enqueue scripts. This should be a simple change but sacrifices the developer experience. Folks need to know whether they're using a module or a script, and they need to include the correct dependency and use it in the correct way: module dependencies are imported, script dependencies are used through globals.

// Our dependent module's has a complicated dependencies array:
$dependencies = array(
  '@wordpress/interactivity',
  // Somehow we declare a script dependency (this form is not currently valid)
  array( 'import' => 'script', 'id' => 'wp-api-fetch' ),
);

// In the dependent module, we have a mix of imports and globals:
import * as interactivity from '@wordpress/interactivity';
const apiFetch = window.wp.apiFetch;

Another approach is to have proxy modules that export the globals provided by a script. The proxy modules would still need to correctly enqueue the associated scripts —there is a module->script dependency— but this would be handled for Core scripts and not a public, maintaining a clear separation between modules and scripts. The developer experience here is improved, instead of depending on some scripts and some modules, modules only depend on modules and developers would not need to access globals, they'd only use import.

Here's what some proxy modules might look like:

// Module wrapper for wp-api-fetch script
// The @wordpress/api-fetch package uses a default export
const apiFetch = window.wp.apiFetch;
export default apiFetch;

// Module wrapper for wp-url script
// The @wordpress/url package uses named exports
export const addQueryArgs = window.wp.url.addQueryArgs
export const getPath = window.wp.url.getPath
export const isURL = window.wp.url.isURL
// etc…

The advantage to using these proxy modules is improved developer experience:

// Our dependent module's dependencies are simpler:
$dependencies = array( '@wordpress/interactivity', '@wordpress/api-fetch' );

// In the dependent module, we only use imports:
import apiFetch from '@wordpress/api-fetch';
import { addQueryArgs } from '@wordpress/url';

The proxy module approach also opens up a potential enhancement. WordPress could include a proxy module and a full module version of scripts. When preparing the modules, we could check whether the script is enqueued or not. If the script is enqueued, we use the proxy module backed by the script. If the script is not enqueued, we use the full module version and the script does not need to be enqueued. This is a path where scripts can remain available but are largely replaced by modules without sacrificing backwards compatibility.

There are some drawbacks to the module proxy approach, mostly that we have an additional request for the proxy module. One solution could be to print the proxy module inline immediately after its associated module (or after the importmap if that's not been printed yet). The proxy modules are essentially lists of exports so are likely small in general.

Another potential downside is that this proposal focused on core scripts, it's not focused on extenders. If the approach works well, we can consider adding the appropriate extension points for extenders to follow the same approach.

#3 follow-up: @gziolo
6 weeks ago

Thank you for sharing initial ideas, @jonsurrell.

Folks need to know whether they're using a module or a script, and they need to include the correct dependency and use it in the correct way: module dependencies are imported, script dependencies are used through globals.

This approach would be manual as we don't have any automated way to detect wp globals and turn them into the list of dependencies even when using @wordpress/scripts tooling.

Another challenge, unrelated to the approach proposed, is ensuring that scripts and the import map are printed in the correct order so that the browser can resolve everything correctly. At the moment, ES Modules and regular scripts don't know anything about themselves.

#4 in reply to: ↑ 3 @jonsurrell
6 weeks ago

Replying to jonsurrell:

There are some drawbacks to the module proxy approach, mostly that we have an additional request for the proxy module. One solution could be to print the proxy module inline …

I don't believe it's possible to print a module with exports inline because there is no way to refer to it later, it doesn't have a name or URL that other modules can import.

#5 @jonsurrell
6 weeks ago

One concern with the proxy module approach was around the ordering of script execution. For example, the proposed proxy module @wordpress/a11y has an implicit dependency on the wp-a11y script. wp-a11y exposes required global variables and therefore must be executed before @wordpress/a11y.

I've done some testing and research and it appears that the ordering of scripts printed on the page will not be a problem. In summary, if the scripts are not async or defer (by default they are not), and modules are not async (by default they are not), they will executed in the order we require.


Enqueued scripts are printed as script tags without async or defer attributes. This means they "are fetched and executed immediately before the browser continues to parse the page." Enqueued scripts will be fetched and executed in the order they appear on the page.

Modules (<script type="module">) are always treated as `defer`, which means:

the script is meant to be executed after the document has been parsed.

If we put those together, we can see that scripts are executed before parsing continues, while modules are executed after the document has been parsed. This means that modules can depend on scripts independent of the ordering of script tags on the page, because scripts execute before modules.

Last edited 6 weeks ago by jonsurrell (previous) (diff)

#6 @gziolo
6 weeks ago

  • Milestone changed from Awaiting Review to 6.6

This ticket was mentioned in Slack in #core-performance by jonsurrell. View the logs.


6 weeks ago

This ticket was mentioned in Slack in #core-editor by gziolo. View the logs.


4 weeks ago

This ticket was mentioned in PR #6239 on WordPress/wordpress-develop by @jonsurrell.


12 days ago
#9

  • Keywords has-patch added

#10 follow-up: @jonsurrell
12 days ago

I've spent more time thinking about this. I see two options:

The first option is to allow modules to depend on scripts. This seems to be a fairly simple and safe change. Scripts continue to exist as scripts for the foreseeable future. Modules would use the scripts as e.g. window.wp.apiFetch. We can adapt existing tooling to help with this so folks can continue to author code like import apiFetch from '@wordpress/api-fetch' and compile it with the expected dependency and result (wp-api-fetch script dependency and const apiFetch = window.wp.apiFetch).

The downside with this first option is that it does not provide a transition strategy from scripts to modules and we sacrifice some of the potential benefits of modules like deferred or conditional/on-demand loading of modules.


The alternative is to expose scripts as modules somehow. Most of the modern scripts available in WordPress are authored as modules, then compiled via webpack as "libraries" — they expose their exports as global values, e.g. window.wp.apiFetch.

The proxy modules I've described in earlier comments provide a mechanism where a script dependency could be used when present or a module used on-demand when necessary. However, the proxy modules are worse in most ways: they're mostly overhead that only serve to use a module or script. The benefit would come if we can determine that a given script is not used on a page so that its corresponding module can be used directly without a proxy. If we cannot meet those conditions or only meet them in extremely rare conditions, then I don't see reason to pursue this approach.

I'll describe some scenarios to explain my thinking:

  • wp-api-fetch script is present. @wordpress/api-fetch module is not a dependency. Behavior is unchanged.
  • wp-api-fetch script is present. @wordpress/api-fetch module is a dependency. In this case @wordpress/api-fetch should point to the proxy module so that the wp-api-fetch script is used and not duplicated as a module.
  • wp-api-fetch script is not present. @wordpress/api-fetch module is a dependency. In this case @wordpress/api-fetch should point to the full @wordpress/api-fetch module to take advantage of the benefits of ES modules.

The difficulty of this comes from knowing whether a given script will be added to a page when we print the module importmap, that's the moment when we must provide a URL pointing to a proxy module or a real module. The importmap must be present on the page before any script type=module or link rel=modulepreload tags. Ideally the module preloads and the importmap are printed early in the page, so if scripts are enqueued late this information may not be available. This was a problem with the Modules API and classic themes.

Maybe proxy modules could always be used with classic themes, but block-based themes could use this strategy of using the proxy or a real module depending on whether a script is enqueued.

#11 @jonsurrell
12 days ago

There's an additional difficulty with some scripts when trying to use them as modules. Some scripts depend on special initialization as inline scripts. There's no obvious way to do this with modules without sacrificing some of the benefits of modules.

For example, wp-api-fetch script is initialized with an inline script like this:

wp.apiFetch.nonceMiddleware = wp.apiFetch.createNonceMiddleware( nonce );
// …more setup code

Where nonce is a server generated nonce for the REST API. This can be achieved in the same way with modules by using script type=module tag and importing @wordpress/api-fetch, but the module will be necessarily required which is undesirable. Ideally the module is only downloaded and initialized if and when it's needed.

A better solution may be to modify wp-api-fetch so that rather than the server injecting imperative code to set it up, instead it searches for some global variables and performs its own setup/initialization when necessary.

This ticket was mentioned in Slack in #core-performance by jonsurrell. View the logs.


11 days ago

#13 in reply to: ↑ 10 ; follow-up: @azaozz
10 days ago

Replying to jonsurrell:

The first option is to allow modules to depend on scripts. This seems to be a fairly simple and safe change.

Big +1. Thinking any "mixing" of modules and "ordinary, old-style" scripts should be kept to a minimum, and the simpler it is handled -- the better.

The downside with this first option is that it does not provide a transition strategy from scripts to modules and we sacrifice some of the potential benefits of modules like deferred or conditional/on-demand loading of modules.

Frankly I'm not sure if the WP old-style scripts can be transitioned to modules in many (most?) cases. The reason it that this would break any (old style) scripts added by plugins that depend on them. So thinking this is a no-issue for now, and perhaps won't be for a long time, if ever.

Seems the way forward would be to deprecate the old-style scripts and replace them with modules, then load them if/when a plugin enqueues them. (Of course, for this the module would have to be able to co-exist with the old-style script, and work together with it. This would probably be quite tricky. But lets look when we get there.)

Last edited 10 days ago by azaozz (previous) (diff)

#14 in reply to: ↑ 13 @cbravobernal
10 days ago

Replying to azaozz:

Seems the way forward would be to deprecate the old-style scripts and replace them with modules, then load them if/when a plugin enqueues them. (Of course, for this the module would have to be able to co-exist with the old-style script, and work together with it. This would probably be quite tricky. But lets look when we get there.)

Can we apply a POC of this solution first to the most used ones?

Applying just to i18n, api-fetch or core-data. As far as I know, community is asking to use core-data in the frontend too, and can be also a reason to "update it" to be frontend compatible and module compatible.

#15 follow-up: @youknowriad
10 days ago

We've discussed this issue a bit with @jonsurrell while no solutions emerged, I wanted to add some opinion here.

First, I'm actually not convinced at all that we should allow modules to depend on scripts (or to use scripts for that matter). I'm not convinced that the use-cases are too big for existing scripts. I do see folks wanting to use apiFetch or internationalization (probably a different one than wp-i18n), date maybe but these are fairly small scripts.

I think it's probably a mistake to try to use scripts in modules for different reasons:

  • A lot of these scripts are initialized server-side using inline scripts that are contextual. If I load wp-api-fetch in the editor or the dashboard, I won't get the same inline scripts. For modules, the opposite is true, the modules is always the same. This property of modules is a good thing in terms of architecture (separation between server and client) and making modules depend on scripts would be hard (which inline scripts to use) and create the issue of making the modules context dependent.
  • A lot of scripts have sub dependencies: polyfills, not certain all of this history is needed for the modules.
  • Add scripts as dependencies of modules, meaning that these scripts will always be printed in the page even if the module is not needed (will be async loaded later), which means the main selling point of modules becomes mute. We'll be shooting ourselves in the foot.

For these reasons, I think it's not worth it to try to load scripts in modules. I think we'd better try having build scripts that creates a script and a module for some selected packages. And I can even see some differences between the script and the module version of something. For instance configuring the translations shouldn't be done in the same way to wp-i18n, we'd have to load contextual translations instead (lazy load them as needed rather than print them server side...)

While this doesn't bring a solution, I think my best advice here would be to not try to solve this in a generic way, instead we should focus on concrete use-cases. I see more value personally (for a start) to be able to use modules within scripts (lazy load modules). For example a very concrete use-case I always bring is to be able to load a "codemirror" module in the editor for the code view for instance. I'd like editPost to be able to do import( '@wordpress/codemirror' ) when needed.

There might be other use-cases we can try with as well, but I don't think we offer a generic solution until we address specific use-cases. The solution can start as experimental/private APIs...

#16 in reply to: ↑ 15 @jonsurrell
5 days ago

@youknowriad I have some questions about the approach of adding new modules for the functionality that modules require. Let's look at an apiFetch module as an example folks are already requesting that exhibits challenges for a module.

The wp-api-fetch script relies on some imperative code to set it up for use in WordPress, that's something we'd need to address. There are likely other things we could improve in its module version.

What's the module version going to look like? Does it try to maintain the same API for the sake of familiarity? What does this mean for the @wordpress/api-fetch package that already includes module and commonjs builds? Will there be a new package for the modern api-fetch module version? Releasing a new major version with breaking changes would be a good way to move forward in the npm ecosystem, but does that imply a breaking change for the wp-api-fetch script? How would the script be maintained independently of the module version?

To be clear, I'm not opposed to the approach. I do like that it provides a way to separate more modern modules from legacy scripts and an opportunity to start with a clean slate, but I do want to explore the some of the difficult questions it raises.


I see more value personally (for a start) to be able to use modules within scripts (lazy load modules).

This may be sufficient for its own ticket. If we want to implement this I think the work will be separate from allowing dependence in the other direction.

Note: See TracTickets for help on using tickets.