Make wordpress.org API connections for updates, plugins and themes configurable to a different location

[jamesking56]

The codebase appears to show that wordpress.org API connections are hard-coded to point to wordpress.org, example:

​https://github.com/WordPress/WordPress/blob/master/wp-includes/update.php#L182

It would be better if these were configuration via config.php so that WordPress hosting companies could configure their own proxy or mirror of the resources on wordpress.org. Saving wordpress.org bandwidth in the process.

[jorbin]

While not a configuration option, it is currently possible to change this using the pre_http_request filter using something like:

add_filter( 'pre_http_request', function( $response,  $parsed_args, $url ) {
    if ( str_starts_with( $url, 'http://api.wordpress.org' ) || str_starts_with( $url, 'https://api.wordpress.org' ) ){
        $new_url = ''; // insert your update server here.
        return wp_remote_request( $new_url, $parsed_args );
    }
    return $response;
}, 10, 3);

[desmith]

Attaching a "I banged it out in like ten minutes" strawman implementation of this.

I'm making the assumption that this hypothetical .org replacement will implement the update-related APIs, but intentionally do NOT route things like the Events and Docs APIs to this new service - those would still go back to .org as the "canonical" source of those things. The details are subject to a lot of discussion, but we have to start somewhere.

[desmith]

(Note that this doesn't directly cover the possibility of a "download mirror," but since the download URL for a plugin or theme is returned by the API, it's indirectly covered there.)

[jamesking56]

Thanks @desmith looks good to me!

@jorbin whilst your solution does work, its a tad messy having to parse every single outbound http request of the system

[desmith]

Realistically, there's a lot more that needs to be done. If you're using something other than the default API endpoint, that should be surfaced to the end-user somewhere, at a minimum in a health check. And there's plenty of potential discussion about whether we need/want the ability to make the API endpoints more flexible (different hosts for plugins vs themes, maybe?).

Like I said, this is the strawman implementation, here to be knocked over and replaced by actually good code instead. :)

[Clorith]

While possible to do with the HTTP API filter, that feels like a crutch in this case.

Referencing the more recent statement from Matt about the risk to WordPress.org over time, and the global impact it would have on sites and services using WordPress if the site were to go away, I think it's reasonable to implement something here that would facilitate the easy switching of sources, either by hosts, or by site owners them selves.

Given the current state, implementing a means to easily run your own mirrors would also be a show of good faith, which I think would be valued by many. If there's agreement that my thoughts here make sense, I'd be happy to milestone it for 6.8, and help anyone wanting to work on it, land it.

For a first iteration, and probably a good place to start as early as for 6.8, I would like to see a common filter for just the api.wordpress.org address to be controllable with a mirror that answers to the same API requests.

By doing this as a filter, and not a constant, you could pass the full query as context and allow the filter to modify the URL based on the intent, as that would cover both use cases without over-complicating the basic implementation.

This could either be a filtered function reused throughout core, or a filter in the HTTP API wrapper so that we only need to maintain a single location, without needing to go through the source and modify all existing links.

The upside to this is that it would simplify the process for hosts to create their own mirrors to reduce load on WordPress.org as a whole, and improve performance. Yes many hosts proxy requests there as is, but this would give the smaller ones with less resources that same starting point.

[Clorith]

Accidentally tabbed and changed to the wrong milestone :)

[jamesking56]

Thanks @Clorith for adding your comments to this, very helpful.

My intention is for web agencies mostly. If you manage 100+ WordPress sites as an agency, you will most likely be looking for a way to ensure that WordPress.org doesn't go down and affect all of your customers. So, just would be nice to configure and host your own mirror to enable all 100+ sites to access approved plugins, themes etc. in the same datacenter.

That's the idea for me, just the first step by allowing WordPress to open up by allowing you to point it to a different API. It would be totally down to the agency to then develop the API responses to match what WordPress.org's API responses look like.

[desmith]

To make my biases clear, I'm looking at this from the POV of a hosting provider. Making it a constant makes it harder to break - we can define it globally, and don't have to worry about individual customers pointing it elsewhere, or plugins (malicious or otherwise) overriding our override. The current workarounds (both a pre_http_request filter like the one earlier in this ticket, and all the things third-parties already do to pull plugins from their own places) should still work with little to no impact.

The filter approach is certainly more flexible, with all both good and ill that implies. You're probably right in saying it's the better long-term approach for WP as a whole. It also is harder to implement, which means I may not be the best one to do it as I'm not an overly experienced dev, but I'll take a look in the next few days.

[desmith]

Need an opinion on something tangential to this, and which could have somewhat more dramatic side effects than "just" replacing a hostname.

I'm working on a patch that would replace most references to api.wordpress.org with a call to a filter-able function, but while looking for hard-coded references to the hostname, I noted that many of them still make requests to http:// and not to https://. The code is simpler if we make this setting/filter be the whole base URL (i.e. https://api.wordpress.org and not just the domain name api.wordpress.org), but that has the implicit side effect of making all the API requests that currently use http, now use https.

All the tests I've done so far, seem to imply that on api.wordpress.org, http is unconditionally redirected to https anyway, but one person doing tests (who doesn't even know how the API works all that well) is far from conclusive.

(EDIT: The above is a fib, my test tooling was "helpfully" making requests to https by default, but I think it's still a valid question to ask whether we should make these requests https-by-default.)

[desmith]

super rough implementation of 'make the WP update API filter'able'

[desmith]

Uploaded the "this took an hour or two" version, which makes the URL filter'able.

Very important notes:

• This is barely tested! It doesn't crash WP but that's pretty much it. In particular, I don't have a working mirror or proxy against which to test.
• This has an implicit side effect of making all calls to api.wordpress.org, use HTTPS (mostly because it was a lot simpler that way, and again in very limited testing it didn't seem to break anything).
• If you're using an alternate location for updates, we should probably do a health check against it, similar to the existing get_test_dotorg_communication()
• I intentionally left references to documentation, credits, events, and the salt generator, pointing to .org instead of overriding them - referring users to the "canonical" location for docs seemed appropriate, I could be persuaded to change the salts though.

Again, it's just a starting point. I assume I've overlooked a bunch of things.

[desmith]

If you do have a mirror/proxy/etc:

add_filter( 'dotorg_api_base', function() {
  return 'https://your.mirror.org/';
});

[Clorith]

This is a great starting point, thank you. I've got some feedback to help move this forward;

The function should ideally also account for environment variables, as there are events within WordPress that fire before it is set up (for example during the installation) which rely on fetching data from the API, the language selector in the setup wizard is a great example of this.

Having the function support all three methods, much like the environment type lookup does, in a pattern of $_ENV < CONST < filter (left to right in increasing priority) would allow a host to define their preferred source on the environment before any additional code can run, and easily deploy this to all their services.

As for the changing of http to https for the default hostname, I believe this has been set so intentionally to allow sites who may be running older versions of WordPress to still access the host even if the bundled CA certificates are expired, so they can still install updates. There may be other security considerations attached to this these days though, but it is a very small thing to change the URL in a singular location in a patch later on based on feedback, so lets keep it as http:// for now to reduce the amount of changes. The HTTP API should upgrade the request to HTTPS if supported after all.

We should also see if we can think of a better function and constant name... naming things is hard, and I don't have a clear direction for what would be correct here, but the WP_DOTORG prefix doesn't feel right, and perhaps it needs to be something less internal ("dotorg" only means something to those who are in the know as it were, and anyone else looking at the code would not as easily see the correlation).

[desmith]

The most annoying part about the http/https thing, is that it's not consistent - some of the existing calls to api.wordpress.org use one, some use the other. I suppose the URL scheme could be an option added to the get_whatever_api_base call, but I'll have to think through the best way to do that -- my assumption is that someone doing a "new" API implementation today would have it be all-https, because it's not 2006 anymore, and I'd like it to default to that when appropriate, and avoid the extra request when possible.

I agree that dotorg in the name is a terrible idea, but I didn't have a better idea myself either :D Things like PLUGIN_API felt wrong because it's also themes and other stuff; UPDATE_API felt wrong because it's more than updates. And WORDPRESS_API is too vague, could easily be misread as something related to an app's own REST stuff.

Didn't even think about initial installs and env vars, so I'll roll all that around in my noodle too.

[desmith]

Updated patch forthcoming.

• For the http/https thing, went with adding a flag to select whether we should return http or https in this case. It's not especially elegant, and by design it only works if you're still using the default hostname - my assumption is that anyone doing a new implementation would do it https-only. (Absolutely open to suggestions on better ways to handle this.)
• Changed the function name, the constant name, and the new env var, to use "update_api" because it seems to be the least-inaccurate option.
• The health check doesn't yet work, in my testbed it's returning a 500, will keep hacking on it, but I wanted to get something out there.

[desmith]

Updated (but still not fully operational)

The wp_get_api_hostname() function would allow hosters the ability to more easily set up local mirrors of the WordPress API.

This pull request summarizes some of the work already put into the ticket, and feedback from other communication platforms, so that all ideas are brought forth.

It introduces a new function, wp_get_api_hostname(), this is very directly named, as it relates to the base hostname used for API requests, and we wish to ensure its self evident what the intent of the function is.

It allows a hoster to define a WP_API_HOSTNAME environment variable, to pre-empt any installation steps, ensuring all requests are passed through their mirror from start to finish in the WP journey if need be.

It is also possible for a site to define the WP_API_HOSTNAME constant in their wp-config.php file, if they wish to diverge from the hosters predefined expectations.

And finally, there's the wp_api_hostname filter, for more fine grained control. Currently it does not pass any contexts, but it is easier to add to a filter later, than to take away from it, so this is a nice starting point for an initial implementation I think.

For more fine grained filtering control, the filters found within the WordPress HTTP API are better suited, but are not available to a site early enough in the flow to be usable for example during setup.

I've also simplified the Site Health outputs, instead of introducing extra checks, re-labeling the reference to "api.wordpress.org" to "the WordPress API" does the same, and by adding a field in the debug data section to reference what API host is returned, the information is still readily available if someone needs to look it up.

Trac ticket: https://core.trac.wordpress.org/ticket/62132

[Clorith]

After the core meeting today, I wanted to add my reasoning for pushing for this, just to hopefully provide some clarity as there have been some initial feedback in comment:1

It is true that it's possible to filter the WordPress HTTP API to modify the API requests, but this not only isn't filterable during setup for example, where calls are made to glotpress, but it is also a very deep filter, and there is a psychological aspect to making something hard to change, telling everyone "we do not want this to be changed".

I find this important with the recent movements in the WordPress ecosystem to give hosters a feeling of freedom and autonomy _if they want it_, and show them the trust we have in their ability to maintain their platforms. By giving them not only an easier means to mirror API requests, but also allowing them to control the use of them based on their environments that _don't_ require them to deploy global must-use plugins or similar on their platforms.

Mirrors can be a valuable resource in reducing network strain on the WordPress.org infrastructure (depending on how they are implemented). Providing an easy, and official, means of implementing a safety net would make this much easier. It would also make things easier on hosters who for various reasons do not have direct access to WordPress.org (I presume most are aware of the mirrors that exist of WordPress.org, down to the level of forum posts, in some countries because the main site is inaccessible).

This looks great @Clorith.

[jorbin]

Thanks, @Clorith. I hadn't considered the pre-installation calls. Those are extremely difficult to modify and the env variable could make that easier.

it is also a very deep filter, and there is a psychological aspect to making something hard to change, telling everyone "we do not want this to be changed".

I don't grok. What makes this a "very deep filter"? I don't think it was hard to write up my example to demonstrate that this is already possible.

Any chance of getting this commited for 6.8? The beta period upon us.

@Clorith @SergeyBiryukov this is looking nice! What remains to be done here? Happy to help!