Make WordPress Core

Opened 4 months ago

Last modified 8 days ago

#62132 new feature request

Make wordpress.org API connections for updates, plugins and themes configurable to a different location

Reported by: jamesking56's profile jamesking56 Owned by:
Milestone: Awaiting Review Priority: normal
Severity: normal Version:
Component: Upgrade/Install Keywords:
Focuses: Cc:

Description

The codebase appears to show that wordpress.org API connections are hard-coded to point to wordpress.org, example:

https://github.com/WordPress/WordPress/blob/master/wp-includes/update.php#L182

It would be better if these were configuration via config.php so that WordPress hosting companies could configure their own proxy or mirror of the resources on wordpress.org. Saving wordpress.org bandwidth in the process.

Attachments (2)

63132-strawman.patch (4.9 KB) - added by desmith 4 months ago.
63132-filter-strawman.patch (15.6 KB) - added by desmith 8 days ago.
super rough implementation of 'make the WP update API filter'able'

Download all attachments as: .zip

Change History (14)

#1 @jorbin
4 months ago

While not a configuration option, it is currently possible to change this using the pre_http_request filter using something like:

add_filter( 'pre_http_request', function( $response,  $parsed_args, $url ) {
    if ( str_starts_with( $url, 'http://api.wordpress.org' ) || str_starts_with( $url, 'https://api.wordpress.org' ) ){
        $new_url = ''; // insert your update server here.
        return wp_remote_request( $new_url, $parsed_args );
    }
    return $response;
}, 10, 3);

#2 @desmith
4 months ago

Attaching a "I banged it out in like ten minutes" strawman implementation of this.

I'm making the assumption that this hypothetical .org replacement will implement the update-related APIs, but intentionally do NOT route things like the Events and Docs APIs to this new service - those would still go back to .org as the "canonical" source of those things. The details are subject to a lot of discussion, but we have to start somewhere.

#3 @desmith
4 months ago

(Note that this doesn't directly cover the possibility of a "download mirror," but since the download URL for a plugin or theme is returned by the API, it's indirectly covered there.)

#4 @jamesking56
4 months ago

Thanks @desmith looks good to me!

@jorbin whilst your solution does work, its a tad messy having to parse every single outbound http request of the system

#5 @desmith
4 months ago

Realistically, there's a lot more that needs to be done. If you're using something other than the default API endpoint, that should be surfaced to the end-user somewhere, at a minimum in a health check. And there's plenty of potential discussion about whether we need/want the ability to make the API endpoints more flexible (different hosts for plugins vs themes, maybe?).

Like I said, this is the strawman implementation, here to be knocked over and replaced by actually good code instead. :)

#6 @Clorith
3 weeks ago

  • Milestone changed from Awaiting Review to WordPress.org

While possible to do with the HTTP API filter, that feels like a crutch in this case.

Referencing the more recent statement from Matt about the risk to WordPress.org over time, and the global impact it would have on sites and services using WordPress if the site were to go away, I think it's reasonable to implement something here that would facilitate the easy switching of sources, either by hosts, or by site owners them selves.

Given the current state, implementing a means to easily run your own mirrors would also be a show of good faith, which I think would be valued by many. If there's agreement that my thoughts here make sense, I'd be happy to milestone it for 6.8, and help anyone wanting to work on it, land it.

For a first iteration, and probably a good place to start as early as for 6.8, I would like to see a common filter for just the api.wordpress.org address to be controllable with a mirror that answers to the same API requests.

By doing this as a filter, and not a constant, you could pass the full query as context and allow the filter to modify the URL based on the intent, as that would cover both use cases without over-complicating the basic implementation.

This could either be a filtered function reused throughout core, or a filter in the HTTP API wrapper so that we only need to maintain a single location, without needing to go through the source and modify all existing links.

The upside to this is that it would simplify the process for hosts to create their own mirrors to reduce load on WordPress.org as a whole, and improve performance. Yes many hosts proxy requests there as is, but this would give the smaller ones with less resources that same starting point.

#7 @Clorith
3 weeks ago

  • Milestone changed from WordPress.org to Awaiting Review

Accidentally tabbed and changed to the wrong milestone :)

#8 @jamesking56
3 weeks ago

Thanks @Clorith for adding your comments to this, very helpful.

My intention is for web agencies mostly. If you manage 100+ WordPress sites as an agency, you will most likely be looking for a way to ensure that WordPress.org doesn't go down and affect all of your customers. So, just would be nice to configure and host your own mirror to enable all 100+ sites to access approved plugins, themes etc. in the same datacenter.

That's the idea for me, just the first step by allowing WordPress to open up by allowing you to point it to a different API. It would be totally down to the agency to then develop the API responses to match what WordPress.org's API responses look like.

#9 @desmith
3 weeks ago

To make my biases clear, I'm looking at this from the POV of a hosting provider. Making it a constant makes it harder to break - we can define it globally, and don't have to worry about individual customers pointing it elsewhere, or plugins (malicious or otherwise) overriding our override. The current workarounds (both a pre_http_request filter like the one earlier in this ticket, and all the things third-parties already do to pull plugins from their own places) should still work with little to no impact.

The filter approach is certainly more flexible, with all both good and ill that implies. You're probably right in saying it's the better long-term approach for WP as a whole. It also is harder to implement, which means I may not be the best one to do it as I'm not an overly experienced dev, but I'll take a look in the next few days.

#10 @desmith
2 weeks ago

Need an opinion on something tangential to this, and which could have somewhat more dramatic side effects than "just" replacing a hostname.

I'm working on a patch that would replace most references to api.wordpress.org with a call to a filter-able function, but while looking for hard-coded references to the hostname, I noted that many of them still make requests to http:// and not to https://. The code is simpler if we make this setting/filter be the whole base URL (i.e. https://api.wordpress.org and not just the domain name api.wordpress.org), but that has the implicit side effect of making all the API requests that currently use http, now use https.

All the tests I've done so far, seem to imply that on api.wordpress.org, http is unconditionally redirected to https anyway, but one person doing tests (who doesn't even know how the API works all that well) is far from conclusive.

(EDIT: The above is a fib, my test tooling was "helpfully" making requests to https by default, but I think it's still a valid question to ask whether we should make these requests https-by-default.)

Last edited 2 weeks ago by desmith (previous) (diff)

@desmith
8 days ago

super rough implementation of 'make the WP update API filter'able'

#11 @desmith
8 days ago

Uploaded the "this took an hour or two" version, which makes the URL filter'able.

Very important notes:

  • This is barely tested! It doesn't crash WP but that's pretty much it. In particular, I don't have a working mirror or proxy against which to test.
  • This has an implicit side effect of making all calls to api.wordpress.org, use HTTPS (mostly because it was a lot simpler that way, and again in very limited testing it didn't seem to break anything).
  • If you're using an alternate location for updates, we should probably do a health check against it, similar to the existing get_test_dotorg_communication()
  • I intentionally left references to documentation, credits, events, and the salt generator, pointing to .org instead of overriding them - referring users to the "canonical" location for docs seemed appropriate, I could be persuaded to change the salts though.

Again, it's just a starting point. I assume I've overlooked a bunch of things.

#12 @desmith
8 days ago

If you do have a mirror/proxy/etc:

add_filter( 'dotorg_api_base', function() {
  return 'https://your.mirror.org/';
});
Note: See TracTickets for help on using tickets.