Proposal: Tracking PHP Extension Usage

[dd32]

WordPress.org currently tracks the PHP usage of the platform and a bunch of other details, but one data point that's been needed time and time again is PHP Extension usage.

While we have some data sources which we refer to sometimes (VaultPress, ​Hosting Test Runner) these don't necessarily represent the very broad and varied hosting environments that WordPress runs on, as evidenced in the past by having to guess and revert/make changes based on bug reports afterwards.

WordPress can't be the only project that would benefit from knowing what ~34% of the web is running either, we're positioned in a very good place to influence other projects and PHP itself.
It could also give some direction to the Site Health project as to what segment of users it should potentially target with extra warnings.

Some decisions that rely upon knowing what PHP extensions are in use include..

• #47699 Can we drop the JSON Extension polyfill?
• #35517 Should the SSH endpoint even be in core since it's been broken for 3 year for some (most?) users
• #17268 Can we use the native gettext extension to improve translation speed?
• Do we still need to support the MySQL PHP extension? Can we just rely upon MySQLi?
• #42352 Can we add support for proper prepared queries based on MySQLi? Do we need to think about ext/MySQL support? (And is PDO an option?)
• #47186 How many users are using 32bit PHP without the libSodium Extension installed? (How wide spread is libSodium in the first place?)

I ​brought this up back in May on Slack and there wasn't any significant concerns raised, so I've finally made this ticket - if we can start sending the data from core I can get the API side storing data and we can find a way to publish the data for all to use as needed in these decisions.

The attached patch is just a quick POC, I'm not sure if there's anything else that should be included (or shouldn't be) but it could include other things like the following in the platform_flags section:

• 'db' => defined( 'WP_USE_EXT_MYSQL' ) && WP_USE_EXT_MYSQL ? 'mysql_forced' : ( function_exists( 'mysqli_connect') ? 'mysqli' : 'mysql' ) to determine MySQL vs MySQLi usage
• 'db_class' => get_class( $wpdb ) to detect the usage of other Database droppins
• web server in use based on the $is_apache, $is_IIS, $is_iis7, $is_nginx globals

On the privacy side of this - The new elements pass through the core_version_check_query_args filter that exists to opt-out of sending this data, and it doesn't contain personal details, so when used and processed in aggregate it should be okay.

Note: Any API would simply be aggregate percentages of an unknown total, ie. { platform: { bits: { 32: 50.5, 64: 49.5 } }, extensions: { json: 99.999, mysqli: 95.4, mysql: 45.2, .... } }

[dd32]

There's also the potential that maybe we'd want to record the versions of the extensions loaded, ie:

$extensions = array_combine( get_loaded_extensions(), array_map( 'phpversion', get_loaded_extensions() ) );

results in something like this on PHP 7.3.3:

...
  'xmlreader' => string '7.3.3' (length=5)
  'xmlwriter' => string '7.3.3' (length=5)
  'mysqlnd' => string 'mysqlnd 5.0.12-dev - 20150407 - $Id: 7cc7cc96e675f6d72e5cf0f267f48e167c2abb23 $' (length=79)
  'cgi-fcgi' => boolean false
  'gd' => string '7.3.3' (length=5)
  'mysqli' => string '7.3.3' (length=5)
  'xdebug' => string '2.7.0' (length=5)

[mikeschroder]

I've mentioned this outside of the ticket, but wanted to mention I think it'd be great if we started tracking availability of PHP extensions for all of the reasons mentioned in the ticket description.

Thanks @dd32!

[adamsilverstein]

This would also be a great way to track php extension support for modern image formats such as WebP and help inform our decision about including them in core (see https://core.trac.wordpress.org/ticket/35725).

Trac ticket:

[adamsilverstein]

48116.2.diff​ adds extensions + version info, as well as image library support data (includes image formats supported by libgd/imagick).

[adamsilverstein]

@dd32 Can we work on adding this feature in 5.8? Are there matching changes we need on the collection side to gather this data?

ps. I also have a PR open to add this to the test runner: ​https://github.com/WordPress/phpunit-test-runner/pull/134

[dd32]

Replying to adamsilverstein:

This would also be a great way to track php extension support for modern image formats such as WebP and help inform our decision about including them in core

Agreed! That's a perfect example of it.

The only request I have regarding additional data, is that it can be boiled down to a minimal list of things on the collection side, unfortunately the stats system is not really designed to track an arbitrary list of data points right now, especially when it's expected that a lot of sites will have those values set. The PHP Extensions even falls into this category, but I think the extra storage/complexity is worth it for something so generic.

For example, the gd_info() call returns a list of key-values where value can be false, ideally we'd probably only send the truthful items over (That way, older versions which don't know about a flag is the same as a newer version that knows about it, but says it doesn't support it)

I don't have any systems with Imagick installed, but it looks like that probably sends ~230 items that will most likely be set for all clients, storing those flags on a per-site basis is a lot of duplication when they're all probably going to be set.

When we're only really interested in one or two pieces of that data, it's a lot to preemptively collect if not really useful.

I guess, core can just send all of those datapoints, but on the collection side we'd boil it down to something like GD Supports WebP, Imagick supports WebP, or Supports WebP: yay, nay. Or other things like PDF Support: yay, nay.

Replying to adamsilverstein:

Can we work on adding this feature in 5.8?

If someone wants to add it on the Core side, I'd be happy to see that. Someone else just needs to lead that, as I'd prefer someone else be involved :)

Are there matching changes we need on the collection side to gather this data?

Yep! That'll be added once the data starts to be sent over to WordPress.org, there's no harm in sending it prior to dotorg actually handling collection for it.

[adamsilverstein]

The only request I have regarding additional data, is that it can be boiled down to a minimal list of things on the collection side,

Ok, that makes sense! I'm mostly interested in how widespread support for modern image formats is. WebP is the current format of choice because of its wide browser support.

In addition to WebP, it might we worth tracking AVIF and JPEGXL, two upcoming formats that offer even better compression and are more efficient on the server side. For example, GD just merged AVIF support: ​https://github.com/libgd/libgd/pull/671.

I guess, core can just send all of those datapoints, but on the collection side we'd boil it down to something like GD Supports WebP, Imagick supports WebP, or Supports WebP: yay, nay. Or other things like PDF Support: yay, nay.

The main metric we need to help inform adoption is "what percentage of WordPresses support this format?" so for collection side we can boil this down to SupportsWEBP: [true/false] SupportsAVIF: [true/false] SupportsJPEGXL: [true/false]

[adamsilverstein]

In 48116.3.diff​

• reduce captured information to the image formats supported by GD and Imagick (when present).

Format extraction based on code added in 5.8 to wp-admin/includes/class-wp-debug-data.php.

@dd32 This greatly reduces the captured data, what do you think?

[dd32]

Replying to adamsilverstein:

In 48116.3.diff​
dd32 This greatly reduces the captured data, what do you think?

Looks fine to me, I haven't run it, but it looks like it's good to go.

The only question I would have is, does the GD output keys always match the case exactly? Is it always WebP Support? Some people can be running very old GD extension versions I recall from years past, so I'm curious if a change made several years ago would affect this.

[adamsilverstein]

The only question I would have is, does the GD output keys always match the case exactly? Is it always WebP Support

I'm not really certain since I only tested in my own environment. I could try looking at some older libgd versions, to see if it ever changed. Also, we could release and watch for submissions with the image info is missing which I think would be the only consequence of the keys changing for some reason.