Proposal: Tracking PHP Extension Usage

[dd32]

WordPress.org currently tracks the PHP usage of the platform and a bunch of other details, but one data point that's been needed time and time again is PHP Extension usage.

While we have some data sources which we refer to sometimes (VaultPress, ​Hosting Test Runner) these don't necessarily represent the very broad and varied hosting environments that WordPress runs on, as evidenced in the past by having to guess and revert/make changes based on bug reports afterwards.

WordPress can't be the only project that would benefit from knowing what ~34% of the web is running either, we're positioned in a very good place to influence other projects and PHP itself.
It could also give some direction to the Site Health project as to what segment of users it should potentially target with extra warnings.

Some decisions that rely upon knowing what PHP extensions are in use include..

• #47699 Can we drop the JSON Extension polyfill?
• #35517 Should the SSH endpoint even be in core since it's been broken for 3 year for some (most?) users
• #17268 Can we use the native gettext extension to improve translation speed?
• Do we still need to support the MySQL PHP extension? Can we just rely upon MySQLi?
• #42352 Can we add support for proper prepared queries based on MySQLi? Do we need to think about ext/MySQL support? (And is PDO an option?)
• #47186 How many users are using 32bit PHP without the libSodium Extension installed? (How wide spread is libSodium in the first place?)

I ​brought this up back in May on Slack and there wasn't any significant concerns raised, so I've finally made this ticket - if we can start sending the data from core I can get the API side storing data and we can find a way to publish the data for all to use as needed in these decisions.

The attached patch is just a quick POC, I'm not sure if there's anything else that should be included (or shouldn't be) but it could include other things like the following in the platform_flags section:

• 'db' => defined( 'WP_USE_EXT_MYSQL' ) && WP_USE_EXT_MYSQL ? 'mysql_forced' : ( function_exists( 'mysqli_connect') ? 'mysqli' : 'mysql' ) to determine MySQL vs MySQLi usage
• 'db_class' => get_class( $wpdb ) to detect the usage of other Database droppins
• web server in use based on the $is_apache, $is_IIS, $is_iis7, $is_nginx globals

On the privacy side of this - The new elements pass through the core_version_check_query_args filter that exists to opt-out of sending this data, and it doesn't contain personal details, so when used and processed in aggregate it should be okay.

Note: Any API would simply be aggregate percentages of an unknown total, ie. { platform: { bits: { 32: 50.5, 64: 49.5 } }, extensions: { json: 99.999, mysqli: 95.4, mysql: 45.2, .... } }

[dd32]

There's also the potential that maybe we'd want to record the versions of the extensions loaded, ie:

$extensions = array_combine( get_loaded_extensions(), array_map( 'phpversion', get_loaded_extensions() ) );

results in something like this on PHP 7.3.3:

...
  'xmlreader' => string '7.3.3' (length=5)
  'xmlwriter' => string '7.3.3' (length=5)
  'mysqlnd' => string 'mysqlnd 5.0.12-dev - 20150407 - $Id: 7cc7cc96e675f6d72e5cf0f267f48e167c2abb23 $' (length=79)
  'cgi-fcgi' => boolean false
  'gd' => string '7.3.3' (length=5)
  'mysqli' => string '7.3.3' (length=5)
  'xdebug' => string '2.7.0' (length=5)

[mikeschroder]

I've mentioned this outside of the ticket, but wanted to mention I think it'd be great if we started tracking availability of PHP extensions for all of the reasons mentioned in the ticket description.

Thanks @dd32!

[adamsilverstein]

This would also be a great way to track php extension support for modern image formats such as WebP and help inform our decision about including them in core (see https://core.trac.wordpress.org/ticket/35725).

Trac ticket:

[adamsilverstein]

48116.2.diff​ adds extensions + version info, as well as image library support data (includes image formats supported by libgd/imagick).

[adamsilverstein]

@dd32 Can we work on adding this feature in 5.8? Are there matching changes we need on the collection side to gather this data?

ps. I also have a PR open to add this to the test runner: ​https://github.com/WordPress/phpunit-test-runner/pull/134

[dd32]

Replying to adamsilverstein:

This would also be a great way to track php extension support for modern image formats such as WebP and help inform our decision about including them in core

Agreed! That's a perfect example of it.

The only request I have regarding additional data, is that it can be boiled down to a minimal list of things on the collection side, unfortunately the stats system is not really designed to track an arbitrary list of data points right now, especially when it's expected that a lot of sites will have those values set. The PHP Extensions even falls into this category, but I think the extra storage/complexity is worth it for something so generic.

For example, the gd_info() call returns a list of key-values where value can be false, ideally we'd probably only send the truthful items over (That way, older versions which don't know about a flag is the same as a newer version that knows about it, but says it doesn't support it)

I don't have any systems with Imagick installed, but it looks like that probably sends ~230 items that will most likely be set for all clients, storing those flags on a per-site basis is a lot of duplication when they're all probably going to be set.

When we're only really interested in one or two pieces of that data, it's a lot to preemptively collect if not really useful.

I guess, core can just send all of those datapoints, but on the collection side we'd boil it down to something like GD Supports WebP, Imagick supports WebP, or Supports WebP: yay, nay. Or other things like PDF Support: yay, nay.

Replying to adamsilverstein:

Can we work on adding this feature in 5.8?

If someone wants to add it on the Core side, I'd be happy to see that. Someone else just needs to lead that, as I'd prefer someone else be involved :)

Are there matching changes we need on the collection side to gather this data?

Yep! That'll be added once the data starts to be sent over to WordPress.org, there's no harm in sending it prior to dotorg actually handling collection for it.

[adamsilverstein]

The only request I have regarding additional data, is that it can be boiled down to a minimal list of things on the collection side,

Ok, that makes sense! I'm mostly interested in how widespread support for modern image formats is. WebP is the current format of choice because of its wide browser support.

In addition to WebP, it might we worth tracking AVIF and JPEGXL, two upcoming formats that offer even better compression and are more efficient on the server side. For example, GD just merged AVIF support: ​https://github.com/libgd/libgd/pull/671.

I guess, core can just send all of those datapoints, but on the collection side we'd boil it down to something like GD Supports WebP, Imagick supports WebP, or Supports WebP: yay, nay. Or other things like PDF Support: yay, nay.

The main metric we need to help inform adoption is "what percentage of WordPresses support this format?" so for collection side we can boil this down to SupportsWEBP: [true/false] SupportsAVIF: [true/false] SupportsJPEGXL: [true/false]

[adamsilverstein]

In 48116.3.diff​

• reduce captured information to the image formats supported by GD and Imagick (when present).

Format extraction based on code added in 5.8 to wp-admin/includes/class-wp-debug-data.php.

@dd32 This greatly reduces the captured data, what do you think?

[dd32]

Replying to adamsilverstein:

In 48116.3.diff​
dd32 This greatly reduces the captured data, what do you think?

Looks fine to me, I haven't run it, but it looks like it's good to go.

The only question I would have is, does the GD output keys always match the case exactly? Is it always WebP Support? Some people can be running very old GD extension versions I recall from years past, so I'm curious if a change made several years ago would affect this.

[adamsilverstein]

The only question I would have is, does the GD output keys always match the case exactly? Is it always WebP Support

I'm not really certain since I only tested in my own environment. I could try looking at some older libgd versions, to see if it ever changed. Also, we could release and watch for submissions with the image info is missing which I think would be the only consequence of the keys changing for some reason.

[adamsilverstein]

@dd32 can I go ahead and commit 48116.3.diff​? Do you need to make adjustments to the endpoint to make this work? This would be very helpful for understanding the image capabilities of WordPress in the wild.

[SergeyBiryukov]

A couple of notes on the latest patch:

• $gd_info and $imagick_info are defined but never used.
• $gd_supported_formats is defined twice.

[costdev]

I see that ​PR 1096, while in Draft, was updated 2 months ago and uses an alternative approach to the $gd_info, $imagick_info and gd_supported_formats variables. However, 48116.3.diff was proposed for commit.

@adamsilverstein, can you confirm whether the PR is an updated version of the patch and is the commit candidate, or if 48116.3.diff does indeed need a refresh?

[adamsilverstein]

@costdev I think you are right, I updated the PR but never updated the patch. I'll get this updated.

[mikeschroder]

We checked into this ticket in a 6.0 ​bug scrub session today.

Under the assumption that the PR is the most recent version of this:
I tested, and the patch generated from the PR seems to apply as expected.

Folks present thought that the most recent PR seemed reasonable, but review from Meta was needed to make sure the format looks good.

@dd32 Might you or someone else from the team have a chance to check to see if it looks ready?

[dd32]

My comments from Comment 12 still stand.

PR 1096 is 48116.2.diff​ which has the "benefit"/"downside" that:

• WordPress.org has to parse the gd/imagick arrays into what data to record.
• We don't really know what data will be in those payloads in the future and whether it might potentially contain PII data.

48116.3.diff​ is an iteration on that, post comment 12 suggestions that:

• Shifts the logic to the client for GD, looking for WebP, AVIS, HEIF support.
• Passes the All the Imageick supported formats through, instead of selecting which ones. Perhaps an array_intersect with $supported_formats is needed (see next)
• Comment 21 still applies. It seems that the second $gd_supported_formats with static array was supposed to be called $supported_formats perhaps?

Doing the parsing on WordPress.org (per 48116.2.diff​) has some benefits though;

• If new versions of the libraries change the format they report in, we can handle it there.
• If we decide we need to record data about some new flag that is present in the GD/Imagick data, we can do so.

That all being said, I now question what flags need to be sent/recorded.
Do we really need to know how many sites support WebP and AVIF? Yes, it would've been nice to know before we implemented webp support how many sites could generate it, but that time has passed.. Likewise for PDF support (Which 48116.3.diff​ might prevent if GD supports it in the future?)
So perhaps doing the parsing on the API side is the best course of action, and we just discard the data until a time comes where we actually want to know the stats it will provide?

Increasing the payload size also has detrimental impacts upon the processing speed of the API requests and time it takes to run, also the length of time it takes the stats processing to run, etc.

So...

• I'm okay with 48116.2.diff​ or 48116.3.diff​ (with fixes)
• I question the need for the GD/Imagick data, but no harm
• Please ensure that the payload size doesn't bloat up with irrelevant information
• Please define the stats wanted to be recorded from GD/Imagick, something that 99% of sites support is not a good data-point for us to record.
• Stats based on the submitted data likely won't be available for sometime, as we'll need to add the recording/generation code for it, determine what to generate daily/weekly/monthly/on-demand/etc.

[pbiron]

As 6.0 Beta will be released within the hour, moving this to 6.1.

[desrosj]

@adamsilverstein Are you able to give this a bit of attention in the coming weeks?

[adamsilverstein]

@desrosj yes, thanks for the ping. I'll review this again soon.

[adamsilverstein]

48116.4.diff​ is the latest version from the PR

Please ensure that the payload size doesn't bloat up with irrelevant information

@dd32 I feel like we could limit the information sent to the specific formats we are interested in. Realistically it would be useful to learn about WebP and AVIF support. Perhaps in the future that might extend to JPEGXL (but that isn't even in GD yet so nothing any time soon). We could limit the data to just support for those formats. Seems like the extra effort on the client site is worth the reduced payload and we can always change the processing if we need to capture a new data point or something else changed. What do you think?

Please define the stats wanted to be recorded from GD/Imagick, something that 99% of sites support is not a good data-point for us to record.

We want to know: what percentage of site's servers support the format?

[pbiron]

Replying to dd32:

For example, the gd_info() call returns a list of key-values where value can be false, ideally we'd probably only send the truthful items over (That way, older versions which don't know about a flag is the same as a newer version that knows about it, but says it doesn't support it)

That could easily be handled by changing the 1 line in the 48116.4.diff​:

'gd_info' => extension_loaded( 'gd' ) ? gd_info() : array(),

to

'gd_info' => extension_loaded( 'gd' ) ? array_filter( gd_info() ) : array(),

[adamsilverstein]

That could easily be handled by changing the 1 line

Thanks @pbiron - makes sense.

Regarding returning all values vs. returning just the data we know we want (AVIF/WebP support) - I'm leaning towards the latter, especially since the Imagick data is quite verbose and most isn't even useful. I'm going update the patch to just report on avif/webp support.

For reference, with Imagick, I get this data:

3FR, A, AAI, AI, ART, ARW, AVI, AVS, B, BGR, BGRA, BGRO, BMP, BMP2, BMP3, BRF, C, CAL, CALS, CANVAS, CAPTION, CIN, CIP, CLIP, CMYK, CMYKA, CR2, CRW, CUR, CUT, DATA, DCM, DCR, DCX, DDS, DFONT, DNG, DPX, DXT1, DXT5, EPDF, EPI, EPS, EPS2, EPS3, EPSF, EPSI, EPT, EPT2, EPT3, ERF, FAX, FILE, FITS, FRACTAL, FTP, FTS, G, G3, G4, GIF, GIF87, GRADIENT, GRAY, GROUP4, H, HALD, HDR, HISTOGRAM, HRZ, HTM, HTML, HTTP, HTTPS, ICB, ICO, ICON, IIQ, INFO, INLINE, IPL, ISOBRL, ISOBRL6, JNG, JNX, JPE, JPEG, JPG, JPS, JSON, K, K25, KDC, LABEL, M, M2V, M4V, MAC, MAGICK, MAP, MASK, MAT, MATTE, MEF, MIFF, MKV, MNG, MONO, MOV, MP4, MPC, MPEG, MPG, MRW, MSL, MSVG, MTV, MVG, NEF, NRW, NULL, O, ORF, OTB, OTF, PAL, PALM, PAM, PANGO, PATTERN, PBM, PCD, PCDS, PCL, PCT, PCX, PDB, PDF, PDFA, PEF, PES, PFA, PFB, PFM, PGM, PICON, PICT, PIX, PJPEG, PLASMA, PNG, PNG00, PNG24, PNG32, PNG48, PNG64, PNG8, PNM, PPM, PREVIEW, PS, PS2, PS3, PSB, PSD, PTIF, PWP, R, RADIAL-GRADIENT, RAF, RAS, RAW, RGB, RGBA, RGBO, RGF, RLA, RLE, RMF, RW2, SCR, SCREENSHOT, SCT, SFW, SGI, SHTML, SIX, SIXEL, SPARSE-COLOR, SR2, SRF, STEGANO, SUN, SVG, SVGZ, TEXT, TGA, THUMBNAIL, TIFF, TIFF64, TILE, TIM, TTC, TTF, TXT, UBRL, UBRL6, UIL, UYVY, VDA, VICAR, VID, VIFF, VIPS, VST, WBMP, WEBP, WMV, WPG, X3F, XBM, XC, XCF, XPM, XPS, XV, Y, YCbCr, YCbCrA, YUV

With GD, I get this data:

GIF, JPEG, PNG, WebP, BMP

[pbiron]

Replying to adamsilverstein:

Regarding returning all values vs. returning just the data we know we want (AVIF/WebP support) - I'm leaning towards the latter, especially since the Imagick data is quite verbose and most isn't even useful. I'm going update the patch to just report on avif/webp support.

I agree...

For reference, I get this data:

Here's what I get on my local machine (haven't checked on any live sites):

GD

<?php
array (
  'GD Version' => 'bundled (2.1.0 compatible)',
  'FreeType Support' => true,
  'FreeType Linkage' => 'with freetype',
  'GIF Read Support' => true,
  'GIF Create Support' => true,
  'JPEG Support' => true,
  'PNG Support' => true,
  'WBMP Support' => true,
  'XPM Support' => true,
  'XBM Support' => true,
  'WebP Support' => true,
  'BMP Support' => true,
  'TGA Read Support' => true,
  'JIS-mapped Japanese Font Support' => false,
)

Imagick

3FR, 3G2, 3GP, AAI, AI, APNG, ART, ARW, ASHLAR, AVI, AVIF, AVS, BGR, BGRA, BGRO, BIE, BMP, BMP2, BMP3, BRF,
CAL, CALS, CANVAS, CAPTION, CIN, CIP, CLIP, CLIPBOARD, CMYK, CMYKA, CR2, CR3, CRW, CUBE, CUR, CUT, DATA,
DCM, DCR, DCRAW, DCX, DDS, DFONT, DJVU, DNG, DOT, DPS, DPX, DXT1, DXT5, EMF, EPDF, EPI, EPS, EPS2, EPS3,
EPSF, EPSI, EPT, EPT2, EPT3, ERF, EXR, FARBFELD, FAX, FF, FILE, FITS, FL32, FLIF, FLV, FPX, FRACTAL, FTP,
FTS, G3, G4, GIF, GIF87, GRADIENT, GRAY, GRAYA, GROUP4, GV, HALD, HDR, HEIC, HEIF, HISTOGRAM, HRZ, HTM,
HTML, HTTP, HTTPS, ICB, ICO, ICON, IIQ, INFO, INLINE, IPL, ISOBRL, ISOBRL6, J2C, J2K, JBG, JBIG, JNG, JNX,
JP2, JPC, JPE, JPEG, JPG, JPM, JPS, JPT, JSON, JXL, K25, KDC, KERNEL, LABEL, M2V, M4V, MAC, MAP, MASK,
MAT, MATTE, MEF, MIFF, MKV, MNG, MONO, MOV, MP4, MPC, MPEG, MPG, MRW, MSL, MSVG, MTV, MVG, NEF,
NRW, NULL, ORA, ORF, OTB, OTF, PAL, PALM, PAM, PANGO, PATTERN, PBM, PCD, PCDS, PCL, PCT, PCX, PDB, PDF,
PDFA, PEF, PES, PFA, PFB, PFM, PGM, PGX, PHM, PICON, PICT, PIX, PJPEG, PLASMA, PNG, PNG00, PNG24, PNG32,
PNG48, PNG64, PNG8, PNM, POCKETMOD, PPM, PS, PS2, PS3, PSB, PSD, PTIF, PWP, RADIAL-GRADIENT, RAF, RAS,
RAW, RGB, RGB565, RGBA, RGBO, RGF, RLA, RLE, RMF, RSVG, RW2, SCR, SCREENSHOT, SCT, SFW, SGI, SHTML, SIX,
SIXEL, SPARSE-COLOR, SR2, SRF, STEGANO, SUN, SVG, SVGZ, TEXT, TGA, THUMBNAIL, TIFF, TIFF64, TILE, TIM, TM2,
TTC, TTF, TXT, UBRL, UBRL6, UIL, UYVY, VDA, VICAR, VID, VIFF, VIPS, VST, WBMP, WEBM, WEBP, WMF, WMV, WPG,
X3F, XBM, XC, XCF, XPM, XPS, XV, YAML, YCbCr, YCbCrA, YUV

[adamsilverstein]

In 48116.5.diff​ (and the PR) I refined the approach to only show AVIF & WebP support when present, for GD and imagick separately. On my local with both Imagick and GD enabled and both having only WebP support, info looks like this:

If my local supported AVIF it would be added to the array.

What do you think @pbiron & @dd32?

[pbiron]

Replying to adamsilverstein:

If my local supported AVIF it would be added to the array.

My local Imagick does support AVIF and it does show up in the array :-)

What do you think @pbiron & @dd32?

48116.6.diff​ is the same as 48116.5.diff​ except:

1. sorts the PHP extensions in a case-insensitive manner (not really necessary, but "nice")
2. fixes a few WPCS errors (​reported on the PR)

I think this is ready for commit...altho Dion should probably weigh in first.

[dd32]

Replying to pbiron:

I think this is ready for commit...altho Dion should probably weigh in first.

I'm good with it 👍!

[pbiron]

thanx Dion!

[adamsilverstein]

Thanks for helping get this ready; I will get this committed.

[adamsilverstein]

My local Imagick does support AVIF and it does show up in the array :-)

Neat! Out of curiosity @pbiron - what are you running locally?

[pbiron]

imagick 3.7.0 (PHP extension) and 7.1.0-18-Q16-HDRI for the ImagicMagick binary...and this is on Windows (in a WAMP-like stack).

But to be honest, I've never used AVIF images before :-)

[adamsilverstein]

In 53753:

Upgrade/Install: track php extensions and image library support for WebP and AVIF.

Add the loaded php extensions as well as whether the server's image libraries support WebP and AVIF image formats to the data payload sent during upgrade checks. Collecting this data with the WordPress.org API will help the project make more data driven decisions about feature support. Note that all data can still be filtered with the core_version_check_query_args filter for privacy reasons.

Props dd32, SergeyBiryukov, mikeschroder, pbiron. 
Fixes #48116.

Fixed in https://core.trac.wordpress.org/changeset/53753

Adds support for tracking SQLite availability in Servers.

Trac ticket: https://core.trac.wordpress.org/ticket/48116

[aristath]

@dd32 Would it be possible to add sqlite in the list of extensions to track? We'll need to make more educated decisions about supporting it in the future, so this data would be invaluable for Core.
(Proposal on Make/core: ​https://make.wordpress.org/core/2022/09/12/lets-make-wordpress-officially-support-sqlite/ was submitted about a month ago)

I'm attaching a patch on ​https://github.com/WordPress/wordpress-develop/pull/3393.

[adamsilverstein]

@dd32 now that we have begun to collect data around PHP extensions and modern image support, do you have an idea about how/where to make this data available?

[dd32]

Replying to aristath:

Would it be possible to add sqlite in the list of extensions to track? .... ​https://github.com/WordPress/wordpress-develop/pull/3393.

Added a comment on the PR, the data being sent over should already include sqlite3 in the extension list, although it seems phpversion( 'sqlite3' ) isn't returning the loaded library version.. so the only data known there would be the version of PHP it's bundled with.

now that we have begun to collect data around PHP extensions and modern image support, do you have an idea about how/where to make this data available?

I hadn't had a chance to actually start storing the data, so don't have any updates on that... Do you have any idea of how best to expose the numbers?