Merge Performant Translations (Ginger MO)

[swissspidy]

Over the past half a year or so, the core performance team spent significant time on extensively analyzing i18n performance in WordPress and finding ways to improve it. This cumulated in the ​Performance Translations feature plugin, which uses PHP files in favor of MO files for translations, which is much faster and benefits from OPcache as well.

This ticket tracks the suggestion of merging Performant Translations, which was originally developed by @dd32 under the name Ginger MO, into WordPress core.

View related make/core post: ​https://make.wordpress.org/core/2023/11/08/merging-performant-translations-into-core/

Performant Translations supports multiple file formats (.mo and .php), as well as multiple text domains and locales loaded at the same time.

This means:

• Loading .mo files as usual will be much faster and use less memory
• If an .mo translation file has a corresponding .php file, that file will be loaded instead, making things even faster and use even less memory
• When switching locales, we no longer have to unload the previous translations, but can instead keep them in memory, making locale switching much cheaper

A quick comparison:

Locale	Scenario	Memory Usage	Load Time
en_US	Default	14.46 MB	124.66 ms
de_DE	Default	27.96 MB	173.44 ms
de_DE	Performant Translations	15.62 MB	132.60 ms

As you can see, translations used to massively slow down a site, but with this new library, there is little to no additional overhead.

• ​i18n performance analysis, contains extensive benchmarks and comparisons of various alternative solutions
• ​Call for testing for Performant Translations

Since this uses a new file format for translations, some changes are required to add support for it in related places such as translate.wordpress.org.

Relevant companion tickets:

• ​Core patch
• ​GlotPress patch
• ​Meta patch

Everything is 100% backward compatible and if there are no PHP translation files, MO files will be loaded as usual.

---

Related: this is an alternative to #17268

[flixos90]

Marking this as a high priority for the 6.5 due to the significant estimated impact this will have for localized sites' performance.

[swissspidy]

In 57083:

Build/Test Tools: Expand performance test scenarios.

Adds new tests for localized sites as well as the dashboard.
Also amends Server-Timing output to measure memory usage in all scenarios.

Props swissspidy, joemcgill, flixos90, mukesh27, mamaduka.
See #59656.
Fixes #59815.

[joemcgill]

The PR for this isn't being picked up by the GH -> Trac connection. It's linked from the ticket description, but for anyone else having trouble finding it, the PR is: ​https://github.com/WordPress/wordpress-develop/pull/5306

[swissspidy]

@joemcgill I think it‘s because it‘s still a draft

[joemcgill]

Replying to swissspidy:

@joemcgill I think it‘s because it‘s still a draft

I've seen other draft PRs get picked up in tickets, like ​this one on #59600. I suspect the additional URLs in your PR description.

[akirk]

Since this ​adds the ability to include a PHP file generated elsewhere, I am worried that this potentially creates a vector for putting malicious code in translation files that didn't exist before.

The contents of the PHP file is predictable, we could parse it easily using PHP's token_get_all() function. We could introduce a "secure" mode where the file would be checked before it is included.

Although, since this is about performance, we likely don't want to add a performance penalty. Thus we could try and use a checksum to ensure the file was not changed after it has been checked.

What do you think?

[swissspidy]

Note that the security aspect has been previously mentioned in the ​corresponding i18n performance analysis post.
The main takeaway is that installing translations is no different than installing a plugin or theme, which can also execute arbitrary code. And WordPress has always considered translations to be trusted.
The post does also mention a static analysis or checksum check for added safety, though doing that at runtime (vs. at install time) will negate the performance wins again. The latter would require additional infrastructure for storing and retrieving checksums too.
Right now the "secure" mode is using the translation_file_format filter to disable PHP file usage.