POMO library performance considerations

[wet]

Profiling a localized WordPress instance with WPLANG defined as 'de_DE' on my testbed shows that loading the .PO files for WP core and TwentyTen increases the frontend runtime by about 45 percent and the backend runtime by about 30 percent.

This is the profiling environment:

• WordPress 3.2-bleeding r17633
• TwentyTen 1.1
• No active Plugins
• Idling webserver on localhost
• Apache 2.2.9
• PHP 5.2.6
• MySQL 5.0.67
• Xdebug 2.1.1
• webgrind 1.1
• Windows 7 SP1

Without WPLANG defined, runtimes for various code paths are as follows:

• Front page with 10 simple posts: 2300 ms
• Admin Dashboard: 3000 ms
• RSS XML feed output: 1130 ms

With the de_DE l10n in effect, runtimes increase as such:

• Front page with 10 simple posts: 4300 ms
• Admin Dashboard: 4280 ms
• RSS XML feed output: 2950 ms

I wonder whether anyone is able to reproduce this grave performance degradation. A sample output from webgrind showing various POMO functions as the top offenders is attached.

[wet]

Webgrind profiling report of site frontpage

[wet]

Replace is_null() with NULL comparison. A low hanging fruit, but 1.5 percent performance increase is maybe better that a sharp stick in the eye. See ​http://hakre.wordpress.com/2011/03/29/php-is_null-vs-null/.

[scribu]

Looks like a fine candidate for 3.2.

[linushoppe]

Also see #17268, there is a patch which reduces the used php-memory for translation about 75%.

[nacin]

Nothing here yet, punting.

[SergeyBiryukov]

pomo-entry-17637.patch​ seems like a valid patch.

[SergeyBiryukov]

Refreshed as per coding standards.

[Chouby]

I observed the same issue.

Please correct me if I am wrong, but it looks like every internationalized strings are loaded on every page. Loading the complete mo file on every page could explain such huge difference of performance between English and localized versions. Moreover, as WorPress is growing, the mo file is growing too (for de_DE.mo, it was 296 ko for 2.8, 386 ko for 3.0 and 435 ko now), so this performance issue is growing too.

So if my assumption is good, the most optimized solution could be to load only the strings needed for the current page. I am not a specialist of the pomo structure, but I imagine that in the worst case it could mean splitting the current mo file in many small files. Obviously it would be difficult to do and maintain.

However, it may be a good solution to split the current mo file in two. One small file for the frontend and the big file for the admin side (it seems to me that there are not a lot of core strings displayed on the frontend). Maybe it would be necessary to load both files on the admin side. I am not sure.

This way, of course, we do not improve the time running on the admin side but we should improve a lot the time running on the frontend. Since, inn most cases, the frontend is loaded much more times than the admin, the server load decrease should be quite perceptible...

[scribu]

@Chouby: Splitting the translation files in two is an idea worth exploring, but you should open a new ticket for it.

[pavelevap]

There are also many strings which can be seen only in multisite version. Maybe they should get right textdomain? Or is there any reason why these strings are not inluded in existing separate multisite .mo file?

[SergeyBiryukov]

Replying to pavelevap:

Or is there any reason why these strings are not inluded in existing separate multisite .mo file?

I guess this also should be investigated in a new ticket.

[pavelevap]

OK, there are 2 new tickets: #19831 #19832

[noah.williamsson]

FWIW, I've experienced similar performance issues as outlined in the original report on an Ubuntu 10.04 LTS (PHP 5.3.2, APC 3.1.3) system running Wordpress 3.3.1.

Switching from Wordpress' own gettext implementation to PHP's native gettext libary (using a patch from #17268) improved page response times by 43% compared to a stock Wordpress 3.3.1 installation where WPLANG is used.

Feature	Response time	Performance hit
Stock 3.3.1, WPLANG not set	139ms	0%
WPLANG set to 'sv_SE'	255ms	45% slower
WPLANG set + patch from #17268	145ms	4% slower

[wet]

Related: ​Drupal tries to adopt Symfony's PoReader.

[rulatir]

IMO the whole l10n infrastructure should be refactored to use on-demand loading. Rationale: AJAX! My ajax requests that go through wp-includes/wp-admin.php load all translated strings, only to return a JSON array with a few rows from the DB that contains no translated strings whatsoever.

[scribu]

We could do on-demand loading of strings on a per-textdomain basis, but that would need some profiling, since each call to translate() would now need an if clause to see if a textdomain load has already been attempted.

[rulatir]

The if can be avoided the same way we currently avoid checking if the .mo file was there in the first place for the given domain: if translations for a domain are not available, the framework creates an instance of NOOP_Translations, rather than Gettext_Translations, see l10n.php:499. We could use the same pattern for on-demand loading: we could write Textdomain_Loader, another implementation of the Translations() interface, and instantiate it where we now instantiate Gettext_Translations. Textdomain_Loader::translate() would do the actual loading and change the global reference to point to the newly crated MO object.

[rulatir]

I am investigating the feasibility of doing this with a plugin using override_load_textdomain filter.

[greencp]

I just published a plugin that implements on-demand loading of mo files and on-demand translating of strings: ​WP Performance Pack.
It uses override_load_textdomain and a new Translations implementation. It already improves performance and memory usage but I'm working on some more improvements by reducing unnecessary translations, of wich there are quite some in WordPress core (e.g. #26746). I also plan to implement usage of native gettext if available.

[ccismaru]

There are a ton of calls to chr and is_null. Checking the code, there can be easily replaced with string equivalent or with language constructs, which will reduce the overhead of a function call as well as other code ran in the internal function bodies.

chr(0) could be replaced with "\0x0", chr(4) with "\0x04". is_null(exp) with exp === NULL. Have tested this and about 22K calls were saved because of these minor changes. I'm aware that the code is more readable keeping them in place, as they are now, but in a shared hosting environment could make a big difference in saving CPU cycles.

[ocean90]

17128.patch​ was applied in [35714].

17128.diff​ replaces the chr() calls mentioned in comment:29.

[ocean90]

In 43635:

I18N: In the POMO library, replace chr() calls for static values with their string representation.

Props ccismaru, ocean90.
Fixes #17128.