New implementation of MO file reader

[soletan]

On moving from a Debian-driven virtual server to one of my dedicated root server running Gentoo Linux I had to encounter poor performance in Wordpress on every click. Other sites hosted on same server respond within a second at most while Wordpress 2.8 took about 10-15 seconds on each click.

I started to upgrade packages, tried to performance-tune Apache and MySQL etc. without any remarkable success. And so I tried to locate the part of code in WP wasting so much time. Finally I stopped on method import_from_file() of class MO in wp-includes/pmo/mo.php.

I'm running WP with German localization available from de.wordpress.org. de_DE.mo takes 300KB disk spaces and about 8 seconds to load with your current class MO here. I did some investigations on the used code and saw potential to improve its performance, thus started to implement my own MO file reader omitting all that POMOReader stuff etc. Finally i got a linear method reading (at least my) MO file up to four times faster.
While server is emerging and compiling in background (e.g. compiling gcc and glibc) benchmarking existing MO file reader and my implementation resulted in a 8s vs. 2s runtime. BTW: other sites hosted on same server didn't remarkably slow down due to background compilation processes.

In addition to reading MO files faster the class tries to utilize wp_cache_* interface (which seems to be useless at the moment, isn't it) and it tries to completely work without instances of class Translation_Entry, which I consider to take much more memory than a normal hash (As I tried to produce instances on successfully parsing MO file performance decreased remarkably by estimated 10%.)

I'd like to provide my class here as-is since I didn't find the time to test it with other MO files or check if it's properly implementing API of class Translation.

[soletan]

Drop-In replacement for wp-includes/pomo/mo.php

[ShaneF]

Moved this to 2.9 release just in-case, but we need some test cases for speed I would guess to see how this compares to the current system.

[Denis-de-Bernardy]

see #10236

[Denis-de-Bernardy]

See this feedback on the patch in particular:

​http://core.trac.wordpress.org/ticket/10236#comment:2

[soletan]

I tried to understand issues reported in comments on #10236. I'm not quite familiar with trac and can't follow the timeline and relations of posts and comments that clearly.

Nevertheless I changed my implementation a bit. I removed any PHP5-specific stuff and added support for the reported multibyte overloading issue. Next I dropped senseless integration of non-persistent WP object cache and added three simple methods to manage cached translation maps in wp-content/cache/mo internally. I tested my previous code on whether its working with plugins' translations as well. Here I'm having wpcf contact form and nextgen gallery both getting translation files successfully loaded by my class.

I don't know current state of things and thus provide my updated post as-is again. Hope you'll find it useful ...

Best Regards,
Thomas Urban

[soletan]

Updated re-implementation, using internal cache, supporting multi-byte overloading

[ryan]

The cache API should be used. It is not useless if someone installs a persistent caching backend. Writing directly to files breaks the cache abstraction and will not work nicely in multi-server MU environments.

[soletan]

Replying to ryan:

The cache API should be used. It is not useless if someone installs a persistent caching backend. Writing directly to files breaks the cache abstraction and will not work nicely in multi-server MU environments.

Even though I consider this cache API including some limitations on automatically testing for cache-external updates to cached data requiring flushing the latter I agree with you in the fact that breaking APIs isn't acceptable.

So here is another version of my implementation using WPs cache API again. I didn't test that but it should work nevertheless.

This version includes further improvements regarding support for MO files not containing hashes maybe fixing previously reported issues on failing to read some plugins' translation catalogues. This was done with the support of H. Rabe from codestyling.de ...

[soletan]

Updated implementation using wp-cache API again, using different way of parsing MO-file header ...

[soletan]

BTW: Did you ever consider to commit yourself on using gettext including dgettext() etc. to support plugin-specific translation catalogues and thus stopping to parse MO-files in PHP yourself? Why isn't code doing it the other way around: if a user doesn't have gettext extension the currently available API is used as a fallback.

Is there a thread I can read to get eligible arguments for not doing this? Or did I miss some runtime configuration option to switch?

[nbachiyski]

I managed to get your ideas for parsing into a new patch, which almost retains the speed of mo3.php, but is still object-oriented. Several plugin authors really liked the objects idea and are already using it in their plugins.

See the patch here: ​http://trac.glotpress.org/changeset/293

On caching: on WordPress.com we have tried caching the read MO file and it didn't work out so well. Since this usually involves transmitting the data over a network it both filled the bandwith with translations and wasn't that much faster.

On using php gettext extension: we haven't revisited it in a long time. It used do depend on server locales and too few servers had it installed, that's why we haven't incorporated it. Also, there were problems with old gettext versions.

[ryan]

(In [12174]) pomo performance improvements. Props nbachiyski. fixes #10165