Use native gettext library when available

[linushoppe]

​Here you say that the GNU gettext-Framework is used. Exactly, "pomo" (file: wp-includes/l10n.php) is a complete own php-implementation of the gettext-program.

I've added a patch to solve this problem. Maybe it is not very good, but it works. On my wordpress-sites, the used php-memory returns from about 65% to about 12% and the site is running much faster when patching wp-includes/l10n.php.

I know that gettext is not available on every wordpress-installation, but when it's available, it should be used.

Sorry for my bad english, I'm german.

Patch to reduce the used php-memory

Related: #17128

After applying this patch I was able to handle a substancial amout more requests in the same amount of time.

Running the same command command in all cases:

siege -c5 -t30s -b http://localhost

Results:

Version	trans/secs	succ. transactions
3.1.2 tagged	2.35	69
with patch from #17128	2.92	86
with just this patch	3.40	100
with this and the other patch	3.50	102

This change is a huge winner if gettext is installed. Would be nice to have this in 3.2.

Looks nice, but I'm wondering if it wouldn't be even faster if we had two definitions of each WP translation function:

if (function_exists ('dgettext')) {
  require_once('translate-gettext.php');
} else {
  require_once('translate-pomo.php');
}

translate-pomo.php would contain the current implementation in WP, while translate-gettext.php would contain the more efficient, gettext based definitions.

It looks like this patch doesn't handle translation with context at all and also I don't understand the need for all that copying of files.

I think this would be better implemented as an alternative implementation of a Translations class - probably could even be done in a plugin then maybe

Replying to scribu:

Looks nice, but I'm wondering if it wouldn't be even faster if we had two definitions of each WP translation function:

if (function_exists ('dgettext')) {
  require_once('translate-gettext.php');
} else {
  require_once('translate-pomo.php');
}

translate-pomo.php would contain the current implementation in WP, while translate-gettext.php would contain the more efficient, gettext based definitions.

Very nice idea. I just wrote this patch to make WP faster, I had no time to make the patch "good-looking" and useful.

We've created a new version of the patch. The new one (patch against l10n.php of Wordpress 3.1.3) can handle php contexts (msgctxt) via $context. We hope that it helps you to integrate this patch into one of the next wordpress versions in order to reduce the used php-memory.

Hi !

Be carefull on Cpanel servers, the folder creation and chmods generates a mkdir error. Added a @mkdir on line 377, and on some servers the folder creation may be mannual...

Moreover, translations are not directly applied. This is a little issue, but can be confusing for users. Must refresh the pages to have the translation applied ;)

WP has a helper function for that called wp_mkdir_p().

The are two problems that may need to be addressed:

• setlocale only works if the locale name matches a system locale.
• two different .mo files may not be used when they contain translations of the same domain. For example, "$locale.mo" and "ms-$locale.mo" (both referenced in the core).

What do we have to change to see this patch in one of the next WP-releases?

Patch against /wp-includes/l10n.php in WP 3.2.0

Love this patch! I get similar performance improvements like mwidmann.

Changing title to avoid confusion with #17128

I think it would be interesting to see this abstracted a bit cleaner. Perhaps, as westi indicated, a separate class.

For example, we already have Translations and NOOP_Translations. We should introduce a new _Translations class.

In the process, we should probably split the classes into three files.

Ideally all of the relevant code should occur in wp-includes/pomo, and none of it in l10n.php. That is simply our API layer.

Agreed. This should be just another translation implementation, like we have multiple implementations for HTTP requests.

Moving back to Future Release until we have the scope discussion for 3.4

native gettext-support php-file

For WordPress 3.3 we did a complete rewrite of the patch to make wordpress
work with PHP's native gettext-support.

The Patch now emulates the PoMo-API and creates an alias from the old class
named "MO" to itself - you just have to make sure that its loaded *before*
pomo is loaded.

File that were modified:

wp-settings.php (new verson attached to the bug report)

Files that were added

wp-includes/pomo/native.php (attached to the bug report)

I've added a patch to create the new native.php and to modify the wp-settings.php.

new wp-settings.php for native gettext-support

Hallo,
I've written a plugin -> ​WP-Performance-Gettext-Patch

I use a modified version of your patches:

• Add Cacheing-Control
• other minor changes

Benchmark:
Linux with Plugin 3x faster

xeno010, can you please explain what "other minor changes" your plugin has and why you've added caching? As far as I know, gettext itself has an own cache-implementation.
Besides, installing a plugin is not as powerful as patching the wordress-sourcecode and only some users would profit from the changes.

I closed ​http://wordpress.org/extend/plugins/wp-performance-gettext-patch/ for download as it applies a patch to core, which is dangerous and an all-around bad idea. Its source is still viewable and accessible ​in SVN.

I tested mentioned plugin and encountered great memory gains. My WP website with installed plugin used 28 MB memory and without this plugin more than 34 MB. This problem is real pain for localized versions, because there are more and more strings with every new release...

I ran into the l10n performance issue initially reported in #17128 on an Ubuntu 10.04 LTS (PHP 5.3.2, APC 3.1.3) system running Wordpress 3.3.1.

I did some testing and meassured the response time for serving a single post page.
Here are my numbers for Wordpress 3.3.1.

Feature	Response time	Performance hit
Stock 3.3.1, WPLANG not set	139ms	0%
WPLANG set to 'sv_SE'	255ms	45% slower
WPLANG set + patch from #17268	145ms	4% slower

As can be seen, the patch (native_gettext.patch) improved page response times by 43% compared to a stock Wordpress 3.3.1 installation where WPLANG is used.

I would like to see this patch merged for 3.4.

I just noticed that native_gettext.patch​ has the following comment:

@license http://creativecommons.org/licenses/by-sa/3.0/de/ Creative Commons Attribution-Share Alike 3.0 Germany

This license is ​not compatible with the GPL license, and therefore can not be included in WordPress Core.

Please consider changing the license to GPL or removing that comment altogether, in which case the GPL is implied.

Any native support we build in should fallback to the current pomo setup if the desired locale is not fully supported. The presence of the gettext extension does not mean that any given locale can be served. Type "locale -a" on your Mac OS or Linux box and see which ones you have. Some hosts install very few locales, often just C, POSIX, and en_US*. Different environments can have different names for the same locale. Also, the directory structure required by gettext is not compatible with what we use now.

FYI I get a 25% memory decrease and I if i benchmark my homepage: I go from around 18 req/s to 30+ req/s with this patch.
There is a typo on line 217: "$domain" should be "$Domain". (causes � bug)
Anyway thanks. And please push this out soon, even if turned off by default.

Not promising anything, but we may just end up doing this.

patch against WP 3.3.1, corrects the $Domain-Bug

I've added a new version of the patch. This patch was created against Wordpress 3.3.1. It fixes the $Domain-Bug mentioned.
More information: ​http://oss.tiggerswelt.net/wordpress/3.3.1/

By the way, the patch ist GPL now.

I tried to test latest patch with WP 3.3.1 and received following error:

Warning: Cannot redeclare class MO in .../wp-includes/pomo/native.php on line 242

Not going to happen for 3.4. Let's see what we can continue to cook up. If we can come up with a sturdy patch that doesn't affect installs without gettext, and handles issues related to LC_* values, I think we'd be open to reconsidering it.

Related: ​Drupal tries to adopt Symfony's PoReader.

+1 for making this patch merged into the core as soon as possible.

(Have done a quick profiling and the those *textdomain* functions really used too many CPU cycles)