Make WordPress Core

Opened 4 years ago

Last modified 2 months ago

#17268 new enhancement

Use native gettext library when available

Reported by: linushoppe Owned by:
Milestone: Future Release Priority: normal
Severity: normal Version:
Component: I18N Keywords: early has-patch needs-refresh
Focuses: performance Cc:


Here you say that the GNU gettext-Framework is used. Exactly, "pomo" (file: wp-includes/l10n.php) is a complete own php-implementation of the gettext-program.

I've added a patch to solve this problem. Maybe it is not very good, but it works. On my wordpress-sites, the used php-memory returns from about 65% to about 12% and the site is running much faster when patching wp-includes/l10n.php.

I know that gettext is not available on every wordpress-installation, but when it's available, it should be used.

Sorry for my bad english, I'm german.

Attachments (6)

wp_gettext.patch (4.1 KB) - added by linushoppe 4 years ago.
Patch to reduce the used php-memory
wp_gettext_v2.patch (4.3 KB) - added by linushoppe 4 years ago.
wp_gettext_v3.patch (4.0 KB) - added by linushoppe 4 years ago.
Patch against /wp-includes/l10n.php in WP 3.2.0
native.php (6.6 KB) - added by linushoppe 4 years ago.
native gettext-support php-file
wp-settings.php (9.7 KB) - added by linushoppe 4 years ago.
new wp-settings.php for native gettext-support
native_gettext.patch (7.9 KB) - added by linushoppe 4 years ago.
patch against WP 3.3.1, corrects the $Domain-Bug

Download all attachments as: .zip

Change History (67)

@linushoppe4 years ago

Patch to reduce the used php-memory

comment:1 @duck_4 years ago

Related: #17128

comment:2 @johnbillion4 years ago

  • Cc johnbillion@… added

comment:3 @mwidmann4 years ago

After applying this patch I was able to handle a substancial amout more requests in the same amount of time.

Running the same command command in all cases:

siege -c5 -t30s -b http://localhost


Version trans/secs  succ. transactions
3.1.2 tagged 2.35 69
with patch from #17128 2.92 86
with just this patch 3.40 100
with this and the other patch 3.50 102

This change is a huge winner if gettext is installed. Would be nice to have this in 3.2.

comment:4 follow-up: @scribu4 years ago

Looks nice, but I'm wondering if it wouldn't be even faster if we had two definitions of each WP translation function:

if (function_exists ('dgettext')) {
} else {

translate-pomo.php would contain the current implementation in WP, while translate-gettext.php would contain the more efficient, gettext based definitions.

comment:5 @westi4 years ago

It looks like this patch doesn't handle translation with context at all and also I don't understand the need for all that copying of files.

I think this would be better implemented as an alternative implementation of a Translations class - probably could even be done in a plugin then maybe

comment:6 @Philipp15b4 years ago

  • Cc hebipp1@… added

comment:7 in reply to: ↑ 4 @linushoppe4 years ago

Replying to scribu:

Looks nice, but I'm wondering if it wouldn't be even faster if we had two definitions of each WP translation function:

if (function_exists ('dgettext')) {
} else {

translate-pomo.php would contain the current implementation in WP, while translate-gettext.php would contain the more efficient, gettext based definitions.

Very nice idea. I just wrote this patch to make WP faster, I had no time to make the patch "good-looking" and useful.

@linushoppe4 years ago

comment:8 @linushoppe4 years ago

  • Cc linus.hoppe@… added

We've created a new version of the patch. The new one (patch against l10n.php of Wordpress 3.1.3) can handle php contexts (msgctxt) via $context. We hope that it helps you to integrate this patch into one of the next wordpress versions in order to reduce the used php-memory.

comment:9 @aphrodite4 years ago

Hi !

Be carefull on Cpanel servers, the folder creation and chmods generates a mkdir error. Added a @mkdir on line 377, and on some servers the folder creation may be mannual...

Moreover, translations are not directly applied. This is a little issue, but can be confusing for users. Must refresh the pages to have the translation applied ;)

Last edited 4 years ago by aphrodite (previous) (diff)

comment:10 @scribu4 years ago

WP has a helper function for that called wp_mkdir_p().

Last edited 4 years ago by scribu (previous) (diff)

comment:11 @xknown4 years ago

The are two problems that may need to be addressed:

  • setlocale only works if the locale name matches a system locale.
  • two different .mo files may not be used when they contain translations of the same domain. For example, "$locale.mo" and "ms-$locale.mo" (both referenced in the core).

comment:12 @linushoppe4 years ago

What do we have to change to see this patch in one of the next WP-releases?

@linushoppe4 years ago

Patch against /wp-includes/l10n.php in WP 3.2.0

comment:13 @wet4 years ago

  • Cc r.wetzlmayr@… added

comment:14 @pavelevap4 years ago

  • Cc pavelevap@… added

comment:15 @kurtpayne4 years ago

  • Cc kurtpayne added

comment:16 @ocean904 years ago

  • Cc ocean90 added
  • Component changed from General to I18N
  • Type changed from defect (bug) to enhancement

comment:17 @MHagemeister4 years ago

  • Cc MHagemeister added

comment:18 @MHagemeister4 years ago

Love this patch! I get similar performance improvements like mwidmann.

comment:19 @scribu4 years ago

  • Summary changed from Using pomo for translations is very inefficient to Use native gettext library when available

Changing title to avoid confusion with #17128

comment:20 @scribu4 years ago

  • Component changed from I18N to Performance

comment:21 @scribu4 years ago

  • Keywords early added
  • Milestone changed from Awaiting Review to Future Release

comment:22 @scribu4 years ago

  • Milestone changed from Future Release to 3.4

comment:23 @sirzooro4 years ago

  • Cc sirzooro added

comment:24 @Bueltge4 years ago

  • Cc frank@… added

comment:25 @swissspidy4 years ago

  • Cc hello@… added

comment:26 @mikeschinkel4 years ago

  • Cc mikeschinkel@… added

comment:27 @nacin4 years ago

I think it would be interesting to see this abstracted a bit cleaner. Perhaps, as westi indicated, a separate class.

For example, we already have Translations and NOOP_Translations. We should introduce a new _Translations class.

In the process, we should probably split the classes into three files.

Ideally all of the relevant code should occur in wp-includes/pomo, and none of it in l10n.php. That is simply our API layer.

comment:28 @scribu4 years ago

  • Keywords needs-patch added; has-patch removed

Agreed. This should be just another translation implementation, like we have multiple implementations for HTTP requests.

comment:29 @westi4 years ago

  • Milestone changed from 3.4 to Future Release

Moving back to Future Release until we have the scope discussion for 3.4

@linushoppe4 years ago

native gettext-support php-file

comment:30 @linushoppe4 years ago

For WordPress 3.3 we did a complete rewrite of the patch to make wordpress
work with PHP's native gettext-support.

The Patch now emulates the PoMo-API and creates an alias from the old class
named "MO" to itself - you just have to make sure that its loaded *before*
pomo is loaded.

File that were modified:

wp-settings.php (new verson attached to the bug report)

Files that were added

wp-includes/pomo/native.php (attached to the bug report)

I've added a patch to create the new native.php and to modify the wp-settings.php.

@linushoppe4 years ago

new wp-settings.php for native gettext-support

comment:31 @xeno0104 years ago

I've written a plugin -> WP-Performance-Gettext-Patch

I use a modified version of your patches:

  • Add Cacheing-Control
  • other minor changes

Linux with Plugin 3x faster

comment:32 @linushoppe4 years ago

xeno010, can you please explain what "other minor changes" your plugin has and why you've added caching? As far as I know, gettext itself has an own cache-implementation.
Besides, installing a plugin is not as powerful as patching the wordress-sourcecode and only some users would profit from the changes.

comment:33 @nacin4 years ago

I closed http://wordpress.org/extend/plugins/wp-performance-gettext-patch/ for download as it applies a patch to core, which is dangerous and an all-around bad idea. Its source is still viewable and accessible in SVN.

comment:34 @pavelevap4 years ago

I tested mentioned plugin and encountered great memory gains. My WP website with installed plugin used 28 MB memory and without this plugin more than 34 MB. This problem is real pain for localized versions, because there are more and more strings with every new release...

comment:35 @noah.williamsson4 years ago

  • Cc noah.williamsson added

I ran into the l10n performance issue initially reported in #17128 on an Ubuntu 10.04 LTS (PHP 5.3.2, APC 3.1.3) system running Wordpress 3.3.1.

I did some testing and meassured the response time for serving a single post page.
Here are my numbers for Wordpress 3.3.1.

Feature Response time Performance hit
Stock 3.3.1, WPLANG not set 139ms 0%
WPLANG set to 'sv_SE' 255ms 45% slower
WPLANG set + patch from #17268 145ms 4% slower

As can be seen, the patch (native_gettext.patch) improved page response times by 43% compared to a stock Wordpress 3.3.1 installation where WPLANG is used.

I would like to see this patch merged for 3.4.

comment:36 @scribu4 years ago

I just noticed that native_gettext.patch has the following comment:

@license http://creativecommons.org/licenses/by-sa/3.0/de/ Creative Commons Attribution-Share Alike 3.0 Germany

This license is not compatible with the GPL license and therefore can not be included in WordPress Core.

Please consider changing the license to GPL or removing that comment altogether, in which case the GPL is implied.

Last edited 4 years ago by scribu (previous) (diff)

comment:37 @ryan4 years ago

Any native support we build in should fallback to the current pomo setup if the desired locale is not fully supported. The presence of the gettext extension does not mean that any given locale can be served. Type "locale -a" on your Mac OS or Linux box and see which ones you have. Some hosts install very few locales, often just C, POSIX, and en_US*. Different environments can have different names for the same locale. Also, the directory structure required by gettext is not compatible with what we use now.

comment:38 @Chouby4 years ago

  • Cc frederic.demarle@… added

comment:39 @gr0b14 years ago

FYI I get a 11% memory decrease and I if i benchmark my homepage: I go from around 18 req/s to 30+ req/s with this patch.
There is a typo on line 217: "$domain" should be "$Domain". (causes � bug)
Anyway thanks. And please push this out soon, even if turned off by default.

Version 1, edited 4 years ago by gr0b1 (previous) (next) (diff)

comment:40 @nacin4 years ago

  • Milestone changed from Future Release to 3.4

Not promising anything, but we may just end up doing this.

@linushoppe4 years ago

patch against WP 3.3.1, corrects the $Domain-Bug

comment:41 @linushoppe4 years ago

I've added a new version of the patch. This patch was created against Wordpress 3.3.1. It fixes the $Domain-Bug mentioned.
More information: http://oss.tiggerswelt.net/wordpress/3.3.1/

comment:42 @linushoppe4 years ago

By the way, the patch ist GPL now.

comment:43 @scribu4 years ago

  • Keywords has-patch added; needs-patch removed

comment:44 @pavelevap4 years ago

I tried to test latest patch with WP 3.3.1 and received following error:

Warning: Cannot redeclare class MO in .../wp-includes/pomo/native.php on line 242

comment:45 @Devstorm4 years ago

  • Cc Devstorm added

comment:46 @nacin4 years ago

  • Milestone changed from 3.4 to Future Release

Not going to happen for 3.4. Let's see what we can continue to cook up. If we can come up with a sturdy patch that doesn't affect installs without gettext, and handles issues related to LC_* values, I think we'd be open to reconsidering it.

comment:48 @tszming3 years ago

+1 for making this patch merged into the core as soon as possible.

(Have done a quick profiling and the those *textdomain* functions really used too many CPU cycles)

comment:49 @knutsp3 years ago

  • Cc knut@… added

comment:50 @meloniq3 years ago

  • Cc meloniq@… added

comment:51 @mbijon3 years ago

  • Cc mike@… added

comment:52 @tszming22 months ago

I have been using the patch for years, any chance to get it merged?

comment:53 @nacin21 months ago

  • Component changed from Performance to I18N
  • Focuses performance added

comment:54 @greencp21 months ago

linushoppe, I included your gettext implementation into my new plugin WP Performance Pack and added you to the list of contributors. It can be used without any chances to core files by using override_load_textdomain.

The plugin needs some more checks if native gettext can be used, but it works for me. Only changes I made to your implementation were adding checks for NOOP_Translations in merge_with and merge_originals_with.

comment:55 @wonderboymusic16 months ago

  • Keywords needs-refresh added

This causes fatal errors in unit tests

comment:56 @benjamin416 months ago

Note that the gettext cache isn't flushed until the restart of the web server [1], but it should be cached after any update. Restarting the web server isn't really an option in shared hosting :)

[1] There seem to be workarounds:

comment:57 @nacin16 months ago

What if we went to an associative PHP array for our strings?

return array( 'English string' => 'translated string' );

How would performance look? We'd take a bit of a hit for loading the file, but it'd avoid the overhead of parsing a MO file, could be opcode-cached, etc.

ryan, nikolay, and I have discussed this as an idea over the years, but we've never tried it out.

comment:58 @greencp16 months ago

I just saw, I didn't mention it in my last comment: I wrote an alternative MO file Reader (read only) that is faster and uses less memory than the current implementation. That's how I started my plugin. http://wordpress.org/plugins/wp-performance-pack/ With caching enabled performance hit is almost minimal compared to not translated WordPress setups.

comment:59 @tszming16 months ago

@benjamin4 , the patch by linushoppe check the md5sum of the file everytime and assign to the domain, so it won't be a problem for shared hosting. (We have changed to filemtime to improve performance.)

comment:60 @markjaquith12 months ago

Okay. To advance this ticket we need someone to explore Nacin's static array mapping idea. Any volunteers?

comment:61 @luciole1352 months ago

I use this hack for 3 years without any problems on my web site.
I have not tested it for a multisite.

To do this I added to wp-config.php file the following lines:

/* Utilisation de gettext pour l'internationalisation */
require( ABSPATH . '/wp-includes/pomo/native.php');
Note: See TracTickets for help on using tickets.