WordPress.org

Make WordPress Core

Opened 3 years ago

Last modified 3 months ago

#17268 new enhancement

Use native gettext library when available

Reported by: linushoppe Owned by:
Milestone: Future Release Priority: normal
Severity: normal Version:
Component: I18N Keywords: early has-patch
Focuses: performance Cc:

Description

Here you say that the GNU gettext-Framework is used. Exactly, "pomo" (file: wp-includes/l10n.php) is a complete own php-implementation of the gettext-program.

I've added a patch to solve this problem. Maybe it is not very good, but it works. On my wordpress-sites, the used php-memory returns from about 65% to about 12% and the site is running much faster when patching wp-includes/l10n.php.

I know that gettext is not available on every wordpress-installation, but when it's available, it should be used.

Sorry for my bad english, I'm german.

Attachments (6)

wp_gettext.patch (4.1 KB) - added by linushoppe 3 years ago.
Patch to reduce the used php-memory
wp_gettext_v2.patch (4.3 KB) - added by linushoppe 3 years ago.
wp_gettext_v3.patch (4.0 KB) - added by linushoppe 3 years ago.
Patch against /wp-includes/l10n.php in WP 3.2.0
native.php (6.6 KB) - added by linushoppe 2 years ago.
native gettext-support php-file
wp-settings.php (9.7 KB) - added by linushoppe 2 years ago.
new wp-settings.php for native gettext-support
native_gettext.patch (7.9 KB) - added by linushoppe 2 years ago.
patch against WP 3.3.1, corrects the $Domain-Bug

Download all attachments as: .zip

Change History (60)

linushoppe3 years ago

Patch to reduce the used php-memory

comment:1 duck_3 years ago

Related: #17128

comment:2 johnbillion3 years ago

  • Cc johnbillion@… added

comment:3 mwidmann3 years ago

After applying this patch I was able to handle a substancial amout more requests in the same amount of time.

Running the same command command in all cases:

siege -c5 -t30s -b http://localhost

Results:

Version trans/secs  succ. transactions
3.1.2 tagged 2.35 69
with patch from #17128 2.92 86
with just this patch 3.40 100
with this and the other patch 3.50 102

This change is a huge winner if gettext is installed. Would be nice to have this in 3.2.

comment:4 follow-up: scribu3 years ago

Looks nice, but I'm wondering if it wouldn't be even faster if we had two definitions of each WP translation function:

if (function_exists ('dgettext')) {
  require_once('translate-gettext.php');
} else {
  require_once('translate-pomo.php');
}

translate-pomo.php would contain the current implementation in WP, while translate-gettext.php would contain the more efficient, gettext based definitions.

comment:5 westi3 years ago

It looks like this patch doesn't handle translation with context at all and also I don't understand the need for all that copying of files.

I think this would be better implemented as an alternative implementation of a Translations class - probably could even be done in a plugin then maybe

comment:6 Philipp15b3 years ago

  • Cc hebipp1@… added

comment:7 in reply to: ↑ 4 linushoppe3 years ago

Replying to scribu:

Looks nice, but I'm wondering if it wouldn't be even faster if we had two definitions of each WP translation function:

if (function_exists ('dgettext')) {
  require_once('translate-gettext.php');
} else {
  require_once('translate-pomo.php');
}

translate-pomo.php would contain the current implementation in WP, while translate-gettext.php would contain the more efficient, gettext based definitions.

Very nice idea. I just wrote this patch to make WP faster, I had no time to make the patch "good-looking" and useful.

linushoppe3 years ago

comment:8 linushoppe3 years ago

  • Cc linus.hoppe@… added

We've created a new version of the patch. The new one (patch against l10n.php of Wordpress 3.1.3) can handle php contexts (msgctxt) via $context. We hope that it helps you to integrate this patch into one of the next wordpress versions in order to reduce the used php-memory.

comment:9 aphrodite3 years ago

Hi !

Be carefull on Cpanel servers, the folder creation and chmods generates a mkdir error. Added a @mkdir on line 377, and on some servers the folder creation may be mannual...

Moreover, translations are not directly applied. This is a little issue, but can be confusing for users. Must refresh the pages to have the translation applied ;)

Last edited 3 years ago by aphrodite (previous) (diff)

comment:10 scribu3 years ago

WP has a helper function for that called wp_mkdir_p().

Last edited 3 years ago by scribu (previous) (diff)

comment:11 xknown3 years ago

The are two problems that may need to be addressed:

  • setlocale only works if the locale name matches a system locale.
  • two different .mo files may not be used when they contain translations of the same domain. For example, "$locale.mo" and "ms-$locale.mo" (both referenced in the core).

comment:12 linushoppe3 years ago

What do we have to change to see this patch in one of the next WP-releases?

linushoppe3 years ago

Patch against /wp-includes/l10n.php in WP 3.2.0

comment:13 wet3 years ago

  • Cc r.wetzlmayr@… added

comment:14 pavelevap3 years ago

  • Cc pavelevap@… added

comment:15 kurtpayne3 years ago

  • Cc kurtpayne added

comment:16 ocean903 years ago

  • Cc ocean90 added
  • Component changed from General to I18N
  • Type changed from defect (bug) to enhancement

comment:17 MHagemeister2 years ago

  • Cc MHagemeister added

comment:18 MHagemeister2 years ago

Love this patch! I get similar performance improvements like mwidmann.

comment:19 scribu2 years ago

  • Summary changed from Using pomo for translations is very inefficient to Use native gettext library when available

Changing title to avoid confusion with #17128

comment:20 scribu2 years ago

  • Component changed from I18N to Performance

comment:21 scribu2 years ago

  • Keywords early added
  • Milestone changed from Awaiting Review to Future Release

comment:22 scribu2 years ago

  • Milestone changed from Future Release to 3.4

comment:23 sirzooro2 years ago

  • Cc sirzooro added

comment:24 Bueltge2 years ago

  • Cc frank@… added

comment:25 swissspidy2 years ago

  • Cc hello@… added

comment:26 mikeschinkel2 years ago

  • Cc mikeschinkel@… added

comment:27 nacin2 years ago

I think it would be interesting to see this abstracted a bit cleaner. Perhaps, as westi indicated, a separate class.

For example, we already have Translations and NOOP_Translations. We should introduce a new _Translations class.

In the process, we should probably split the classes into three files.

Ideally all of the relevant code should occur in wp-includes/pomo, and none of it in l10n.php. That is simply our API layer.

comment:28 scribu2 years ago

  • Keywords needs-patch added; has-patch removed

Agreed. This should be just another translation implementation, like we have multiple implementations for HTTP requests.

comment:29 westi2 years ago

  • Milestone changed from 3.4 to Future Release

Moving back to Future Release until we have the scope discussion for 3.4

linushoppe2 years ago

native gettext-support php-file

comment:30 linushoppe2 years ago

For WordPress 3.3 we did a complete rewrite of the patch to make wordpress
work with PHP's native gettext-support.

The Patch now emulates the PoMo-API and creates an alias from the old class
named "MO" to itself - you just have to make sure that its loaded *before*
pomo is loaded.

File that were modified:

wp-settings.php (new verson attached to the bug report)

Files that were added

wp-includes/pomo/native.php (attached to the bug report)

I've added a patch to create the new native.php and to modify the wp-settings.php.

linushoppe2 years ago

new wp-settings.php for native gettext-support

comment:31 xeno0102 years ago

Hallo,
I've written a plugin -> WP-Performance-Gettext-Patch

I use a modified version of your patches:

  • Add Cacheing-Control
  • other minor changes

Benchmark:
Linux with Plugin 3x faster

comment:32 linushoppe2 years ago

xeno010, can you please explain what "other minor changes" your plugin has and why you've added caching? As far as I know, gettext itself has an own cache-implementation.
Besides, installing a plugin is not as powerful as patching the wordress-sourcecode and only some users would profit from the changes.

comment:33 nacin2 years ago

I closed http://wordpress.org/extend/plugins/wp-performance-gettext-patch/ for download as it applies a patch to core, which is dangerous and an all-around bad idea. Its source is still viewable and accessible in SVN.

comment:34 pavelevap2 years ago

I tested mentioned plugin and encountered great memory gains. My WP website with installed plugin used 28 MB memory and without this plugin more than 34 MB. This problem is real pain for localized versions, because there are more and more strings with every new release...

comment:35 noah.williamsson2 years ago

  • Cc noah.williamsson added

I ran into the l10n performance issue initially reported in #17128 on an Ubuntu 10.04 LTS (PHP 5.3.2, APC 3.1.3) system running Wordpress 3.3.1.

I did some testing and meassured the response time for serving a single post page.
Here are my numbers for Wordpress 3.3.1.

Feature Response time Performance hit
Stock 3.3.1, WPLANG not set 139ms 0%
WPLANG set to 'sv_SE' 255ms 45% slower
WPLANG set + patch from #17268 145ms 4% slower

As can be seen, the patch (native_gettext.patch) improved page response times by 43% compared to a stock Wordpress 3.3.1 installation where WPLANG is used.

I would like to see this patch merged for 3.4.

comment:36 scribu2 years ago

I just noticed that native_gettext.patch has the following comment:

@license http://creativecommons.org/licenses/by-sa/3.0/de/ Creative Commons Attribution-Share Alike 3.0 Germany

This license is not compatible with the GPL license and therefore can not be included in WordPress Core.

Please consider changing the license to GPL or removing that comment altogether, in which case the GPL is implied.

Last edited 2 years ago by scribu (previous) (diff)

comment:37 ryan2 years ago

Any native support we build in should fallback to the current pomo setup if the desired locale is not fully supported. The presence of the gettext extension does not mean that any given locale can be served. Type "locale -a" on your Mac OS or Linux box and see which ones you have. Some hosts install very few locales, often just C, POSIX, and en_US*. Different environments can have different names for the same locale. Also, the directory structure required by gettext is not compatible with what we use now.

comment:38 Chouby2 years ago

  • Cc frederic.demarle@… added

comment:39 gr0b12 years ago

FYI I get a 25% memory decrease and I if i benchmark my homepage: I go from around 18 req/s to 30+ req/s with this patch.
There is a typo on line 217: "$domain" should be "$Domain". (causes � bug)
Anyway thanks. And please push this out soon, even if turned off by default.

Last edited 2 years ago by gr0b1 (previous) (diff)

comment:40 nacin2 years ago

  • Milestone changed from Future Release to 3.4

Not promising anything, but we may just end up doing this.

linushoppe2 years ago

patch against WP 3.3.1, corrects the $Domain-Bug

comment:41 linushoppe2 years ago

I've added a new version of the patch. This patch was created against Wordpress 3.3.1. It fixes the $Domain-Bug mentioned.
More information: http://oss.tiggerswelt.net/wordpress/3.3.1/

comment:42 linushoppe2 years ago

By the way, the patch ist GPL now.

comment:43 scribu2 years ago

  • Keywords has-patch added; needs-patch removed

comment:44 pavelevap2 years ago

I tried to test latest patch with WP 3.3.1 and received following error:

Warning: Cannot redeclare class MO in .../wp-includes/pomo/native.php on line 242

comment:45 Devstorm2 years ago

  • Cc Devstorm added

comment:46 nacin2 years ago

  • Milestone changed from 3.4 to Future Release

Not going to happen for 3.4. Let's see what we can continue to cook up. If we can come up with a sturdy patch that doesn't affect installs without gettext, and handles issues related to LC_* values, I think we'd be open to reconsidering it.

comment:48 tszming23 months ago

+1 for making this patch merged into the core as soon as possible.

(Have done a quick profiling and the those *textdomain* functions really used too many CPU cycles)

comment:49 knutsp23 months ago

  • Cc knut@… added

comment:50 meloniq20 months ago

  • Cc meloniq@… added

comment:51 mbijon18 months ago

  • Cc mike@… added

comment:52 tszming4 months ago

I have been using the patch for years, any chance to get it merged?

comment:53 nacin3 months ago

  • Component changed from Performance to I18N
  • Focuses performance added

comment:54 greencp3 months ago

linushoppe, I included your gettext implementation into my new plugin WP Performance Pack and added you to the list of contributors. It can be used without any chances to core files by using override_load_textdomain.

The plugin needs some more checks if native gettext can be used, but it works for me. Only changes I made to your implementation were adding checks for NOOP_Translations in merge_with and merge_originals_with.

Note: See TracTickets for help on using tickets.