3.0 package size will cause memory limit problems

[nacin]

I've seen multiple reports that the 3.0 download is too much for the 32MB limit. I can also reproduce this on a reputable shared host.

We do increase to 256MB once the package is downloaded, to unzip the file. But that doesn't help here -- download_url is struggling. We also can't add anything to update-core of course, as we haven't even downloaded the package yet.

After talking this over with MarkJaquith and westi, this is fixed in [14491], where we will give anyone with the manage_options capability a 256MB memory limit in wp-admin.

That said, this will still blow up on many hosts and configurations with the default WP memory limit (that'd be 32MB) when upgrading from 2.9.2 to 3.0. Telling everyone who experiences the error to add a constant is a bad idea from basically every standpoint (starting with PR) and will severely hurt 3.0 adoption.

So, we need to prune the 3.0 download package. For reference, it is currently ~3.2MB, while 2.9 was only 2.5MB, 2.8 was 2.4MB, and 2.7 was 1.9MB. Obviously, a lot of this is due to the merge.

I can't come up with many things we can remove, but I can start with two folders of items: CodePress (I was planning to do this first-thing in 3.1 anyway), and the theme-compat folder (fall back to Kubrick, and if it doesn't exist, tough luck).

If that's not enough to get us under the bar, we'll need to get more creative. As a start, image gradients in wp admin that can become CSS3 (we just did this with the admin headers), and other images that can be compressed and/or become sprites. I have other crazy ideas I'll keep to myself for now.

[azaozz]

Next step would probably be to remove all .dev.js and .dev.css and minify the php (i.e. remove phpdoc and comments), then have a separate "developer" package that includes them. Although I would hate to do that.

Another solution seems to be to add support for multi-step download and split the package in 2 (too late to try this for 3.0).

[nacin]

I would hate to do that as well.

The crazy idea I had would be to release 2.9.3 with [14491] backported, then program api.wordpress.org to return 2.9.3 as the most recent version available, until the site reports its version as 2.9.3. That might need to be what we implement in the future for < 3.0 installs, assuming the package does indeed continue to grow in future releases.

I imagine we can avoid all of this though, at least now. We should be able to remove enough cruft first.

[wpmuguru]

Replying to azaozz:

Next step would probably be to remove all .dev.js and .dev.css

Let's try that and see how much it reduces the zip size.

In past versions the typical difference is size between WP & MU zips was ~100K.

The bulk of the increased size is the images in TwentyTen. Total space used by the images is running in the range of 400K & the compression rate is very low on png images. Is there a way we could send out TwentyTen in WP with only the default image(s) and then offer an upgrade that includes the other images through WP.org extend?

[hakre]

Implementing a build process that is creating different packages (minified, classic etc.) isn't a bad idea per se. Maybe too professional but I think the team can cope with that :P

For the images: Use JPG instead of PNG that works much better on compression for fotos. That should do the trick for the size.

[nacin]

Photos should be JPG, yes. Let me try that, I can also pngcrush others.

Going to work on this today. Really don't want to remove the development scripts, let's see if codepress and image compression can shrink the nightly enough. Next step would be theme-compat.

[nacin]

Now that I check, Twenty Ten headers are already jpegs. So not much to gain there.

[nacin]

(In [14527]) Remove codepress dir. see #13307.

[azaozz]

Can also remove the concatenated uncompressed wp-tinymce.js (258KB) and fall back to separate loading if for some reason compression cannot be used. Running tests now will have a patch later. That would trim the zip by about 60KB (as much as one header image).

[nacin]

On my server at least, removing the CodePress directory was surprisingly enough to get it below whatever the threshold is, and enable a download at a 32MB memory limit. I tested that on the four HTTP transports that work on my server (all but httpext).

I still think we should prune the package more, but I want to say we don't need to take drastic measures. Maybe just work on image compression and remove theme-compat just to get it down a little more.

[hakre]

We could remove akismet and dolly. Akismet isn't useable w/o an API key at all, so manual actions are required for use anyway and dolly, well it is useless in many senses.

If we go with a build script, we will have much more options here.

[nacin]

Replying to hakre:

We could remove akismet and dolly.

We're not unbundling either plugin.

[hanifb]

We could always separate content and code,

Ex: The update script first downloads the latest-core.zip, extracts it, installs it - then redirects you to a page where latest-content.zip extracts and installs.

The latest-content.zip package should be independent so if by some magic, the install would fail, WP would not break and can resume the update process.

[azaozz]

Patch is too big, attaching it zipped.

[azaozz]

The wp-tinymce.js is there mostly for completeness. That's the uncompressed version of wp-tinymce.js.gz. It would speed up loading on browsers that don't support gzip compression (or the support is buggy) but I don't think there are any browsers currently in use that can run TinyMCE and don't support gzip.

[WraithKenny]

I've no idea how hard this is, but could the packages for WP 3 and 2010 be separate? First DL and install WP proper, then as part of install, DL and install 2010, then activate, so that the packages are smaller.

[blepoxp]

If your looking for any small change you can find, I uploaded all images in trunk to ​http://www.smushit.com/ysmush.it/ just for kicks and it claims to have "Smushed 7.07% or 52.39 KB from the size of your image(s)."

I've don't know how the service technically works and acknowledge that replacing the files it 'smushed' would be tedious but I thought I'd throw it out there. I was honestly hoping it would compress it more.

[paulgregory]

Whilst TwentyTen should be part of a fresh WP3 install, I am not at all convinced it should be included in the upgrade - which appears to be where the size problem lies.

Anyone with 2.9.x upgrading to 3.0 already has a theme, and is unlikely to want to change theme immediately on upgrade. So I think the TwentyTen download should be excluded from the upgrade package, and available separately for people that have a need for it.

This appears to me to be the simplest solution to the problem - cutting out the part that existing WordPress users need the least.

[nacin]

Replying to paulgregory:

Whilst TwentyTen should be part of a fresh WP3 install, I am not at all convinced it should be included in the upgrade - which appears to be where the size problem lies.

Aside from disagreeing with that point, we currently use the same package for both installs and upgrades. I don't think we should change that process.

---

I have crunched some numbers. When reinstalling 2.9 on a shared host, memory usage immediately after download_url() peaked at 26.76 MB, and was at 19.56 MB just before the download_url() call. download_url() thus used 7.2 MB of memory for the 2.5 MB package, or 2.88 MB of memory per MB of file.

For an upgrade from 2.9 to 3.0 (sans CodePress etc.), memory usage was 22.52 MB and 31.39 MB, respectively. That's 8.87 MB of memory for the 3.1 MB download package, or 2.77 MB of memory per MB of file.

When using the beta 2 build, memory usage was slightly higher, *just* enough to go over the default memory_limit threshold.

So, I still think we need to do some pruning. A blog with a few more plugins than this one will probably make it over the edge in the current nightly. My thoughts:

• Crush all PNGs.
• Squeeze a few KB out of each JPG header image.
• Remove theme-compat. Also prevents sites currently falling back to an i18n version of Kubrick from having things mucked up.

And see where we're at. Also, consider removing the non-gzip tinymce.js. We should also backport [14491] anyway.

[nacin]

We could also package Twenty Ten 1.0 with say two headers, then include Twenty Ten 1.1 in the theme directory, with more headers. I'd prefer the 2.9.3 -> 3.0 process over that however.

[masonjames]

Replying to nacin:

We could also package Twenty Ten 1.0 with say two headers, then include Twenty Ten 1.1 in the theme directory, with more headers. I'd prefer the 2.9.3 -> 3.0 process over that however.

+1 for the 2.9.2 -> 3.0 idea. I'm for that over separating the download/upgrade into two parts. Twenty Ten 1.1 isn't a horrible idea either though. The additional header images are just that - additional.

[dremeda]

+1 for the headers in 2010 1.1.

We could do image replacement for all images used in the install/upgrade as considered in comments above.

[nacin]

(In [14718]) Remove autocomplete and codepress. for autocomplete, see #13283. for codepress, see #13307.

[ozh]

Note that there will be probably more problems with localized versions of WP, as they'll come with the .mo & .po bundled...

[Jonnyauk]

Here is a quick fix - I have reduced the headers directory in twentyten down from 459k down to 217k, representing an overall saving of 242k.

​http://jonnya.net/transfer/wp30-twentten-headers-compressed.zip

The images have been re-compressed, with negligible loss in quality. A slightly better image quality could be achieved by compressing from the original source images, rather than already compressed JPEG's.

For thank kind of saving (and for WordPress users to have that much choice from install), I think it's fair to sacrifice a-little image quality. Of-course, if a couple of header images were dropped from the selection - that would represent a good saving too!

[chipbennett]

Replying to masonjames:

Replying to nacin:

We could also package Twenty Ten 1.0 with say two headers, then include Twenty Ten 1.1 in the theme directory, with more headers. I'd prefer the 2.9.3 -> 3.0 process over that however.

+1 for the 2.9.2 -> 3.0 idea. I'm for that over separating the download/upgrade into two parts. Twenty Ten 1.1 isn't a horrible idea either though. The additional header images are just that - additional.

Doesn't the TwentyTen 1.1 approach merely delay the problem?

Once all the included header images are included in version 1.1, and that version gets packaged with subsequent WP packages, the problem re-appears. (Unless, of course, future WP versions between now and 2011 include the original, 1.0 version of TwentyTen.)

[ozh]

I'd hate to see comments and formatting being removed from the code and I think it'll never happen, however this might be a last resort option with unmodified classes, such as Simplepie, that are easily available in a clean format on the web.

Simplepie for instance is 388K and can be shrunk by more than 100K just removing comments. (Other classes such as JSON or PHPMailer may not be worth the hassle)

[Otto42]

This is just a thought, but some of the transports could be optimized.

The current approach for all of them is to download the entire file into memory, then write it out to a file. At least some of the transports have ways to do direct file downloads to a file stream instead.

In the Curl transport, for example, we always use CURLOPT_RETURNTRANSFER, to return the data. If we did something like this instead:

$fp = fopen('/some/file/name.zip', 'w');
$ch = curl_init('http://example.com/whatever');
curl_setopt($ch, CURLOPT_FILE, $fp);
...

Then it would download the file directly to the stream. No memory limit issues anymore.

If the http api had a new method such as, say, get_file(), then we could support this sort of thing. And the transports that couldn't do it could abstract it using the normal memory download-and-save approach.

This may not be feasible for 3.0's release, but I think it is the only real solution for the long term.

[azaozz]

Another last resort option: we could remove the Prototype.js/Scriptaculous (290KB) and load it from googleapis when queued from a plugin. It hasn't been used in core for a long time and don't think there are many plugins that use it.

[nacin]

Replying to azaozz:

Another last resort option: we could remove the Prototype.js/Scriptaculous (290KB) and load it from googleapis when queued from a plugin. It hasn't been used in core for a long time and don't think there are many plugins that use it.

Ryan or Mark suggested that today. Definitely something else to consider.

[ryan]

We removed interface.js, which saved quite a bit. We might remove importers per #13465.

[wpmuguru]

(In [14764]) move importers to plugin repo, see #13307

[nacin]

(In [14765]) Remove non-gzipped version of tinymce. Brace for impact. props azaozz. see #13307.

[nacin]

(In [14770]) Shake away old files during the upgrade process. see #13307.

[michaelh]

Regarding the removal of the wp-admin/import folder--don't people put other importers in there also?

So just in case, attaching patch to detail the standard 'core' files in that folder.

Wonder if there's a way to delete the import folder if it is empty after the standard files are removed?

[michaelh]

detail core files to delete in import folder

[nacin]

Importers should not be in wp-admin/import, plugins can register an importer.

[michaelh]

No way to actually do an import now.

After you install the new WordPress Import plugin (or any of the new importer plugins) you can't visit Tools->Import->WordPress as it just displays the plugin install screen reflecting that latest version is installed.

Also under Tools->Tools, the two links to convert categories to tags and tags to categories report "Cannot load importer".

[nacin]

We have decreased the package size by 7.5% since this ticket opened. That puts us at 2.79 MB, from 3.01 MB. (On disk, I'm seeing 2.9MB vs 3.2MB.) That's still a 14% increase over version 2.9-latest, however. Good progress.

We'll fix the import stuff.

[jane]

On an up note, it's still smaller than an mp3 of most songs. :)

[Denis-de-Bernardy]

There's 2.9MB worth of files in wp-includes/js. Surely some things could get dropped in there...

On a separate note... hint, hint:

​http://core.trac.wordpress.org/ticket/10611#comment:6

[ozh]

A lot of minified .js scripts are very similar in size to their .dev.js version (wp-ajax-response.dev.js 3k, wp-ajax-response.js 2.1k) and I'm not sure it's top priority to save a few extra bytes on scripts that are loaded in the admin area only.

So, an option could be to ship a full .dev.js + a minified .js only if that saves more than XX kbytes.

[ozh]

(very same comment applies pretty much to all the files in wp-admin/css)

[pavelevap]

Ad removed importers: How will be transfered localization from main WordPress .po and .mo files to specific plugins? In latest generated .pot were all (hundreds) translated strings removed.

Ad package size: Please note that localized versions also contain .po and .mo file (several hundreds kB).

[blepoxp]

There's a file called export copy.php in /wp-admin/includes right now. Is that related to another ticket? If not, i imagine it can be nuked.

[nacin]

Glenn, I'm thinking you did that locally while you were working on #10317? ​http://core.trac.wordpress.org/browser/trunk/wp-admin/includes/

[Otto42]

We could lose a few bytes and files by completing ticket #4967 and getting rid of the old unused RDF and RSS 0.92 feeds.

[ryan]

Losing tinymce.js and the importers bought us quite a bit of extra room. There will probably be some setups with lots of plugins and big po and mo files that will bust the limit, but there's only so much butchering we can do. The workaround is to do a manual upgrade. I think we've done enough here.

[nacin]

Agreed.

[Denis-de-Bernardy]

Replying to ryan:

The workaround is to do a manual upgrade. I think we've done enough here.

As much as you've probably merely fixed enough of the symptoms for now, you've left the causes untouched. I would know, in that I've been distributing a WP zip that is 3MB+ since WP 2.5. The causes are the huge number of files, WPFS, and the unzip() function. Those ought to be optimized. Heavily.

At 2.9M, you're at the point where problems start to appear on servers with the suhosin patch and multitudes of apache/php modules. You might not get many in 3.0, but I can guarantee you that you'll get quite a few as the zip's size increases.

I'm re-opening because, for users who experience the issue, a manual upgrade is the last thing they think of. We should add a notice on the upgrade screen. Along the lines of:

"The upgrader doesn't work on all servers. If the upgrade fails on your server, don't panic! Upgrade WP manually by following these instructions (with a link to a codex article), and things will be fine." (Note: the instructions should tell users to delete the .maintenance file.)

[nacin]

You're missing a significant amount of discussion from the past two weeks that went in to the decision to mark this as fixed.

We're getting memory limit errors *downloading* the package. There is nothing we can do other than alter the size of the package.

A stopgap is in place -- [14491] -- to prevent these errors once the user has reached 3.0, i.e. a 3.0 to 3.0.1 upgrade.

Yes, we are only fixing the symptoms. All other optimizations will take place in a new ticket on a new milestone.

[Denis-de-Bernardy]

Replying to nacin:

We're getting memory limit errors *downloading* the package. There is nothing we can do other than alter the size of the package.

Yeah... I convinced of the same in 2007.

Consider, for a moment, whether that that specific feedback might be erroneous, and whether what's really happening might not be what I described in #10611 -- i.e. related to clean-ups and unzips.

But heck, don't take my word for it. I've been upgrading -- and supporting -- thousands of sites in the past years, using a zip that is larger than WP 3.0, occasionally on awkward platforms. You haven't. (Those few who have been testing almost certainly have good servers.) Just saying...

Anyway, I've raised more tickets than I can count related to this problem. I've also fixed the issue on my end. (By splitting the zip in bite-sized chunks, if you need to know.) Considering your reply, the only answer I can think of is: good luck.

Adding a short notice in the WP upgrade screen won't hurt though...

[otto42]

The downloader can be fixed, as I suggested above. I'm pretty sure that is indeed the problem in a lot of cases. Downloading the file to memory before saving to disk is silly.

Still, that's a 3.1 thing, probably needs its own ticket.

[ryan]

Background: We unzip in memory because the only place we can directly write to disk is temp space, and we have to jump through hoops to reliably get a place in temp to write. File operations anywhere else must be abstracted, which means it usually must go through ftp. otto42's suggestion of leveraging transports that can stream to a file is a good one that we should follow up on, but as he mentioned that is a 3.1 thing. We've learned the hard way that seemingly simple changes to transport setup are likely to blow up in our faces.

There's not much more we can do here for 3.0, which goes RC soon. Addressing root causes will have to wait for 3.1.

[westi]

Not fixed in the future ;-)