Support Partial core upgrades

[dd32]

It'd be nice that for point releases, Partial core upgrades were supported.

Ie. Instead of copying 1,000 files, 999 of which are identical, only a single file is updated on the system.

This could be done in a few ways.

1. Zip archive only contains changed files
2. The contents of the files in the zip and the local filesystem are compared before copying (Or even just the CRC32's)

are the first 2 which come to mind.

There'd also be a rather large speed up in terms of not writing temporary files to the filesystem first.. Which in itself, Takes a long time, and can make some upgrades take twice as long.. (but is needed)

I'd be willing to work on the implementation of it as well.

[dd32]

As well as comparing the file contents, you'd also check for new files of course.. and still iterate over the deleted files list..

I'm not sure this should be supported for plugins (Well, The 2nd option is a possibility, but that'll require major changes in the process) quite yet, But could be implemented for plugins in the future.. I'd prefer to keep this disucssion centered around core upgrades for now.. As thats where i feel it'll be more appropriate.

[westi]

I think this is a really good idea for point releases and point releases only.

It needs to do something to confirm that the rest of the files are OK too though.

I did suggest this during last nights meetup :-)

[dd32]

I think this is a really good idea for point releases and point releases only.

I'm not too sure of that, Take this example for example:

$list = $zip->get_list_of_files();
foreach $list as $file 
   if $partial_upgrade_enabled &&
      file_exists(ABSPATH . $file) &&
      file_get_contents(ABSPATH . $file) === $zip->extract_as_string($file)
   then
      Continue;
   ..Extract the file and overwrite the local one here.. 
   ..Either the new file is different, or is a new file..
end foreach;

It saves IO time basically, only changed files get written.

If the CRC32 route was taken, it could be even faster as the Zip decompression wouldn't have to be done.

It needs to do something to confirm that the rest of the files are OK too though.

Thats where the CRC32 or file contents check(as above) would be used

I dont think having a zip with only the changed files is a good solution, was just an offered one.

I did suggest this during last nights meetup :-)

I only skim read it.. Must've been where the idea came from, I dont think there' was that much chat about it if i missed it?

[AaronAsAChimp]

Does it have to be this complicated?

What if this was handled when the packages were created?
For example, you would create two packages wordpress-2.6.4.zip and wordpress-2.6.4-from-prev.zip.

So, if you had version 2.6.2 the packages wordpress-2.6.3-from-prev.zip and wordpress-2.6.4-from-prev.zip would be downloaded and unzipped in that order.

SVN already knows what files have changed, creating the packages wouldn't be that difficult. It would also save bandwidth

[dd32]

There was a bit of chat on IRC about the options.

Your idea of multiple step zips is actually more complex than the offered suggestions.

The main thing against partial upgrade zips is the potential of other files being changed.

One solution which was suggested was:

• API on wordpress.org to return a list of md5's of files
• WP then checks to see if all files match, And if only the changed files differ
  • Download the Partial upgrade zip
• Else
  • Download the full upgrade zip
• Continue as normal.

Basically the thinking is, Dont trust the install the user is running, dont count on all the files being exactly as we expect them to be.

I'm going to write up a proof of concept patch based off the psuedocode above, just to show the potential implications of that method, and how much of a speedup it may bring.

One thing that many people are not sure about is where the main bottleneck is

• Downloading the large Zip file
• The Zip Decompression (This is mainly memory usage IMO)
• The IO time
• The 2nd IO time

Personally i think its mainly the IO..

[Denis-de-Bernardy]

One thing that many people are not sure about is where the main bottleneck is

• Downloading the large Zip file
• The Zip Decompression (This is mainly memory usage IMO)
• The IO time
• The 2nd IO time

Personally i think its mainly the IO..

Oh, I can give you some feedback on that. Keep in mind that my own zip is 7M+. :-P

• download takes a few seconds
• cleaning up after a failed install is very very very slow (see #10913)
• unzip is *very* slow and resource hungry
• copying actually is pretty fast
• cleaning up is, again, very very very slow

What frequently happens on my end with WP 2.8.x sites one of the following:

1. the zip gets downloaded
2. unzipping occurs, takes a lot of time, FTP times out about when it's done
3. WP starts to copy, fails due to the FTP time out, and bails
4. WP can't clean-up
5. User tries again
6. the zip gets downloaded
7. FTP times out during the clean-up
8. User tries again
9. the zip gets downloaded
10. FTP finishes up the clean-up
11. Return to step 2 (until user decides it's hopeless)

The way I work around it is by hooking into update_feedback. On each feedback, I force flush the output so the user gets some kind of feedback. I then disconnect the FTP (some hosts disallow overriding time-outs, meaning that #10407 or the newer ticket needs more fixing), I reconnect the FTP, and I reconfigure the FTP so it times out after 10 minutes on the host where that works. So far it's pretty successful.

I considered uploading partial zips, but it seemed too risky (can't trust the version sent by the site, especially after a failed upgrade); the same goes for having a special file with the md5 of every file. Keeping the md5 in options is no good either -- if a manual upgrade occurs, it breaks.

That is to say, I think it's more important to fix #10913, which is the main culprit for failed upgrades.

[Denis-de-Bernardy]

I'd picture some UI for this one if we move forward: a checkbox that offers this option, in the event someone wants to reinstall.

[Denis-de-Bernardy]

actually, nevermind that last remark. there is no need for UI if we stick to upgrades, and proceed with all of the files for reinstalls.

[nacin]

In [18224]:

Update Core UI changes:

• Update Now instead of Update Automatically.
• The first core update is now a primary button.
• Hide the version number from the 'You are about to install WP in English' warning for development versions, as it lies and suggests you will be downgraded.

see #10611.

[nacin]

In [18225]:

Core support for partial updates. see #10611.

[nacin]

[18224] were changes requested by Jane and also a bug I noticed.

[18225] was the result of a desire to have an absolute minimum amount of code go into core for this feature. Indeed, there are only two changes: A new API that delivers serialized responses modeled after the object that WordPress used to construct using the newline response (thus minimizing the changes required for the upgrader), and a bit of code in the upgrader to select which of four packages to download.

The packages are as follows:

• partial - This zip uses the reported $wp_version to deliver an in-branch partial zip. If you're on 3.1.x and are upgrading to 3.1.y, you'll get something like wordpress-3.1.y-partial-x.zip from the API. Partial builds won't be used when you request a re-install. I'm working on a separate patch now that will further validate on the core side that the right partial build is being used.

• new_bundled - This zip has the newest bundled theme. The version in which the latest bundled theme was added is also included in the response. Core will no longer touch any existing theme, it will only deliver a new one, and only if CORE_UPGRADE_SKIP_NEW_BUNDLED isn't set. Core will never touch or restore an existing plugin.

• no_content - This zip contains no plugins or themes. It will however include /languages once the localized build system supports these new builds (and until we have language packs, which I'm aiming for 3.3).

• full - This zip contains everything. Core should never fall back to this once the build process is in place.

This ticket can be closed as fixed once:

• The new build process on WP.org is in place, functional, and tested. It's important to note that none of this comes into play until 3.2.1, so I'm establishing a system to test this on my sandbox. (Beta Tester updates rely on nightly builds.)
• version-check/1.6-beta/ is moved to 1.6.

[nacin]

In [18227]:

Extra check for partial updates: Make sure the API response matches the current version of WP. see #10611.

[nacin]

To update the status of this:

• The new build process is complete, and locally tested. Results are fantastic. I'll be getting it onto WP.org early next week.

• version-check/1.6 will remain at beta for now.

• Still need to re-do the localized build process. That should be fun.

[nacin]

Accepting this for now. This ticket can remain open past RC1.

[nacin]

Closing this. See #17759.