WordPress.org

Make WordPress Core

Opened 5 years ago

Closed 3 years ago

#10611 closed task (blessed) (fixed)

Support Partial core upgrades

Reported by: dd32 Owned by: nacin
Milestone: 3.2 Priority: normal
Severity: normal Version: 2.9
Component: Upgrade/Install Keywords:
Focuses: Cc:

Description

It'd be nice that for point releases, Partial core upgrades were supported.

Ie. Instead of copying 1,000 files, 999 of which are identical, only a single file is updated on the system.

This could be done in a few ways.

  1. Zip archive only contains changed files
  2. The contents of the files in the zip and the local filesystem are compared before copying (Or even just the CRC32's)

are the first 2 which come to mind.

There'd also be a rather large speed up in terms of not writing temporary files to the filesystem first.. Which in itself, Takes a long time, and can make some upgrades take twice as long.. (but is needed)

I'd be willing to work on the implementation of it as well.

Attachments (1)

10611.diff (5.5 KB) - added by nacin 3 years ago.

Download all attachments as: .zip

Change History (20)

comment:1 dd325 years ago

As well as comparing the file contents, you'd also check for new files of course.. and still iterate over the deleted files list..

I'm not sure this should be supported for plugins (Well, The 2nd option is a possibility, but that'll require major changes in the process) quite yet, But could be implemented for plugins in the future.. I'd prefer to keep this disucssion centered around core upgrades for now.. As thats where i feel it'll be more appropriate.

comment:2 westi5 years ago

  • Cc westi added
  • Keywords dev-feedback removed

I think this is a really good idea for point releases and point releases only.

It needs to do something to confirm that the rest of the files are OK too though.

I did suggest this during last nights meetup :-)

comment:3 dd325 years ago

I think this is a really good idea for point releases and point releases only.

I'm not too sure of that, Take this example for example:

$list = $zip->get_list_of_files();
foreach $list as $file 
   if $partial_upgrade_enabled &&
      file_exists(ABSPATH . $file) &&
      file_get_contents(ABSPATH . $file) === $zip->extract_as_string($file)
   then
      Continue;
   ..Extract the file and overwrite the local one here.. 
   ..Either the new file is different, or is a new file..
end foreach;

It saves IO time basically, only changed files get written.

If the CRC32 route was taken, it could be even faster as the Zip decompression wouldn't have to be done.

It needs to do something to confirm that the rest of the files are OK too though.

Thats where the CRC32 or file contents check(as above) would be used

I dont think having a zip with only the changed files is a good solution, was just an offered one.

I did suggest this during last nights meetup :-)

I only skim read it.. Must've been where the idea came from, I dont think there' was that much chat about it if i missed it?

comment:4 AaronAsAChimp5 years ago

  • Cc AaronAsAChimp added

Does it have to be this complicated?

What if this was handled when the packages were created?
For example, you would create two packages wordpress-2.6.4.zip and wordpress-2.6.4-from-prev.zip.

So, if you had version 2.6.2 the packages wordpress-2.6.3-from-prev.zip and wordpress-2.6.4-from-prev.zip would be downloaded and unzipped in that order.

SVN already knows what files have changed, creating the packages wouldn't be that difficult. It would also save bandwidth

comment:5 follow-up: dd325 years ago

There was a bit of chat on IRC about the options.

Your idea of multiple step zips is actually more complex than the offered suggestions.

The main thing against partial upgrade zips is the potential of other files being changed.

One solution which was suggested was:

  • API on wordpress.org to return a list of md5's of files
  • WP then checks to see if all files match, And if only the changed files differ
    • Download the Partial upgrade zip
  • Else
    • Download the full upgrade zip
  • Continue as normal.

Basically the thinking is, Dont trust the install the user is running, dont count on all the files being exactly as we expect them to be.

I'm going to write up a proof of concept patch based off the psuedocode above, just to show the potential implications of that method, and how much of a speedup it may bring.

One thing that many people are not sure about is where the main bottleneck is

  • Downloading the large Zip file
  • The Zip Decompression (This is mainly memory usage IMO)
  • The IO time
  • The 2nd IO time

Personally i think its mainly the IO..

comment:6 in reply to: ↑ 5 Denis-de-Bernardy5 years ago

One thing that many people are not sure about is where the main bottleneck is

  • Downloading the large Zip file
  • The Zip Decompression (This is mainly memory usage IMO)
  • The IO time
  • The 2nd IO time

Personally i think its mainly the IO..

Oh, I can give you some feedback on that. Keep in mind that my own zip is 7M+. :-P

  • download takes a few seconds
  • cleaning up after a failed install is very very very slow (see #10913)
  • unzip is *very* slow and resource hungry
  • copying actually is pretty fast
  • cleaning up is, again, very very very slow

What frequently happens on my end with WP 2.8.x sites one of the following:

  1. the zip gets downloaded
  2. unzipping occurs, takes a lot of time, FTP times out about when it's done
  3. WP starts to copy, fails due to the FTP time out, and bails
  4. WP can't clean-up
  5. User tries again
  6. the zip gets downloaded
  7. FTP times out during the clean-up
  8. User tries again
  9. the zip gets downloaded
  10. FTP finishes up the clean-up
  11. Return to step 2 (until user decides it's hopeless)

The way I work around it is by hooking into update_feedback. On each feedback, I force flush the output so the user gets some kind of feedback. I then disconnect the FTP (some hosts disallow overriding time-outs, meaning that #10407 or the newer ticket needs more fixing), I reconnect the FTP, and I reconfigure the FTP so it times out after 10 minutes on the host where that works. So far it's pretty successful.

I considered uploading partial zips, but it seemed too risky (can't trust the version sent by the site, especially after a failed upgrade); the same goes for having a special file with the md5 of every file. Keeping the md5 in options is no good either -- if a manual upgrade occurs, it breaks.

That is to say, I think it's more important to fix #10913, which is the main culprit for failed upgrades.

comment:7 Denis-de-Bernardy4 years ago

I'd picture some UI for this one if we move forward: a checkbox that offers this option, in the event someone wants to reinstall.

comment:8 Denis-de-Bernardy4 years ago

actually, nevermind that last remark. there is no need for UI if we stick to upgrades, and proceed with all of the files for reinstalls.

comment:9 Denis-de-Bernardy4 years ago

  • Keywords bug-hunt added

comment:10 Denis-de-Bernardy4 years ago

  • Keywords featured added; bug-hunt removed

comment:11 nacin3 years ago

  • Milestone changed from Future Release to 3.2
  • Type changed from enhancement to task (blessed)

nacin3 years ago

comment:12 nacin3 years ago

In [18224]:

Update Core UI changes:

  • Update Now instead of Update Automatically.
  • The first core update is now a primary button.
  • Hide the version number from the 'You are about to install WP in English' warning for development versions, as it lies and suggests you will be downgraded.

see #10611.

comment:13 nacin3 years ago

In [18225]:

Core support for partial updates. see #10611.

comment:14 nacin3 years ago

[18224] were changes requested by Jane and also a bug I noticed.

[18225] was the result of a desire to have an absolute minimum amount of code go into core for this feature. Indeed, there are only two changes: A new API that delivers serialized responses modeled after the object that WordPress used to construct using the newline response (thus minimizing the changes required for the upgrader), and a bit of code in the upgrader to select which of four packages to download.

The packages are as follows:

  • partial - This zip uses the reported $wp_version to deliver an in-branch partial zip. If you're on 3.1.x and are upgrading to 3.1.y, you'll get something like wordpress-3.1.y-partial-x.zip from the API. Partial builds won't be used when you request a re-install. I'm working on a separate patch now that will further validate on the core side that the right partial build is being used.
  • new_bundled - This zip has the newest bundled theme. The version in which the latest bundled theme was added is also included in the response. Core will no longer touch any existing theme, it will only deliver a new one, and only if CORE_UPGRADE_SKIP_NEW_BUNDLED isn't set. Core will never touch or restore an existing plugin.
  • no_content - This zip contains no plugins or themes. It will however include /languages once the localized build system supports these new builds (and until we have language packs, which I'm aiming for 3.3).
  • full - This zip contains everything. Core should never fall back to this once the build process is in place.

This ticket can be closed as fixed once:

  • The new build process on WP.org is in place, functional, and tested. It's important to note that none of this comes into play until 3.2.1, so I'm establishing a system to test this on my sandbox. (Beta Tester updates rely on nightly builds.)
  • version-check/1.6-beta/ is moved to 1.6.

comment:15 nacin3 years ago

In [18227]:

Extra check for partial updates: Make sure the API response matches the current version of WP. see #10611.

comment:16 nacin3 years ago

  • Keywords needs-testing added; featured removed

To update the status of this:

  • The new build process is complete, and locally tested. Results are fantastic. I'll be getting it onto WP.org early next week.
  • version-check/1.6 will remain at beta for now.
  • Still need to re-do the localized build process. That should be fun.

comment:17 ocean903 years ago

  • Cc ocean90 added

comment:18 nacin3 years ago

  • Keywords needs-testing removed
  • Owner changed from dd32 to nacin
  • Status changed from new to accepted

Accepting this for now. This ticket can remain open past RC1.

comment:19 nacin3 years ago

  • Resolution set to fixed
  • Status changed from accepted to closed

Closing this. See #17759.

Note: See TracTickets for help on using tickets.