Use Firefox 3.5 native offline cache to add "turbo" functionality

[mfinkle]

Recent Wordpress releases add a "turbo" feature that uses Gears local cache to store many static resources locally and increases the overal performance of the admin panel.

Firefox 3.1 will support the same functionality natively, without the need for a plugin. WordPress could streamline it's user experience and minimize dependencies if it used the new Firefox feature.

​http://developer.mozilla.org/en/docs/Offline_resources_in_Firefox discusses how the offline resources work in Firefox. Firefox 3.0 did not support enough of the HTML5 offline spec to enable this feature for WordPress, but Firefox 3.1 will and the feature will appear in nightly versions of Firefox 3.1 soon.

[Otto42]

Is there any documentation on what, exactly, FF 3.1 will support? I know that the HTML 5 spec on this changed midway through the draft, leading different browsers to support differing functionality...

More info:
​http://www.nczonline.net/blog/2008/5/18/and_so_the_html_5_trouble_begins

[mfinkle]

Replying to Otto42:

Is there any documentation on what, exactly, FF 3.1 will support? I know that the HTML 5 spec on this changed midway through the draft, leading different browsers to support differing functionality...

Firefox 3 was caught in some of the spec change, but Firefox 3.1 is shaping up to completely support the HTML5 spec for offline cache:
​http://www.w3.org/TR/2008/WD-html5-20080122/#appcache

[Otto42]

Patch to add an HTML5 cache manifest to the wp-admin

[Otto42]

This patch is based on the gears manifest. Simple and easy. Might be worth combining the two manifest types into one file though.

It works with FF 3.0, even though it doesn't give any benefits since FF3 doesn't use it for online use. Still, future proofing and whatnot.

[mfinkle]

Replying to Otto42:

It works with FF 3.0, even though it doesn't give any benefits since FF3 doesn't use it for online use. Still, future proofing and whatnot.

The patch looks good (from a Firefox perspective) and should start getting speed benefits when the (almost landed) offline cache fallback features show up in Firefox 3.1 nightly builds.

I'll post here when that happens.

[caesarsgrunt]

Doesn't Safari also have its own offline cache? If so, does this mean it's native one should be used, too? Or was it just databases?

+1 for Fx3.1 cache, anyway.

[Otto42]

Replying to caesarsgrunt:

Doesn't Safari also have its own offline cache? If so, does this mean it's native one should be used, too? Or was it just databases?

The webkit engine does indeed have some support for the offline cache, but it hasn't made it into Safari for Mac/Windows yet.

Supposedly, the iPhone's latest version of Safari DOES implement it, and will load the manifest files into its cache when using the "Add to Home Screen" option.

[mfinkle]

Firefox 3.1 beta 2 has been released and Firefox 3.1 beta 3 will be released soon. Firefox should be in good shape to support the new offline cache manifest. Could someone try the patch with a Firefox 3.1 beta?

[Otto42]

The patch I made then is almost certainly out of date now, and would need to be refreshed.

Still, it would be nice if there was some way to rig it so that the manifest for Google Gears and the Firefox one were the same, so that updates didn't need to happen in both places. Any way to detect the requester, to serve the correct manifest file?

[blizzard@…]

Hi! I'm wondering if there's anything we can do @ Mozilla to help move this along. I talked with Matt briefly about it a couple of weeks ago and he thought it would make 2.8, but I just wanted to make sure.

Thanks!

[Otto42]

The patch I made ages ago has fallen way behind trunk and would need to be refreshed.

Ideally, it would be brought into line with the existing Google Gears functionality.

Extra-ideally, it would somehow be magically combined with the existing Google Gears functionality. If there were some certain way to determine what type of "manifest" was being requested by a browser (Gears vs. HTML5), then this would be rather simple to do, actually.

[azaozz]

Replying to blizzard@…:

Hi! I'm wondering if there's anything we can do @ Mozilla to help move this along.

Hi, it seems pretty straightforward, the only concern may be that we use query strings to refresh some files and that might be problematic with some net caches. Usually the user can refresh (Shift + reload) to get the latest versions. Does the HTML5 local storage add "cache busting" headers when refreshing resources or is there another method/settings we should use?

Replying to Otto42:

Extra-ideally, it would somehow be magically combined with the existing Google Gears functionality. If there were some certain way to determine what type of "manifest" was being requested by a browser (Gears vs. HTML5), then this would be rather simple to do, actually.

Gears sends a custom header "X-Gears..." when capturing resources, unfortunately it doesn't seem to send it when getting the manifest. So it seems we will need a separate manifest for HTML5. Also will have to avoid running both Gears and HTML5 caching, looks like this can be done from js.

[ryan]

Move generation of file list to function

[ryan]

Patch adds a get_manifest() function that generates an array of files and their versions which can then be presented in whatever manifest format desired.

[ryan]

get_manifest() + gears-manifest.php + cache-manifest.php

[ryan]

Are their equivalents to the Gears LocalServer src and ignoreQuery?

[azaozz]

(In [11076]) get_manifest() + gears-manifest.php + cache-manifest.php, props ryan, see #7262

[azaozz]

[11077] Lets try the HTML 5 offline resources storage (Firefox 3.5 only)

[ryan]

Currently failing miserably for me with 3.5beta4. Says 1.5MB is stored. Styles don't load. Removing offline data brings everything back to life.

[ryan]

Current manifest

[blizzard@…]

Can you give me a quick test case? What procedures are you going through to generate the negative results? If it's a bug in our impl (possible!) I would love to know more since we still have time to fix it ahead of Firefox 3.5.

[ryan]

Steps to repro: Visit the WP admin. Click "Allow" when Firefox prompts you to store data for offline use. Things look good for a few page loads but eventually the stylesheets are lost. In Preferences I see that data is stored for offline storage. Removing that data restores stylesheets. I can set you up on a 2.8 test blog if that is helpful. You can reach me at ryan@….

This is likely our bug since I'm still not sure we're doing the right things with our manifest. See the questions from azaozz and I above.

[azaozz]

Currently we're trying the simplest way: the same manifest is included on all pages and lists only js, css and images. All URLs are relative, the manifest and all pages are at the same location.

Seems that after a page is added to the master list it cannot open the cache resulting in no js and css...

[Denis-de-Bernardy]

might the issues be related to the fact that we're concatenating scripts?

[demetris]

IF

I define SCRIPT_DEBUT as true (turns concatenation off)

AND

remove version strings from URIs (by commenting out lines 38–40 in class.wp-styles.php and 93–95 in class.wp-scripts.php)

AND

comment out the version bit in line 45 of cache-manifest.php:

echo $file[0] /* . '?ver=' . $file[1] */ . "\n";

... then it works.

(I’m testing with Minefield 3.6a1pre, sending “Firefox/3.5” as user agent.)

[Denis-de-Bernardy]

So that would be FF-related.

[blizzard@…]

Can we get a small test case to work with? I'm not 100% sure what the issues are here because I don't know the WP code base but I'm more than happy to chase down an issue if it's on our end. And now would be the time to fix it as we enter the end of our release cycle.

Thanks!

[Denis-de-Bernardy]

@blizzard: I may be understanding this wrong, but based on demetris' comment, it would be related to our using script urls with parameters.

in other words, declarations for scripts such as this wouldn't work:

<script src="/script.js?ver=1.0"></script>

but this would:

<script src="/script.js"></script>

in our case, there is an additional complexity, as we do something like:

<script src="/concat-scripts.php?scripts=foo,bar"></script>

with turbo enabled, we'd probably want to disable script concat. but the issue on versions that we pass into the urls would remain. these are used to force a browser cache refresh when versions change.

[demetris]

@blizzard: What Denis describes is what I understand too and what I did to make the offline cache work:

1. I disabled JS and CSS concatenation (which is new in WP2.8, and on by default).

2. I disabled the addition of version strings to JS and CSS URIs.

By default, the concatenated files have URIs like this:

<​http://op111.net/wp-admin/load-styles.php?c=0&dir=ltr&load=global,wp-admin&ver=4bcab14dcde636cd0dc030359e18c5ab>

... while the versioned files are like this:

<​http://op111.net/wp-content/plugins/tangofy/css/fugue-shadowless.css?ver=2.8-bleeding-edge>

... or like this:

<​http://op111.net/wp-admin/css/colors-fresh.css?ver=20090509>

[blizzard@…]

I'm not sure what's going on here. Let me talk to some people, build up a couple of test cases and see what I find.

[ryan]

(In [11280]) Remove manifest from html until we get it working. see #7262

[azaozz]

Replying to blizzard@…:
Here's a copy of the ​dashboard in wp-admin (not small though). It has exactly the same structure and files.

I'm not sure if we can actually use the HTML 5 offline storage the same way as Gears offline storage.

With HTML 5 the page that contains the manifest link is cached too even when it's listed in the NETWORK: whitelist. That's not going to work well with dynamic pages generated from php when we only need to cache the other resources, js, css, etc. Perhaps there's a workaround for this.

[blizzard@…]

One of our Mozilla guys, Paul Rouget, is going to help you with this. Either

1. We've got some issues in our impl (very possible) or

2. We might need help understanding what we need to do to support this over the long term. Might be that your mental model is tied to gears so we might need to do some planning for a later release and add or change some features.

[vingtetun]

I guess the error is from the manifest#78 which says to store a css file named css/colors-fresh?ver=20081210 and on the the example given by azaozz the website says to load css/colors-fresh?ver=20090503.
Firefox never load css/colors-fresh?ver=20081210 from the cache and so the style isn't applied. If we use a js bookmark for dynamically alter the page, the style are finally load from the cache ( wireshark didn't show any network traffic )

[
javascript:(function(){var%20headID%20=%20document.getElementsByTagName("head")[0];%20var%20cssNode%20=%20document.createElement('link');%20cssNode.type%20=%20'text/css';%20cssNode.rel%20=%20'stylesheet';%20cssNode.href%20=%20'css/colors-fresh.css?ver=20081210';%20cssNode.media%20=%20'all';%20cssNode.setAttribute('id','colors-css');headID.appendChild(cssNode);})()
]

[rouget]

It works well with the right CSS reference (via the Vingtetun bookmarklet).

Azaozz, please tell us if it fixes your issue.

[azaozz]

In [11327] Make sure all colors stylesheets have the same version string.

[rouget]

Do you run into any other issues about the offline mechanism?
Please let me know how we can help.

[azaozz]

Replying to rouget:

Do you run into any other issues about the offline mechanism?
Please let me know how we can help.

The css loads properly but some of the js files are still missing. Re-checking all to make sure there aren't any other inconsistencies there.

Perhaps we are trying to use the offline storage in a non-standard way: trying to store all js, css and image files but not the html (php output).

[rouget]

Replying to azaozz:

Replying to rouget:

Do you run into any other issues about the offline mechanism?
Please let me know how we can help.

The css loads properly but some of the js files are still missing. Re-checking all to make sure there aren't any other inconsistencies there.

Did you re-check all? Are there still any inconsistencies? In this case, can you provide us a minimal test-case?

Perhaps we are trying to use the offline storage in a non-standard way: trying to store all js, css and image files but not the html (php output).

I think you don't need to store the html (if you look at the http queries you will see that there's no request, except for the manifest).

[hakre]

In the long and beloved tradition of WordPress: Please use absolute URIs in the manifest, not relative ones (even though I hate to say that), thanks.

[azaozz]

Managed to get it to (mostly) work with two exceptions:

1. No matter how the manifest is set, all html files containing a link to it (all php output) are always cached. For example when editing a post, moderating comments, changing screen options, the changes are saved but the page always goes back to the previous state after reload.

2. The workflow of our (not-so-standard) setup is to load the manifest and cache the non-html resources (js, css, images..) linked in there, then use these cached files for all html pages loaded from the same domain when they request any of these resources, but not cache the actual html pages as they usually change very frequently. This happens automatically in Gears but seems impossible here.

We seems to be missing the

INTERCEPT: list of [urlnamespaces, namespace-handler url]

section in the manifest and perhaps the

AUTOCACHE: list of [urlnamespaces, namespace-handler url]

section too.

If we are able to use the INTERCEPT: section, # 1 above would be solved too as we could load a separate file containing the link to the manifest and let it be cached, while all other html files would use the resources from this cache without being cached. This would still need some workarounds to ensure that the manifest is checked for changes frequently.

[azaozz]

Current cache manifest

[peaceablewhale]

Is this too late to be included in WordPress 2.8 as 2.8 is already in RC? I suggest delaying the milestone to 2.9.

[azaozz]

Replying to peaceablewhale:

Is this too late to be included in WordPress 2.8 as 2.8 is already in RC? I suggest delaying the milestone to 2.9.

Yes, unfortunately we can't use the HTML 5.0 offline storage the same way we use Gears (at least for now).

[azaozz]

(In [11537]) Remove cache-manifest.php, see #7262

[DWizzy]

Replying to Otto42:

Replying to caesarsgrunt:

Doesn't Safari also have its own offline cache? If so, does this mean it's native one should be used, too? Or was it just databases?

The webkit engine does indeed have some support for the offline cache, but it hasn't made it into Safari for Mac/Windows yet.

As of version 4.0 (final release a month ago), Safari officially supports offline cache. Of course, Firefox 3.5 is out now, too. Perhaps the aforementioned bugs were only in RC's and not in the final version? Unfortunately, I can't try installing a patch on a Wordpress blog myself to test this.

[hakre]

Getting Firefox 3.5 now for testing but have no Idea how a test would look like.

[miken32]

Would it work to have a hidden iframe load with the manifest attribute on the subdocument's html element? Kind of a hack, but would get around the problem of the master entry always getting cached.

[azaozz]

Replying to miken32:

Would it work to have a hidden iframe load with the manifest attribute on the subdocument's html element?

Unfortunately no. Tried that too but the resources (js, css, etc.) that are cached in the iframe are not used in the main window. This is actually the largest shortcoming of the HTML 5 implementation: cached resources are not used for other pages on the same domain.

This works good for 'static' web sites but is kind of useless for dynamic sites where the same page can be slightly different on every load and the number of pages is different depending on what the user is doing.

For example Gears works in exactly the opposite way: requests for cached resources for the same domain are always served from the cache, automatically. So once all resources of a web site are stored in the offline cache, they are never requested from the server (not even HEAD requests) making the site run faster and eliminating useless traffic. Of course the resources can be updated when needed through the manifest.

[Edified]

Replying to azaozz:

Replying to miken32:

Would it work to have a hidden iframe load with the manifest attribute on the subdocument's html element?

Unfortunately no. Tried that too but the resources (js, css, etc.) that are cached in the iframe are not used in the main window. This is actually the largest shortcoming of the HTML 5 implementation: cached resources are not used for other pages on the same domain.

I think your information is either outdated or wrong.
Cache Manifests work as expected in current versions of both Firefox and Safari. I'm using it in my theme now.

In my testing all pages that have the same cache manifest list use the same local cache files.

You can verify this easily by watching your server logs as you load pages.

[azaozz]

Replying to Edified:

I think your information is either outdated or wrong.
Cache Manifests work as expected in current versions of both Firefox and Safari. I'm using it in my theme now.

Yes, it follows the HTML 5.0 specification, however this specification is designed only for 100% caching (bringing a web app offline) and cannot be used to cache only non-dynamic parts of a web page. So it caches the HTML first (we don't want that) and then it caches the css, js, images, etc. (we want only these).

For comparison Gears caches all files that are in the manifest and then uses that cache for all requests for these files regardless of the web page that loads them (from the same domain of course).

[Edified]

I see- you're right. Quite annoying- I'll be going back to gears.

Thanks for your time anyway.

[rfdparker2002]

It's worth noting that the Linux version of Chrome/Chromium doesn't support Gears - just HTML5 Local Storage - as the developers said they felt that Gears will soon by replaced by HTML5 Local Storage (see ​http://code.google.com/p/chromium/issues/detail?id=17443#c4 ), yet as things stand WordPress Turbo still only supports Gears and all of Google's own services still use only Gears (although they're working on moving Docs over) even though Google's Gears team said they're discontinuing Gears development ( ​http://gearsblog.blogspot.com/2010/02/hello-html5.html ).

[azaozz]

In my opinion it's worth it trying to extend the HTML 5.0 specification to cover user cases similar to how we use Gears. As all newer browsers are implementing it, imagine the millions of web sites that would benefit and the millions of megabytes of web traffic that will be saved even if only WordPress uses this approach.

And I'm sure a lot of web apps would start using it this way if that was possible. It makes a lot of sense to store the static components of a dynamically generated web page locally and manage them from the server eliminating all requests. This would also save some energy (reduced server load, less Internet traffic) which IMHO is a good enough reason by itself.

[blizzard@…]

Yeah, when we had our caching summit a few weeks ago we tried reaching out to Wordpress people to join us and I don't think we ever got someone to come to it.

Note that the localstorage thing is actually two things:

localStorage (in all major browsers today)
IndexedDB (prototyping in IE, Chrome and Firefox)

But that doesn't help with super-aggressive caching, which is what I think you want.

Note that the admin interface really could be a full HTML5 web app, with manifest and all. Also you can dynamically add stuff to the manifest with Firefox extensions. But that's going to require a lot of work on your part, I suspect.

[azaozz]

Replying to blizzard@…:

Yeah, when we had our caching summit a few weeks ago...

Unfortunately I wasn't able to attend this time.

Note that the localstorage thing is actually two things:

localStorage (in all major browsers today)
IndexedDB (prototyping in IE, Chrome and Firefox)

We are not using the DB part at all.

But that doesn't help with super-aggressive caching, which is what I think you want.

It's not even supper-aggressive catching. It is a "server managed catching in the browser".

What we currently do in Gears:

1. The browser downloads all "static" files (JS, CSS, images) and stores them in the local storage for the current domain. The URLs for these files are in the manifest.

2. Whenever a web page coming from the same domain requests one of these files it is fetched from the local storage instead of the server. This is handled automatically by Gears and no requests are made to the server at all.

3. Gears downloads the manifest frequently (or this can be triggered manually from JS) and compares it with the previous version. If the manifest has different version string it parses it again, downloads the new/changed files and removes old files that aren't listed there any more. So when we need to refresh a file we change the manifest version and update the listed URL in it. All this happens in the background.

The end result is that when the user enables it, this stores about a megabyte of "static" content on their hard drive and eliminates about 90% to 95% of the requests to the server, even the HEAD requests. This of course speeds up page loading considerably and also reduces unneeded web traffic.

It is possible to achieve something similar (but not as straightforward) with super-aggressive caching however that is much harder as it is controlled by the web server/hosting company and there are literally thousands of different configurations that have to be taken into account.

[Malcjohn]

Replying to blizzard@…:

localStorage (in all major browsers today)
IndexedDB (prototyping in IE, Chrome and Firefox)

But that doesn't help with super-aggressive caching, which is what I think you want.

2 Reasons for "wontfix" and "closed".

1 ) Now impossible.... to do so, because NOT everybody is using the newest browser

2 ) ​http://gearsblog.blogspot.com/2011/03/stopping-gears.html

[nacin]

because NOT everybody is using the newest browser

It's called progressive enhancement.

stopping-gears.html

We're aware: #11301. This ticket is about leveraging HTM5 localStorage, not Gears.