Create "WP Heartbeat" API

[azaozz]

The purpose of this API is to simulate bidirectional connection between the browser and the server. Initially it will be used for autosave, post locking and log-in expiration warning while a user is writing or editing.

The idea is to have a relatively simple API that sends XHR requests to the server every 15 seconds and triggers events (or callbacks) on receiving data. Other components would be able to "hitch a ride" or get notified about another user's activities.

In the future this can be used to block simultaneous editing of widgets and menus or any other tasks that require regular updates from the server.

I know, ​WebSockets are meant for this. Unfortunately few servers support them.

WebSockets use port 80, so it would simply be a matter for us to code it.

PHP 5.3 examples: ​https://github.com/nicokaiser/php-websocket and ​http://socketo.me/

Replying to nacin:

WebSockets use port 80, so it would simply be a matter for us to code it.

PHP 5.3 examples: ​https://github.com/nicokaiser/php-websocket and ​http://socketo.me/

These need to be long-running processes (they're essentially separate servers on that socket), which may get killed by services internally. Long-polling (as per azaozz's suggestion) is a better idea given the variability of servers.

I played around with that library this morning - and can tell you that both client & server implementations are lacking. I spent about 2 hours hacking with both - and was unable to get chrome or Firefox to establish a connection. I ended my exploration thinking that AJAX / transients are the way to go.

Replying to rmccue:

Replying to nacin:

WebSockets use port 80, so it would simply be a matter for us to code it.

PHP 5.3 examples: ​https://github.com/nicokaiser/php-websocket and ​http://socketo.me/

These need to be long-running processes (they're essentially separate servers on that socket), which may get killed by services internally. Long-polling (as per azaozz's suggestion) is a better idea given the variability of servers.

I agree - and most Webhosts will kill any process around 180 seconds.

Very interesting. How does the document editor on Google do simultaneous editing?

Replying to knutsp:

Very interesting. How does the document editor on Google do simultaneous editing?

I was thinking the same thing - Google docs does multi user simultaneous editing VERY well. They have per-line locking, meaning that multiple users can edit the same doc @ the same time. I have played around with G'Docs before - trying to cause collisions and errors. The only time I remember being able to cause an error was by explicitly trying to edit the same line that I could see someone else was also editing. They handle that type of error by reverting your changes - and in real time - to minimize potential data loss.

In regards to implementing such a feature in WordPress - It would require both the server / client architecture that we're discussing here, as well as a separate patch / plugin for TinyMCE. Anyone have experience with the text parsing / event architecture of the TinyMCE API? I've written several plugins for TinyMCE - but those are used to extend it - and are not involved with the innards of the editor like this type of plugin will be.

Concurrent editing is not on the table, yet :)

It cannot be done with a TinyMCE plugin. To be able to lock rows (or even paragraphs) TinyMCE would most likely have to move away from using contenteditable and do direct DOM manipulation so only the currently edited row (text node) will be "unlocked" and the rest will be a standard, non-editable HTML.

As far as I know Google docs do XHR saves every second while the user is typing, and in addition do polling every 10-15 sec. We will be doing something similar but instead of saving to the server, we will be saving to the browser's local storage (see #23220).

Replying to azaozz:

We will be doing something similar but instead of saving to the server, we will be saving to the browser's local storage (see #23220).

I just read about this on ​the Mozilla site - article from May 2012. Aside from the statement "terrible performance", I wonder which browsers support this. WP has always cared about a lot of browsers and I doubt that IE7 has a proper equivalent of localStorage.

Replying to F J Kaiser:

Could we move this discussion to #23220, replied there.

Replying to azaozz:

Concurrent editing is not on the table, yet :)

It cannot be done with a TinyMCE plugin. To be able to lock rows (or even paragraphs) TinyMCE would most likely have to move away from using contenteditable and do direct DOM manipulation so only the currently edited row (text node) will be "unlocked" and the rest will be a standard, non-editable HTML.

Sounds like TinyMCE would have to implement this type of feature in it's core?

Rather than a strict 15 seconds, maybe Wordpress should send back a value specifying a suggested time for the client to wait until the next 'beat'. We can set it to 15 seconds by default, but if we know something will be available in 5 or 10 seconds, or if the server is under heavy load and to wait for 30 seconds.

It would provide a lot more flexibility in the future and opens the door to basic load management plugins.

Replying to TJNowell:

Rather than a strict 15 seconds, maybe Wordpress should send back a value specifying a suggested time for the client to wait until the next 'beat'...

Yes, was thinking/testing something similar. Perhaps we can speed up the beat when user B performs an action that would need response from user A like opening a post that is already being edited by user A, etc.

We can add support for this on both sides, PHP and JS.

Generally we have three options:

Standard polling

• 15 seconds interval, simple to implement, simple to use and understand.
• Covers all current user cases, however can "feel" slow when taking over a post lock.
• Possibly add throttling to improve user experience.

Long polling

In essence it would look something like this on the PHP side (on the JS side this is a standard XHR that runs in recursion):

function wp_poll() {

	for ( $i = 0; $i < 10; $i++ ) {
		$val = apply_filters( 'wp_poll', array() );

		if ( ! empty($val) ) {
			echo json_encode($val);
			exit;
		}

		sleep(2);
	}

	echo '0';
	exit;
}

• Better user experience than standard polling as it will deliver notifications to the browser faster (speed is only limited by the time needed to do a XHR). Also can be used when/if we implement concurrent editing.
• Ties resources on the server. There will be one process per user that will be running almost constantly.
• Needs to poll the DB.
• May run into problems with caching in PHP.

Hybrid

Start with polling and switch to long-polling when needed instead of reduced interval (as described above). This still needs some time to switch both user A and user B to long-polling, but preforms better after that.

May run into problems with caching in PHP.

I think that's a pretty big con, WordPress currently works in the form of pageloads, most people are OK with a page using stale cached data it retrieved 10ms ago, or even 100ms ago, once we start to talk in the 10~15second range though, that can be a lot of stale cached data in an environment where you're trying to make things "instantanious".

Yeah, long-polling would break the perception of pageloads as it "waits" in the same process for 10-15 sec. for something to change, polling the DB (or memcached) to detect that change.

For example apply_filters( 'wp_poll', array() ) cannot be used to do get_option('...') as once loaded, options are stored in the $alloptions global.

Exactly my point, it means we can no longer use our API's, or would be required to write code that bypasses our own API's, or deliberately flushed the local cache every loop iteration.

The last part (flushing local cache on loop iteration) would mostly take care of it actually, It would avoid the cache issues, and allow the poll to skip the time to load and parse WP files.
I do however, think that it may be best left to a plugin for initial implementation, and us focus on a solution that reduces resource impact upon the server - WordPress can use quite a bit of memory in a page load, and most shared servers might not like a few resource hungry processes running for 15x their normal lifespan.

WordPress can use quite a bit of memory in a page load, and most shared servers might not like a few resource hungry processes running for 15x their normal lifespan.

Definitely. We can get away with a quick XHR request every 15 seconds, but probably not a long-poll like this. It's clever, but probably too clever for now.

The last part (flushing local cache on loop iteration) would mostly take care of it...

You're right, it will take care of most places we "cache" in PHP but not of constant inside classes and functions. So we may still run into problems with our APIs. Add to that the extra server resources and long-polling doesn't look very appealing.

Agree that we should do normal polling and make it easier for plugins to implement long polling. Speeding the "heartbeat", perhaps down to 5 sec. plus time taken by the XHR, would also improve the user experience/reduce waiting to take over a post lock by up to 10 sec.

In 23216.patch: first-run. Still need to decide whether to do separate filters for each data set sent through heartbeat (can be confusing as the filter name will depend on the data) or one filter for any data.

Another (major) consideration is use of custom events in JS. Would be best to get our own API #21170 up to speed and use it.

In 23355:

Heartbeat API: first run, see #23216

Seeing wp.heartbeat.interval(5); - What should happen if there are more then one plugins which are changing the interval on a page? Which one will win? How should it be handled?

Replying to ocean90:

Currently the last one to set the interval wins. This implements the idea to be able to speed up or slow down the "beat" when needed (comment:21). wp.heartbeat.interval() gets the current interval and wp.heartbeat.interval(5) sets it. So a plugin can do:

if ( wp.heartbeat.interval() > 10 )
    wp.heartbeat.interval(10);

The default interval can also be set when initializing with the 'heartbeat_settings' filter in PHP.

Been thinking how to best implement this speed up/slow down. Seems the most common case would be to speed up when we expect certain response from the server, like while taking over a post lock: we need to notify user B and unlock the post for editing as soon as possible after user A releases the lock.

In that terms wp.heartbeat.interval() should probably accept two int arguments, first is the interval in seconds, second would be for how many "beats" (optional, default 10?). Then we can protect a lower value for the interval so plugins don't stomp each other.

Another possibility is to have three values: fast(5), medium(15), slow(30). Then setting it to slow will only be possible if the current value is normal, and setting fast will look into the second argument: for how long/how many "beats", and will revert to medium after that.

The first run uses the JS global pagenow to identify from which screen the XHR was sent. That is different from the PHP global $pagenow (in JS it is $current_screen->id;). Will rename it to avoid confusion.

Another consideration: if the user has two tabs open with the same admin screen, on the PHP side they all look the same. We can detect when a tab is "blurred" and throttle the requests, but we may still run into problems if several tabs are open (more than 10?). Should we make the "beat" very slow, perhaps every 2 min, or even pause it if the user is not on the browser tab?

I would say slow it down a lot, but don't stop it. For example, if you have a tab open in the background and lose a post lock there, it would be nice to come back to the appropriate message rather than waiting for the heartbeat to start back up.

In 23382:

Heartbeat API: throttle down when the window looses focus or when the user is inactive, always send 'screen_id', change the interval settings to 'fast' (5sec), 'standard' (15sec) and 'slow' (60sec), the interval can be changed from PHP, see #23216

Small plugin to demo and test the first run.

In [23382] there are two additions: detect when the window looses focus and when the user is not present. The first is looking at $(window).blur() and the second is looking for keyboard and mouse activity every 30 sec. Both ways also temporarily attach events to all iframes on the screen.

On detecting window blur or no user activity the interval is slowed down to 120 sec. Currently the "no activity" timeout is 5 min, so even if the window is in focus but the user doesn't type or move the mouse it will still slow down to 120 sec.

All this would need more testing and perhaps adjusting. There is some debug code in [23382] that outputs to the browser console. To turn that on, type or paste

wp.heartbeat.debug = true

in the console and run it on the Edit Post screen, or install heartbeat-test-plugin.php and test on the Dashboard.

Seeing the error in IE 8 on Edit Post screen in trunk:

Message: Could not delete 'window.heartbeatSettings'
Line: 34
Char: 4
Code: 0
URI: http://trunk.wordpress/wp-includes/js/heartbeat.js?ver=3.6-alpha-23386

In 23389:

Heartbeat API: fix error in IE < 9, props SergeyBiryukov, don't attempt to bind events to cross-domain iframes, see #23216

In 23481:

Heartbeat API: add nopriv actions, add JS 'heartbeat-send' event, see #23216

In 23482:

Fix typo in wp_heartbeat_settings, see #23216

/wp-includes/js/heartbeat.min.js is still missing.

Replying to markoheijnen:

/wp-includes/js/heartbeat.min.js is still missing.

Added in [23616].

Replying to azaozz:

The purpose of this API is to simulate bidirectional connection between the browser and the server. Initially it will be used for autosave, post locking and log-in expiration warning while a user is writing or editing.

So does this mean that Wordpress will be implementing a database sync framework? With a web based publishing platform, how are you going to save data locally to be synced with the server upon reconnecting? Since most Wordpress sites are not multi-server setups, how is the heart-beat going to work on a "Single Point of Failure "if the server crashes, storm takes out internet ect?

I could understand such a system if you were running ManageWP.com, they could check for connection loss and re-connection since their system (acting as a middle man) could then sync the data. I just don't see or can understand any benefit to adding this stuff if it can't do any more than what the current system does.. Unless of course you are going to re-write the admin utilizing something like ​http://appjs.org/ but I would doubt it. Am I missing something?

In regards to the future plans, Why not do something like most Wiki's and set a "being edited" flag.. You can show a notice to another editor that the article is currently being written/edited.. And or use ajax like live chat clients letting each editor see that the current document is being editing by such and such, once document is saved, give the green light to whos next. When the document is saved or canceled, remove the "being edited" flag.

Replying to cbunting99:

With a web based publishing platform, how are you going to save data locally to be synced with the server upon reconnecting?

Related: #23220

You can show a notice to another editor that the article is currently being written/edited..

Related: #23312

In 23692:

Logged out warnings, heartbeat: remove nopriv_autosave as it doubles the functionality of the logged out warnings, move wp_ajax_nopriv_heartbeat() under No-privilege Ajax handlers in ajax-actions.php, see #23295, see #23216

In 23882:

Heartbeat: improve setting the errorstate, add ajaxurl to the settings object when loading on the front-end, some code cleanup, see #23216

As I read the code on the original submission, the heartbeat runs always, regardless of whether it has any consumers using it. I'd suggest that it should only run if it has consumers - i.e., a more sophistcated API. If it does so, then it could potentially *reduce* server load (since all the parts of WP wanting a regular poll would ride onto the one heartbeat, instead of implementing their own). However if it runs *always*, even if not doing anything useful, it may be problematic.

If running always, then I'd suggest that some evaluation of performance impact is done for typical use cases. Otherwise, even if its impact is small, we may get more budget hosts saying "WordPress is a big performance hog", or more armchair tech pundits will start saying "don't run WordPress on a low-budget server".

What makes me suspect that it will have a real-world, measurable impact is that I have a plugin which if you open its admin page, it polls via AJAX to check the 'last log message' every 5 seconds. If I open two admin pages on localhost, then I can hear the CPU fa
n ramp up. CPU load goes from 0.40 (average when I'm just clicking around) to about 1.1 (average with 2 admin pages open with polling every 5 seconds). This is on a 1-year-old low-end laptop (T4500 dual-core, 4Gb, so no swapping is going on). So, not comparable to to proper web hosting, but possibly comparable to people who like to manage their own WordPress install and who buy a bottom-end VPS for it. This makes me think that it's non-trivial and that a proper, scientific evaluation would be valuable so that we'd know one way or the other whether it affects anyone or not. Based on my experience, a poll-every-15-seconds heartbeat means that every 1 admin page is adding about 0.2 to the average CPU load. So if a hosting company had 10 WordPress admin pages open at any one time on a server, they need an extra 2 CPUs, were they crazy and using Pentiums from low-budget laptops - which would be crazy of them; but you get the point - it's potentially a measurable affect.

David

As I read the code on the original submission, the heartbeat runs always, regardless of whether it has any consumers using it.

Yes, currently 'heartbeat' is loaded everywhere in the admin and used by the login expiration warnings. Potentially we could be doing these warnings only on screens that need them most, like the Edit Post screen, all settings screens, etc.

Heartbeat runs on a 15 sec. interval only when the browser window is in focus and the user is active. If the window is "blurred" or there aren't any mouse and keyboard activities for 5 min, that interval drops to 2 min.

There will probably be some impact but as far as I see it's going to be insignificant. Not sure why your test "server" is performing so poorly. With the exception of the Edit Post screen, most users generally spend little time on most admin screens, a distant second is the Comments screen. So even if we load heartbeat only on the Edit Post screen (where it's most used atm) the impact wouldn't change.

Well, here's a crude estimation. On a 4-core Intel Xeon X3360 @ 2.83GHz webserver that I use, loading the WordPress dashboard (the server is using PHP via CGI) uses 0.3 seconds of CPU time (not wall time). On my laptop it's about double that. I've seen plenty of cheap VPSes that are comparable to my laptop.

Once very 15 seconds = one CPU can serve 50 WP admin logins on the Xeon, or one CPU can serve 25 WP admin logs on a cheaper setup. That seems like significant enough to take into account, or at least do more investigation into.

For example, we know that many web hosts disable loopback connections - pointless, annoying, but they do it. If word gets round that you can spare your CPUs + get more out of your servers by blocking heartbeat connections ("and all that happens is that WordPress users don't see login warnings, or lose some advanced functionality"), that'll not be fun. I wonder if something like login expiration warnings is worth taxing CPUs for every 15 seconds; or at least, the algorithm needs more attention. For example, login-expiration time is known when the page loads. Wouldn't it make more sense to just re-poll that once a minute before the known time rather than ask the question every 15 seconds? Rather than having a heartbeat every 15 seconds regardless of demand, why not have a heartbeat only when demanded by a consumer? (And for login expiration warnings, why not just use a one-off JavaScript setTimeout rather than polling it anyway?).

@DavidAnderson You seem to be ignoring the fact that when you load admin pages from localhost on your laptop, your CPU is running the browser as well as the server. Indeed your laptop is most likely running all sorts of things that a VPS does not run. I think it is reasonable that WordPress can assume that everyone's hosting can cope with more than 4 pageloads/heartbeats a minute. Any hypothetical 'It's the apocalypse and CPU cycles are expensive' ration-time poster like "Is Your Heartbeat Strictly Necessary?" would be right alongside "Think Before You Click That Thing Again So Soon!"

You can rest assured that blocking heartbeat connections would be a much more targeted and deliberate act for a host to do than disabling loopback connections. Besides, the possibility of something being stopped is a poor reason not to start something.

@PaulGregory,

The 0.3 of CPU time was only measuring the CPU usage due to the PHP CGI process, not of any other processes. Or, if you're referring to the laptop running hotter (no swapping, recent model), then you're ignoring the fact that that's still a real-world effect. People will file bug reports: "Hey, my laptop fan goes every time I log into my WordPress development site dashboard. Why does it do that? It never used to on 3.5!". They'll want a reply that does more than reference imaginary posters. What I'm saying is not "Hey, stop that, bad idea!", but that a more thorough evaluation of the likely effects - *or lack thereof* - would be potentially valuable; and consideration of whether some things slated for the "heartbeat" (e.g. logout warning) would be better achieved a less resource-intensive way. To give another real-world example, I lived in Africa for 5 years. Telling a website developer on a 6-year-old laptop that "yes, heartbeat is bringing your laptop to a grinding halt, but that's life in 2013! Get a new laptop!" is all the same as telling him "don't use WordPress, it's not for you!".

David

Replying to DavidAnderson:

I wonder if something like login expiration warnings is worth taxing CPUs for every 15 seconds; or at least, the algorithm needs more attention.

Related: ticket:23295:44

In 24139:

Fix typo in heartbeat.js. see #23216.

This was really slowing down my localhost installation and I'm worried about the affect it will have on on page responsiveness. Partly the issue was because of resolving the DNS of localhost, but it was also taking too much CPU in general.

Here's an article I wrote on ways to slow down the interval or remove the heartbeat altogether: ​http://www.seodenver.com/wordpress-heartbeat/

Perhaps the high CPU usage was caused by the DNS lookups? When properly set, the localhost DNS lookups should be virtually instant as they come from the hosts file. That may be the reason I don't see any significant change when I set the interval from 15 to 60 sec.

In 24268:

Heartbeat: don't connect when a response is not expected by any script (nothing to send), remove debug logging, clean up code formatting, see #23216

I have the debug log enabled and have been using trunk while testing a plugin of mine in development. It creates posts during an ajax action, and I've found that my debug log is filling up with the following errors:

[20-May-2013 01:41:07 UTC] PHP Notice:  Trying to get property of non-object in /wp-includes/capabilities.php on line 1070
[20-May-2013 01:41:07 UTC] PHP Notice:  Trying to get property of non-object in /wp-includes/capabilities.php on line 1074
[20-May-2013 01:41:07 UTC] PHP Notice:  Trying to get property of non-object in /wp-includes/capabilities.php on line 1076
[20-May-2013 01:41:07 UTC] PHP Notice:  Trying to get property of non-object in /wp-includes/capabilities.php on line 1077
[20-May-2013 01:41:07 UTC] PHP Notice:  Trying to get property of non-object in /wp-includes/capabilities.php on line 1077
[20-May-2013 01:41:07 UTC] PHP Notice:  Trying to get property of non-object in /wp-includes/capabilities.php on line 1080
[20-May-2013 01:41:07 UTC] PHP Notice:  Trying to get property of non-object in /wp-includes/capabilities.php on line 1080

I put some debug things (debug_backtrace() written to the log) in place to see what was happening and traced it back to the new heartbeat ajax action. For some reason the actual post ID is never getting sent.

Once the error starts, it continues to write the same 7 error lines each time the heartbeat action happens (every 15-60 seconds?). Needless to say, this fills up my debug log pretty quickly.

Hopefully it will help you if I provide my debug log (with my own debug data injected) here. ​http://b.ustin.co/1ZYp

It's very possible I'm doing something wrong here, but nothing stands out to me, and it's definitely related to the new heartbeat api.

Thanks for the detailed description and logs. Happens when there are no posts listed in the list table on the Posts screen. Patch coming up.

Fixed in [24299].

Replying to azaozz:

Thanks for the detailed description and logs. Happens when there are no posts listed in the list table on the Posts screen. Patch coming up.

Awesome, thanks! Always glad to know I'm not crazy or breaking things.