Magpie AND SimplePie cache needs looking at

[Otto42]

The magpie RSS reader caches feeds in the wp_options table for a period of 1 hour, currently. This bloats the options table pretty severely if somebody uses a lot of RSS feeds, and leaves RSS feed caches lying around if somebody ceases to use them anymore.

This needs some work done on it to clean up old or unused feeds after some period of time.

Suggestion: It would be nice if this used the object caching stuff instead. Problem with this notion is that the object cache doesn't actually cache anything anywhere persistent by default, so perhaps it could use the object cache if there is some kind of actual persistent caching plugin hooked to it.

Suggestion 2: Get rid of Magpie. Really. Switch to SimplePie or some other feedreader that a) works properly and b) is still being actively developed/maintained.

[DD32]

See Also: #5378

[jidanni]

Today I examined a dump of the database of my vanilla install.

I discovered the horribly bloated table wp_options rss* items.

Let's say we wish this junk was not in our database, but we don't want
to pull it out manually, but instead attempt via the admin interface
to keep it out.

OK, apparently these are "Show on screen" stuff from wp-admin. One clicks
to have them not shown on screen, but apparently they will forever be
in the database. Indeed, they comprise the biggest part of the
database upon vanilla install.

Anyway, one removes it from "screen options" but the rss_* bloat in the
wp_options table doesn't go away.

[ryan]

You can disable the feed dashboard modules if you don't want them to reappear. Right now we're stuck with caching to the options table. Re-closing.

[gazouteast]

I have no idea why Matt and Ryan are so insistent this is not an issue - it has persisted through to WP 3.0.1 even though Magpie was ditched and replaced with SimplePie a while back.

Over the weekend, I checked all my blogs, they all suffer from this - one blog with just 120 posts and 40 comments, has got a 64MB wp_options table with 4500+ rows in it - all due to aged Magpie rss cache entries.

Another site (using 3.0.1) with NO posts, pages, or comments other than the Hello World one, that's been sitting dormant waiting time to build it, was installed originally with 2.9.2 and has been upgraded to 3.0.1, and it somehow had over 2000 transient_timeout and rss_transient_timeout entries in wp_options.

There is something seriously out of wack with the rss caching (in my opinion it should not be caching into wp_options anyway - it should have it's own table to prevent the inexperienced from completely wiping the whole wp_options table).

I can only assume lead-devs keep closing this ticket because they don't know what's causing the problem, nor how to fix it - so there's your red rag - prove me wrong by fixing it, or prove me right by closing the ticket again.

[Denis-de-Bernardy]

magpie is deprecated.

[dd32]

Note: Every version bump the old rss_% options are deleted ​http://core.trac.wordpress.org/browser/trunk/wp-admin/includes/schema.php#L364

364	        // delete obsolete magpie stuff
365	        $wpdb->query("DELETE FROM $wpdb->options WHERE option_name REGEXP '^rss_[0-9a-f]{32}(_ts)?$'");

See #10592 + [12210]

[gazouteast]

@ Denis - I don't care if it is deprecated or not, the problem is persistent across all my blogs, whether first built using WP 2.3 all the way through to brand new clean 3.0.1 builds that have NEVER had any plugins, and NO theme except twentyten.

There is something still within WP that is causing this issue - it is neither resolved nor invalid.

I lost another site yesterday due to this - one with 2000 posts in it - and the hosts want money to restore the database backup. In this instance the wp_options table crashed with 77 MB of data and 145,000 rows in it.

I lost an entire hosting account two weeks ago with 8 domains in it, and those hosts also want money to send me the backups. My other domains on several hosts are also under threat due to this issue.

(Side topic - it is only US hosts that are blackmailing and profiteering enough to demand payment for restoring backups in a crisis - UK and Asian hosts (that I use) do not do that - it's very much a GRRRR point for me - but something to think about asking hosting companies before adding them to the WP hosting directory).

@dd32 - I haven't verified my sites have that code (please say where it should be) but I can absolutely guarantee it is not triggering, plus it is only half the issue - what about the entries with transient_ as the prefix? Those are not addressed by that code, and they make up around 35-40% of the entries.

If any of the devs want unrestricted access to all my sites on all hosts, contact me - I'll gladly give it in order to get this resolved (for everyone, not just me). If you'd prefer I send you a database dump - same applies.

I guarantee this is not fixed - there is something somewhere in all 2.9.x and 3.x versions that is perpetuating the MagPie bloat, and/or causing it to not be removed.

Also, I have noticed in server error logs, that wp_cron triggered events that include "scheduled_delete" related to the admin dashboard rss widgets (in particular "akismet_scheduled_delete\" and "wp_scheduled_delete\"), are causing database timeouts and "gone away" errors, therefore the scheduled deletes are not occurring. wp_unschedule_event also commonly appears in those errors.

Again, this is happening on all blogs on all hosts - far more on some than on others.

Regarding the accelerating rate of "SQL Server Gone Away" I am seeing on some domains, I have some suspicions mod_fcgid max_processes and max_process_duration limits (on shared hosting) may be contributory where wp_cron has many items to include in a triggered run. I am building up a new VPS today where I have full control of that, and am migrating some sites in the process to intrinsically test that theory - the sole purpose of the new VPS account - therefore I expect to have more data about it later this week.

Pseudo crons such as wp_cron have regularly been cited as server stress points (it happened with many different forum scripts too) because they rely on visitor traffic to trigger - on quiet sites (e.g. tightly geo-focussed sites during night-time quiet patches), this causes a task accumulation which then swamps resources when a visitor lands. Genuine crons to not cause this issue as they trigger according to the set period, regardless of traffic levels. I hope to do some experimenting on this (moving wp_cron to genuine cron) a little later in the year - I need to build a site for hacking-core first though.

Gaz

[nacin]

None of that has to do with the fact that core doesn't use Magpie, plugins shouldn't either, and that we're killing off Magpie caches on every database upgrade.

If you still have issues with any of the unrelated things you're talking about, you're welcome to visit the support forums. (Please, though, don't badger the moderators.)

[nacin]

Restoring the title.

[gazouteast]

.
Nacin - please stop blindly closing this ticket without giving time for input and feedback - you're doing no favours to the reputation of yourself or WordPress.

Here's another reason why this should stay open for further feedback -

Here's another _transient_feed_ entry type in wp_options that's beginning to fill up the table on a new site - in the home site table for a newish multi-site install (I haven't looked at the others yet)

a:4:{s:5:"child";a:1:{s:0:"";a:1:{s:3:"rss";a:1:{i:0;a:6:{s:4:"data";s:1:" ";s:7:"attribs";a:1:{s:0:"";a:1:{s:7:"version";s:3:"2.0";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:1:{s:0:"";a:1:{s:7:"channel";a:1:{i:0;a:6:{s:4:"data";s:0:"";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:2:{s:0:"";a:3:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:44:"link:http://domain.com/ - Google Blog Search";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:96:"http://blogsearch.google.com/blogsearch?scoring=d&ie=ISO-8859-1&num=10&q=link:http://domain.com/";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:78:"Your search - <b>link:http://domain.com/</b> - did not match any documents. ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:36:"http://a9.com/-/spec/opensearch/1.1/";a:3:{s:12:"totalResults";a:1:{i:0;a:5:{s:4:"data";s:1:"0";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:10:"startIndex";a:1:{i:0;a:5:{s:4:"data";s:1:"1";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:12:"itemsPerPage";a:1:{i:0;a:5:{s:4:"data";s:2:"10";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}}}}}}}}s:4:"type";i:128;s:7:"headers";a:9:{s:4:"date";s:29:"Wed, 22 Sep 2010 15:04:38 GMT";s:6:"pragma";s:8:"no-cache";s:7:"expires";s:29:"Fri, 01 Jan 1990 00:00:00 GMT";s:13:"cache-control";s:25:"no-cache, must-revalidate";s:12:"content-type";s:23:"text/xml; charset=UTF-8";s:10:"set-cookie";s:138:"PREF=ID=e5d45768c6809909:TM=1285167878:LM=1285167878:S=azuY3v6OcWLQ778d; expires=Fri, 21-Sep-2012 15:04:38 GMT; path=/; domain=.google.com";s:22:"x-content-type-options";s:7:"nosniff";s:6:"server";s:4:"bsfe";s:16:"x-xss-protection";s:13:"1; mode=block";}s:5:"build";s:14:"20090627192103";}

There's no Magpie reference, but it is an RSS item according to the beginning elements. According to the ending elements, it also expired seven days ago, but it is still in the database.

I have no idea what is inserting that entry to the options table - I can only assume it's from the incoming links widget in the dashboard - anyone confirm or deny that?

I have even less idea why it has not deleted - it expired a week ago. There are earlier non-deleted entries of the same format too.

The failure for it to be removed a week after expiry, shows the MagPie non-deletion issue has migrated to Simplepie and persisted there, according to my understanding of what I'm seeing - input anyone?

Therefore this trac topic remains as valid as when it was opened two years ago - more so, than then I would suggest.

[nacin]

That's the Incoming Links widget.

Expired transients are deleted once the transient is fetched post expiration. They are not deleted otherwise. They are also not autoloaded, so they do nothing to damage performance.

This has nothing to do with Magpie or Simplepie or any bug, for that matter.

[gazouteast]

OK - but still curious why that particular blog (6 months old with 3 posts) has got around 20 of those entries in wp_options, all expired, all for the same criteria, spread across the full age of the install.

I guess that's an "Ask me one on Sport" type of question?

Still, maybe the new VPS build will allow me to get some answers later in the week.