Excessive overhead in widgets.php on large site

[ronnieg]

My very large (WP 4.1.2 with all current theme and plugins) and complex real estate production website now has over 6,100 pages. Somewhere between page 5890 and 6100, the widgets.php admin page started to fail to completely load, and the server timed it out at 30 seconds. The "Available Widgets" section is there, but none of the active widgets are displayed. The front end of the site works fine, and all previously setup widgets are where they should be and with correct content.

My hosting service tech reports that the failure occurred as follows: "The error log shows maximum execution time exceeded (30s) in wp-includes/post.php function get_page_children. Specifically: PHP Fatal error: Maximum execution time of 30 seconds exceeded in .../public_html/wp-includes/post.php on line 4286".

Disabling all plugins and going to 2015 theme doesn't resolve the problem. My development site has 5890 pages, exactly same theme and active plugins as the production site, and it barely loads correctly, taking right at 30 seconds, 90% of the time, but fails 10% of the time with same result as the production site. This page count threshold for failure or success seems to indicate an overhead / performance issue with widgets.php and/or the functions that it executes. I added following to to wp-options.php, per suggestions in support forums for similar performance issues, and that didn't help:

define('WP_MEMORY_LIMIT', '128M');
define('CONCATENATE_SCRIPTS', false);

My hosting provider has graciously (and temporarily) increased the page load timeout limit. However, I do not understand why the function causing this problem is even needed by the widgets admin screen, or why it would need to (apparently) attempt to access every page / post and their children in the database to do its job.

If someone is eventually going to work on this issue, I can provide access to and/or test any code changes on my dev site.

[boonebgorges]

See #10852.

[ronnieg]

Replying to boonebgorges:

See #10852.

Granted, and should help. But that really doesn't address why that function is even needed or used for widget maintenance at all.

[LewisCowles]

It does seem to be a fair point, digging through the stack to find out why now, as the issue reported does not seem to need posts or pages (why would it need to cycle post heirarchies?)

At the least for managing widgets via widgets.php legacy screen I can see no reason to use that function.

What is most surprising is that this is affecting only that one page, can OP confirm this?

[ronnieg]

It is also affecting Pages / All Pages, which doesn't load anything at all, not even partial html. But #10852 seems like it should address that one, and I also have an acceptable work-around for that (plugin: Pages Tree View) so I didn't bring it up for this particular bug report. However, it does beg the question as to why every page / post has to be accessed all at once at the beginning in the Pages / All Pages function, instead of using a paginated approach, and just accessing the next 20 to however many it needs for a particular display page.

The Options / Reading page also seems to have a similar problem, but I have not researched that in detail yet. But it does seem strange that it also would need to access every page / post at the page load time.

[boonebgorges]

widgets.php has the Pages widget, which contains a dropdown of pages. Options > Reading contains a dropdown of pages for the "front page" option. Generating this dropdown means building the page tree. In some of these cases we might consider building the dropdown asynchronously, or only partially, or something like that.

However, it does beg the question as to why every page / post has to be accessed all at once at the beginning in the Pages / All Pages function, instead of using a paginated approach, and just accessing the next 20 to however many it needs for a particular display page.

When paginated page results are displayed in a way that respects page hierarchy, the hierarchy more or less has to be loaded in full before it can be decided which results belong on a given page. If you have a structure like this

A

AA

AAA

AB

ABB

AC

B
C

CA
CB

CBA
CBB
CBC

the second page of results when per_page = 5 should contain AC, B, C, CA, and CB. There's no way to calculate this without knowing what the whole tree looks like. See #15459.

[ronnieg]

In the case of the widgets.php page, if the Pages widget is not in an active widget area, which it is not on my site, nor on any of the many other sites I have built or support, then it shouldn't be necessary to build the drop-down list for it.

If it is dragged into an active widget area, then and only then should it be necessary to build a drop-down list. (Hmmm? A drop-down list of 6000+ pages? How would that ever be displayed and /or scrolled through?) And even then, it seems to me that specifying a starting page for that widget should be like the current custom menu build process. Let the user/admin open the widget in its assigned active widget area, then search for a starting page by page name, and only then would it need to build any trees.

And IMO, given the simplicity of custom menu building with the current WP version, and the plethora of available plugins that allow an admin to add a custom menu anywhere they want, in a widget area or on a page/post, and given the heavy resources utilization, I believe that the Pages widget should actually be a top candidate for permanent removal from WP. Comments on that idea?

Similar suggestion for the Options / Reading page. Give it the ability search for pages / posts when selecting one, instead of automatically creating a massive drop-down list, one that couldn't ever be displayed in any case. The required code is already there in the custom menu building process. These things may have made sense when WP sites were a couple dozen pages at most, but no longer, when larger and more complex sites are now more the norm.

In the meantime, I will try to figure out how to permanently disable the Pages widget, remove it from the Available Widgets list, and suppress its useless (IMO) overhead.

[ronnieg]

Replying to boonebgorges:
In some of these cases we might consider building the dropdown asynchronously, or only partially, or something like that. ...etc...

That seems to be exactly what the plugin "Pages Tree View" does. It displays only the pages at the root level at first, and only opens child pages of a selected root page as required, when the parent page is clicked, and then only for one child level at a time, and so on, not every page at all levels at once.

This "on demand / drill down" process makes a lot more sense to me, both functionally and performance wise. It also encourages and rewards a well structured page hierarchy design and implementation, when there are no more than a dozen or so main pages at the root level, and no more than a few dozen or so child pages under each major parent page. In the case of my 6000+ page site, I can drill down to the exact page I am looking for in seconds, with no more than 2-3 clicks.

All this perceived need for every page all at once is based on the assumption that every page in the hierarchy will need to be displayed, when from a user/admin standpoint, I am generally only interested in and working on a very small subset at any one point in time, so I am happy to be able to "drill down" into very specific branches of the hierarchy as I see fit and need. That's why the "Pages Tree View" plugin works so well for me and my clients, even though Pages / All Pages fails due to excessive overhead, combined with the very reasonable page load time limits dictated by my hosting service, so they can support good page load and response times for all of their customers.

[ronnieg]

I unregistered the WP default Pages widget with:

function unregister_default_widgets() { unregister_widget('WP_Widget_Pages'); }
add_action('widgets_init', 'unregister_default_widgets', 11);

As I wanted and expected, the Pages widget is no longer in the Available Widgets list. However, the functions that get the full list of all pages are still running, and the page takes 45 seconds to load now that the page load timeout has been relaxed. Shouldn't the logic that initiates this process confirm that the Pages widget, the one it is supposedly doing all of this for, is still registered? Is this a separate bug that needs a new ticket opened? Or is there some other WP default widget that also thinks it needs a list of every page on the site?

[LewisCowles]

I actually slipped into my own WordPress live site(personal live site), and slipped in var_dump of debug_backtrace to the mentioned function header. I then navigated to widgets.php in wp-admin, which dropped me into widgets page on the admin backend, and there is no backtrace from the function mentioned in #10852

This must be a separate issue, or a theme or plugin calling that function

[ronnieg]

Plugin culprit identified. Closed.