Make WordPress Core

Opened 15 years ago

Closed 14 years ago

Last modified 7 years ago

#12734 closed defect (bug) (fixed)

Nav Menus don't scale for big sites

Reported by: ptahdunbar's profile ptahdunbar Owned by: ptahdunbar's profile ptahdunbar
Milestone: 3.0 Priority: high
Severity: blocker Version: 3.0
Component: Menus Keywords:
Focuses: Cc:

Description

Solution:

  • limit all post types (except pages) and taxonomies to just 15 and sort by most popular.
  • Use suggest.js for autocomplete

Patch is in progress.

Attachments (3)

Popup Search.png (67.0 KB) - added by paperreduction 15 years ago.
Thickbox search to choose tags for menu.
Popup Search annotated.png (83.6 KB) - added by paperreduction 15 years ago.
Thickbox search to choose tags for menu (annotated with variations).
Screen shot 2010-05-22 at 10.52.18 AM.png (46.7 KB) - added by technosailor 15 years ago.
I think this is still an issue. The following menu required 49MB of memory. White screened for a memory_limit = 32M setup

Download all attachments as: .zip

Change History (37)

#1 @ptahdunbar
15 years ago

  • Status changed from new to accepted

#2 @jane
15 years ago

I'm super against limiting selection to 15 with autocomplete search being the backup. Selection and recall are two different things, and if we're giving them a menu creation system, the types we make available (which was supposed to be pages, categories, custom links only in v1) need to be fully available, not limited. If I've misunderstood what you're trying to suggest on this ticket, I apologize, and would ask for a more specific description. I'm not opposed to adding a Popular tab in the module itself i it seems like an issue, but it would be a blocker for me if the Add Categories modules did not list all the categories.

#3 @ptahdunbar
15 years ago

The primary bottleneck is on the menus page loading all the objects (post, pages, media, cats, tags, etc.) from the db and looping through them which is super memory intensive.

I discussed this with ryan and he came up with the idea to limit them by 15 (excluding pages, so that all pages are retrieved). Links aren't a bottleneck (yet) as it's new to WordPress so they also won't be limited. And I'll make sure categories aren't limited too.

FYI: There will be a filter to remove that limit for the other objects.

#4 @mikeschinkel
15 years ago

  • Cc mikeschinkel@… added

So how would someone be able to add a menu for one that isn't in the 15?
Hopefully I misunderstand this because it seems extremely limiting.

Rather than limit to 15 why not do load-on-demand using AJAX?

#5 @nacin
15 years ago

I agree with an ajax solution if possible. Otherwise, a limit is not a valid compromise. The compromise would be to remove the screen options and go back to allowed_in_menus filters.

#6 @ptahdunbar
15 years ago

Well the compromise is using search. If you're building a new nav menu, 9 out of 10 times, you probably already know which items are going to be in the menu.

The tentative filter is: wp_nav_menu_unlimited-{object_name}, which accepts an empty string for unlimited, or an integer for a specified amount of items to return.

I'll look more into an ajax approach.

#7 @jane
15 years ago

Search is not a compromise. Search a different workflow altogether. If it says "View All" then they need to be able to view all.

#8 @ryan
15 years ago

We limit tags to the most popular on the edit post form. That's because loading them all, even via AJAX, is not remotely feasible. The same is true of posts. As it is, loading all pages and categories is going to kill the menu screen.

#9 @jane
15 years ago

How does Woo do it?

#10 @ryan
15 years ago

Woo pulls in all pages and categories. Some sites have hundreds or thousands of each. Pulling in the categories is one of the main contributors to slow loading edit post forms. We've talked in the past of getting rid of the full list of categories on the edit post form if more than x categories. The menu admin is going to be hit extra hard since it pulls in both pages and categories (and possibly tags, posts, and other things).

#11 @ryan
15 years ago

Suggesting the following:

  • Show all for hierarchical post types and taxonomies (pages and categories). These tend to have less items and are typically what people actually want to add to the nav menu.
  • Show most popular for non-hierarchical taxonomies (tags). It is not uncommon to have thousands of tags, thus we never show all at once in existing UI. We show only the most popular.
  • Show most recent for non-hierarchical post types (posts). Posts usually number in the thousands and can have long titles. Even via an asyncronous load we shouldn't pull all of that in.

#12 @voyagerfan5761
15 years ago

  • Cc WordPress@… added

#13 @hanifb
15 years ago

I agree with ryan.

We could implement a thickbox search posts function that returns the post id.

#14 @paperreduction
15 years ago

I'm for the thickbox search, here's just an idea:

  1. Open thickbox with a small search form at the top,
  2. List of the top 20 or so items in a table right under the search,
  3. Include paging navigation with ajax to pull in next page (for those that like browsing, rather than searching),
  4. Allow user to select one or more items and click to add.

Perhaps some screen shots would help me.

@paperreduction
15 years ago

Thickbox search to choose tags for menu.

@paperreduction
15 years ago

Thickbox search to choose tags for menu (annotated with variations).

#15 @jorbin
15 years ago

  • Cc aaron@… added

#16 @gazouteast
15 years ago

  • Cc gazouteast added

Noooo - please no more thick thin or light boxes in admin - just opening the insert url box locks me up for almost 10 seconds at times in the editor. I've also had to inset images the hard way - via phpMyAdmin direct into the database - at other times because the media box took so long to open.

Make the display system for pages and categories a conditional based on existing number of them - they never appear on the same admin page after all - if less than 200 pages or categories, load them all at editor load, if more than that, then load ALL the top level pages or categories, with an ajaxed drop down off each for their next level, then their next level and so on.

With WP 3.0 those really big sites should be departmentalising to sub-blogs anyway if their hierarchies are so out of control as this topic suggests. Please don't punish the vast majority of your users with small to medium size hierarchies who are seeing no real performance issue with them loading in the editor pages.

For non-hierarchicals like tags - I'd suggest what is needed is a "moderator" tool to rationalise the tags -
examples =
eBay, ebay, Ebay, EBAY = 4 different spellings for the same tag.
US, USA, America, United States, etc = another tag that suffers from bloat due to variations.

There should be a semi-automated method to merge such examples into one tag .... perhaps an admin page that "lower-case alphanumerically" sorts all the tags based on the initial letter, using a tabbed menu, so only one tag initial at a time can be loaded? Admins could then trigger a merge to the correct (or most common) usage. This would drastically reduce tag list sizes, and heavily reduce the issue discussed above, especially on the biggest sites.

Gaz

#17 @nacin
15 years ago

We're not doing thickbox, and we're probably eliminating them for media next time we tear up media/upload.

We're ajax paging the nav menu meta boxes.

With WP 3.0 those really big sites should be departmentalising to sub-blogs anyway

That's not the purpose of multisite. Maybe a valid use case for some, but hardly a valid suggestion for all.

I'd suggest what is needed is a "moderator" tool to rationalise the tags

I have no idea what this has to do with this ticket, but scribu has a Merge Tags plugin. http://wordpress.org/extend/plugins/merge-tags/

#18 @ryan
15 years ago

(In [14557]) Reduce number of queries in wp_get_nav_menu_items() by fetching posts and terms with batch queries. see #12734

#19 @ryan
15 years ago

(In [14558]) No need to setup the nav menu items again. wp_get_nav_menu_items() handles it. see #12734

#20 @ryan
15 years ago

(In [14559]) Unset some vars when done with them to reduce peak memory usage. see #12734

#21 @ryan
15 years ago

(In [14560]) Don't prime menu item cache if a persistent cache is being used. Prime the menu item cache only on the first call to wp_get_nav_menu_items(). Avoids unneeded queries when wp_get_nav_menu_items() is called repeatedly. see #12734

#22 @ryan
15 years ago

(In [14562]) Defer term counting until all nav menu items are updated. see #12734

#23 @ryan
15 years ago

(In [14567]) No need to recount. see #12734

#24 @ryan
15 years ago

(In [14568]) Don't update term or meta cache for posts queried in wp_nav_menu_item_post_type_meta_box(). see #12734

#25 @ryan
15 years ago

(In [14569]) Don't wp_count_terms(). Just count the results returned from get_terms(), which has to run anyway. see #12734

#26 @ryan
15 years ago

  • Resolution set to fixed
  • Status changed from accepted to closed

@technosailor
15 years ago

I think this is still an issue. The following menu required 49MB of memory. White screened for a memory_limit = 32M setup

#27 @technosailor
15 years ago

  • Resolution fixed deleted
  • Status changed from closed to reopened

#28 @filosofo
15 years ago

technosailor, I created a menu similar in structure to your example. When I lower the memory just enough to exhaust it on wp-admin/nav-menus.php, it's also so low that it fails on other admin pages.

When I look at the memory use trace generated by Xdebug, I don't see anything associated with nav menus in particular.

So, if

  • your error logs show that the white screen was from memory exhaustion
  • the same memory level doesn't affect admin pages in general

then could you possibly attach a memory trace for your failure page load?

Thanks

#29 @ryan
15 years ago

I've been making some pretty big and nested menus and showing extra boxes for custom taxonomies and posts. The peak memory usage has stayed pretty sane. Well under 32. Of course my initial limit is pretty low since I'm not running a bunch of plugins.

#30 @markjaquith
15 years ago

On my live blog, with a whole bunch of plugins, a 75 page menu with crazy hierarchy (up to the max depth allowed) only takes 18 MB at its peak. A brand new blank menu peaks at 3.65 MB. A twenty member menu uses 5.53 MB

All these numbers are memory_get_peak_usage() / 1024 / 1024 in admin_footer, on a save of the menu. Displaying the menu takes a little less.

Hierarchy doesn't affect it to any significant degree.

I'm guessing Aaron's install is already using a significant amount of memory as a base, or that he has a plugin that is somehow intensifying the memory usage of the menu code.

#31 @nacin
15 years ago

  • Resolution set to fixed
  • Status changed from reopened to closed

#32 @radial
14 years ago

  • Cc radial added
  • Resolution fixed deleted
  • Status changed from closed to reopened

This is definitely not fixed. We have a VPS with 1.98ghz processor and about 1.5gb of RAM.&nbsp; Our nav menu contains about 150 items. When saving the menu, it may take a full minute just to save this.&nbsp; CPU usage for the nav-menus.php process peaks out at about 85% for that minute.&nbsp; If the user thinks somethings wrong and backs out or reloads, the saving stops and cuts off the menu items that have not been saved.&nbsp; It's a PITA to have to recreate the menu, especially when it's not easy to find the pages because they aren't listed in any order, and the search for pages doesn't work.<br>

#33 @scribu
14 years ago

  • Resolution set to fixed
  • Status changed from reopened to closed

Please open a new ticket if this is still a problem in WP 3.1-alpha.

#34 @westonruter
7 years ago

In 41982:

Menus: Don't prime nav menu items cache if a persistent cache is being used.

Amends [14560].
Props david.binda.
See #12734.
Fixes #41662.

Note: See TracTickets for help on using tickets.