WordPress.org

Make WordPress Core

Opened 9 years ago

Closed 8 years ago

#3614 closed defect (bug) (fixed)

Scaling issue with hundreds of pages

Reported by: markjaquith Owned by:
Milestone: 2.5 Priority: high
Severity: critical Version: 2.0.7
Component: Optimization Keywords: needs-patch
Focuses: Cc:

Description

http://wordpress.org/support/topic/99953

WordPress appears to be having scaling issues with Page.

The symptom is freezing, crashing or extreme sluggishness when saving a Page.

The bottlenecks are likely the page_uris and rewrite_rules options. With hundreds of pages, these options are going to be huge, introducing both a MySQL burden for saving/storing them, and a PHP burden serializing/unserializing/manipulating them.

Tricky issue, this.

Attachments (4)

manage pages diff.patch (5.9 KB) - added by pirom 8 years ago.
Solves the freezing, crashing or extreme sluggishness of Page Management view for WP 2.3 (Trunk) [Patch Version 5772]
3614-page-after-filter.diff (1.3 KB) - added by DD32 8 years ago.
Add the catch-all page rules after the filtering is done to ensure custom rules are before
page_attachment_query.diff (1.3 KB) - added by ryan 8 years ago.
Query page attachments from page requests
enable_generic_page_rules.diff (874 bytes) - added by ryan 8 years ago.

Download all attachments as: .zip

Change History (55)

comment:1 @ryan9 years ago

[4773] should help. I thought I checked this in long ago, but evidently I forgot to. Went ahead and put it in 2.1.

comment:2 @markjaquith9 years ago

That'll buy us a little bit of breathing room. Good first step. Ideally, I'd like to see page rewrite rules be compressed into a couple generic rules, stuck at the bottom of the list.

comment:3 @rob1n8 years ago

  • Component changed from General to Optimization
  • Priority changed from normal to high
  • Severity changed from major to critical

comment:5 @ryan8 years ago

Patch is old but might have some bits worth gleaming.

comment:6 @besonen8 years ago

  • Cc davidb@… added

comment:7 @charleshooper8 years ago

  • Cc charleshooper added

comment:8 @rob1n8 years ago

  • Milestone changed from 2.2 to 2.3

comment:9 @rob1n8 years ago

  • Keywords needs-patch added

comment:10 @pirom8 years ago

I found a patch for WP 2.0.x and WP 2.1.x at:
http://www.marshinsoftware.com/software/wordpress_page_management_patch/

I tested the WP 2.1.x version on WP 2.2 and both my users & I haven't found any problems.

I tried to test it thoroughly by going through the change logs from WP 2.1 to WP 2.2 and copying the changes over, but I was only able to get through the first quarter.
There were no changes.

@pirom8 years ago

Solves the freezing, crashing or extreme sluggishness of Page Management view for WP 2.3 (Trunk) [Patch Version 5772]

comment:11 @ryan8 years ago

  • Milestone changed from 2.3 to 2.4 (next)

comment:12 @foolswisdom8 years ago

Relates to #5303?

comment:13 @kicksonfire8 years ago

I have been dealing with this issue for months and have found no solution either.. I indeed emailed the developers. What I want to know it ... What are the guarentees that this issue will be addressed in the next Wordpress version (2.4) ? Because it has been 11 months since the initial start of this ticket... and many releases have been released since.

comment:14 follow-up: @ryan8 years ago

I would like to make some improvements in 2.4, but there are no guarantees it will happen given everything else that is scheduled for 2.4. Here are my thoughts on how to address the page rewrite rules.

We currently have 9 rewrite rules per page. This, obviously, sucks.
The rules are there to avoid collisions for those who use category or tag
bases of "/" or those using %category%, %tag%, or %postname% at the
beginning of their permalink structure. Most people don't do this,
making all of those page rules a huge waste. For those who don't use
any of the above, let's add a catch-all set of page rules at the end
of the rewrite rule list. Those who do use the above will get the old
behavior. Losing hundreds of rewrite rules (thousands for some
people) and the need to update the rewrite rules every time a page is
changed would greatly improve performance.

comment:15 in reply to: ↑ 14 @DD328 years ago

Replying to ryan:

We currently have 9 rewrite rules per page.

The other thing i noticed about that, Is that they're all before the more commonly used pages(eg. blog posts, comments, categories/tags)

Maybe its possible to have them ordered somehow so that less commonly accessed URLs(such as trackback/attachment) are after the rest of the rules?

Some people of course, Will want pages first before the blog post rules if they're using it for CMS reasons, but the majority of page requests are going to be post related, not page related.

(I'm not too sure if this can be solved by changing priorities on some hooks, i've not looked into the rewrite details)

comment:16 @ryan8 years ago

(In [6351]) Don't save page and attachemtn uris to page_uris and page_attachment_uris. This is not needed. Add an option to use wildcard page rewrite rules instead of per-page rules. see #3614

comment:17 follow-up: @ryan8 years ago

I did a little housekeeping. We were saving page and attachment rules in the options DB as page_uris and page_attachment_uris. This is not needed now that we cache the entire rewrite rule set to the options DB. page_uris and page_attachment_uris are now gone. I also added an option to force use of a generic set of wildcard page rules. Set WP_Rewrite::user_verbose_rewrite_rules to false to get the generic rules. With this set, you will get nine page rules total instead of nine per page. This code is still rough, so you have to change line 188 by hand. After changing true to false, visit your permalink options page and click the update button to force a rule refresh.

comment:18 in reply to: ↑ 17 @DD328 years ago

Replying to ryan:

Set WP_Rewrite::user_verbose_rewrite_rules to false to get the generic rules. With this set, you will get nine page rules total instead of nine per page.

The catch all page option breaks any rules which are added by plugins via the 'generate_rewrite_rules' action or 'rewrite_rules_array' filter, as they're added after the catch-all page rule.

@DD328 years ago

Add the catch-all page rules after the filtering is done to ensure custom rules are before

comment:19 @DD328 years ago

Add the catch-all page rules after the filtering is done to ensure custom rules are before

Allthough on 2nd thoughts, The API of add_rewrite_rule() should probably be used instead by plugins, in which case, $this->extra_rules would go before $page_rules in the merge.

comment:20 @oledole8 years ago

I've posted on this issue in this thread in the wp-forum http://wordpress.org/support/topic/99953?replies=27#post-657683.
"
Using the patch ... some pages don't show up in Manage->pages, e.g. no subpages below level 3, and almost no pages with two-digit pageID. Any ideas way?"

"Regarding the patch: I'm using the page category organiser-plugin, and it opens its "manage pages -page" without problems. It looks as if it's using a similar function as the patch above to get the pages to list, so that might be a way to proceed ...?

comment:21 follow-up: @kicksonfire8 years ago

Hey,
I added the new rewrite.php file. Replaced the line 188 (true to false). And created a page and it worked super fast! I resaved it and work fine. BUT HUGE PROBLEM. First of all the parent category I created the page in no longer works and also the page i created no longer works. I get a 404 error for both. So this fix definitely did not work for me.... I have to go back to my old rewrite.php file now....

comment:22 in reply to: ↑ 21 @kicksonfire8 years ago

Replying to kicksonfire:

Hey,
I added the new rewrite.php file. Replaced the line 188 (true to false). And created a page and it worked super fast! I resaved it and work fine. BUT HUGE PROBLEM. First of all the parent category I created the page in no longer works and also the page i created no longer works. I get a 404 error for both. So this fix definitely did not work for me.... I have to go back to my old rewrite.php file now....

Also, when i reloaded my old rewrite file, the page I created started working and everything was fine. So basically the new file worked to create the page yet it didnt display it. The old rewrite file was able to show the page.

comment:23 @ryan8 years ago

What are your permalink structure, category base, and tag base? Give examples of links that 404.

comment:24 @kicksonfire8 years ago

I am using /%postname% in my custom permalinks field... I actually deleted the pages already. But if you want to look into it. I will grant you access its my please. Please contact me on ceokhan@… or kicksonfire@…

comment:25 @ryan8 years ago

You can't turn off verbose page rules for permalink structures starting with %postname% or %category%. Those are wildcards that match everything. Since they come before the page rules, the page rules are never reached. Permalink structures like yours are the reason we have to do verbose rules for pages. We need those rules to avoid rewrite collisions.

The /%year%/%monthnum%/%day%/%postname%/ permalink structure avoids such collisions for all but one case. Pages that have names that look like years (2007) will 404 because they collide with the %year% rule. With verbose rules on, the page will win and the year will lose.

comment:26 @kicksonfire8 years ago

So what are my options? What am I to do? Am I suppose to live with this? Pleae advise me....

@ryan8 years ago

Query page attachments from page requests

comment:27 @ryan8 years ago

Patch attached that allows querying both pages and page attachments from a pagename request. This is needed to support page attachments with the non-verbose rules.

kicksonfire, your permalink structure complicates things. We'd probably have to add a new query var that queries against post, page, and attachment post_types for a given post_name.

comment:28 follow-up: @skydrak8 years ago

I only get the php error: Fatal error: Call to undefined method wpdb::insert() in /var/www/*/html/wp-includes/post.php on line 730 when i upload the new post.php. :(

comment:29 in reply to: ↑ 28 @DD328 years ago

Replying to skydrak:

I only get the php error: Fatal error: Call to undefined method wpdb::insert() in /var/www/*/html/wp-includes/post.php on line 730 when i upload the new post.php. :(

What version of WordPress are you using? AFAIK wpdb::insert() is only available on trunk(2.4), and nothing in this ticket uses the insert() function.

comment:30 follow-up: @skydrak8 years ago

I use the latest version, 2.3.1.

Tested it again with removing the code by hand in the post.php, but same problem as before: Crashing the server.

comment:31 in reply to: ↑ 30 @DD328 years ago

Replying to skydrak:

I use the latest version, 2.3.1.

Yeah, You cant just use the latest post.php file, theres too many changes in it which is incompatible with the rest of the 2.3.1 source code.

You'll have to apply the changes in this changeset: [6351] to a stock 2.3.1 file if you want to use them, and even then, i'm not sure if that'll work due to oher file changes.

comment:32 @skydrak8 years ago

Well, i have tested it... Works fine, permalinks aren't stored in the database now. But same problem as before: i a will save a static page my server will crash, so i have to rename the index.php everytime if i want to save a static page. That sucks.

comment:33 @ryan8 years ago

(In [6377]) Query page attachments from page requests so that they can both use the same set of rewrite rules. see #3614

comment:34 @ryan8 years ago

Patch automatically uses generic page rules if a compatible permalink structure is being used. No need to change the value by hand.

comment:35 @kicksonfire8 years ago

Alright... So is this the final solution. is the problem solved? Now I am really confused as there are 4 different patches:

manage pages diff.patch (5.9 kB) - added by pirom on 06/28/07 15:11:01.
3614-page-after-filter.diff (1.3 kB) - added by DD32 on 12/04/07 02:31:45.
page_attachment_query.diff (1.3 kB) - added by ryan on 12/11/07 08:43:10.
enable_generic_page_rules.diff (0.9 kB) - added by ryan on 12/13/07 21:38:14.

Which ones do I need to use if I haven't edited any of the original files as of yet...
Please advise.

comment:36 @ryan8 years ago

(In [6398]) Use generic page rewrite rules for compatible permalink strucures. see #3614

comment:37 @ryan8 years ago

You need [6351], [6377], and [63948]. These diffs are against svn trunk, however, not 2.3.1. I don't know if they will apply to 2.3.1. None of these changes will make a difference to permalink structures starting with %postname%. Supporting that takes a lot more work and may not be fixed for the next release.

comment:38 @skydrak8 years ago

ryan, DD32 you are the greatest. Added the modifications to my wp 2.3 blog and it works perfectly. If someone is interested in the modified files I can upload them.

comment:39 @kicksonfire8 years ago

Hey skydrak,
I am running 2.3.1 and I am kind of a novice with this kind of stuff. I would greatly appreciate it if you can upload the modified files so I can just upload them onto my server. THank you in advance!

comment:41 @kicksonfire8 years ago

Thank you for the files but big problem... AFTER i added them to wp-includes to replease the old ones, I created a page and got this error:

Fatal error: Call to undefined method: wp_rewrite->page_uri_index() in /home/kofcom/public_html/wp-includes/rewrite.php on line 290

comment:42 in reply to: ↑ 40 ; follow-up: @daltonrooney8 years ago

Thanks for providing those files, Skydrak. I was experiencing the same issue and installed the modified files, which did seem to fix the initial problem.

An interesting side-effect is that I can no longer access pages that are children of my main blog page. I have Wordpress isntalled in a sub-folder, so most of my pages are peers with the blog page. The few that are sub-pages of the blog show up in the manage section of the admin, but when I click "view" or try to navigate to the URL directly, I get a 404.

comment:43 @daltonrooney8 years ago

Oops, I can't access paged archives on the blog, either. Looks like this patch won't work for me on 2.3.1.

comment:44 in reply to: ↑ 42 @ryan8 years ago

Replying to daltonrooney:

Thanks for providing those files, Skydrak. I was experiencing the same issue and installed the modified files, which did seem to fix the initial problem.

An interesting side-effect is that I can no longer access pages that are children of my main blog page. I have Wordpress isntalled in a sub-folder, so most of my pages are peers with the blog page. The few that are sub-pages of the blog show up in the manage section of the admin, but when I click "view" or try to navigate to the URL directly, I get a 404.

Can you provide example URLs as well as your permalink structure?

comment:45 @daltonrooney8 years ago

Nuts. My permalink structure is just %postname%, and reading through this thread again I realize that is the problem. Changing the permalink structure to include a year first seems to have taken care of that issue, sorry for the false alarm.

comment:46 @kicksonfire8 years ago

I am trying to implement this on http://www.kicksonfire.com but I will have to change my permalinks. The only thing I have a problem with not is that all my pages have links to eachother and are part of a link structure. If i change the permalinks, none of these links will work and I will have to edit over 2000+ pages... Any thoughts.

comment:47 @rmccue8 years ago

Wouldn't it be better to just write static rewrite rules to .htaccess if we can?

comment:48 follow-up: @kicksonfire8 years ago

Got the pages issue fixed with all the help here and the patches. But one problem I run into now is that I have lost all my traffic due to my permalinks change.... I have tried several permanent redirect plugins but none of them work on my blog for some reason! Is it because I have changes these files??

comment:49 in reply to: ↑ 48 @DD328 years ago

Replying to kicksonfire:

Got the pages issue fixed with all the help here and the patches. But one problem I run into now is that I have lost all my traffic due to my permalinks change.... I have tried several permanent redirect plugins but none of them work on my blog for some reason! Is it because I have changes these files??

Unless we know the permalink structure, Its impossible to suggest help, You'd be better off asking in the support forum i think.

It'll be possible to write some .htaccess redirects for your permalink structure though, just dont think you'll find it in a pre-made plugin.

comment:50 @kicksonfire8 years ago

Well, I couldn't use these patches unless I changed my permalinks as Ryan stated earlier... I was using /%postname%/.... Now I have changed it to: /%year%/%monthnum%/%day%/%postname%/ in order for the patches to work properly. Everything works now... except that all my old slug pages crawled by google return a 404 error. I tried SEVERAL permalinks migration plugins (Advanced Permalink & Dean's Migration) but they didn't work for me as my files have been changed... Please help me because I completely get evaporated from all search engines.

comment:51 @ryan8 years ago

  • Resolution set to fixed
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.