Multisite rewrite rules are lost after core network upgrade

[litemotiv]

On two different multisite installations i am seeing that rewrite rules for individual blogs are not generated after a core upgrade, leading to 404s until they are manually regenerated.

I'll mark the severity as major since it leads to a potential large number of URLs being unreachable, please feel free to change this if necessary.

[jeremyfelt]

Hi @litemotiv, thanks for the report.

A few questions for you:

• Can you confirm that both of these installs were an upgrade from WordPress 5.2 to 5.3?
• Do both multisites have the same plugins installed? I hesitate to ask for a list, but it may be helpful.
• Are you using an object cache plugin on each?
• Was the upgrade performed through the network admin, through WP-CLI, or through another method?
• Did the option for rewrite_rules change to the default or was it still configured as before, just not working?

[litemotiv]

Thanks for looking into this one @jeremyfelt, to answer your questions:

• One of the multisites (small number of blogs) has been showing this issue for some time, perhaps about 1 year. I meant to look into it earlier but since the other one didn't exhibit this problem i figured it would be a local problem. The second multisite (about 70 blogs) has shown this behavior for the first time today with the update to 5.3

• The plugins are about 80-90% the same on both multisites, here is a screenshot of the plugins list on the multisite that exhibited this problem for the first time today with 5.3: ​https://i.imgur.com/Q5uF43I.png.

• No object cache is used, we use Varnish on the servers for full-page caching.

• The upgrade was done through the network admin.

• All the rewrite rules including custom ones are intact but it seems they were flushed and not regenerated. Manually visiting the permalinks settings page on each of the individual blogs restores them okay.

[litemotiv]

Some extra information:

• I currently have not updated the first multisite to 5.3 yet, so if there is anything you would like to know about that installation before i run the update please let me know.
• On the second multisite that showed this issue for the first time today, there are no specific errors in the server logs except a few pertaining to the collation of Formidable Forms tables, they look like this:

2019/11/13 12:20:01 [error] 17005#17005: *30496906 FastCGI sent in stderr: "PHP message: WordPress databasefout Illegal mix of collations (utf8mb4_unicode_520_ci,IMPLICIT) and (utf8mb4_unicode_ci,IMPLICIT) for operation '=' bij query SELECT c.*, p.post_name FROM vm2_frmpro_copies c LEFT JOIN vm2_211_frm_forms f ON (c.copy_key = f.form_key) LEFT JOIN vm2_211_posts p ON (c.copy_key = p.post_name) WHERE blog_id != 211 AND ((type = 'form' AND f.form_key is NULL) OR (type = 'display' AND p.post_name is NULL)) ORDER BY type DESC gemaakt door do_action('wpmu_upgrade_site'), WP_Hook->do_action, WP_Hook->apply_filters, FrmAppController::network_upgrade_site, rest_do_request, WP_REST_Server->dispatch, FrmAppController::api_install, FrmAppController::install, FrmMigrate->upgrade, do_action('frm_after_install'), WP_Hook->do_action, WP_Hook->apply_filters, FrmProDb::upgrade, FrmProCopiesController::install, FrmProCopy::install, FrmProCopy::copy_forms, FrmProCopy::get_templates_to_copy, FrmDb::check_cache" while reading upstream, client: 127.0.0.1, server: <domain>, request: "GET /wp-admin/network/upgrade.php?action=upgrade&n=5 HTTP/1.0", upstream: "fastcgi://unix:/var/run/php/php7.0-fpm.sock:", host: "<host>", referrer: "<referrer>/wp-admin/network/upgrade.php?action=upgrade"

The first multisite has no mixed collations for the Formidable tables (they are all utf8mb4_unicode_ci there), so it could be unrelated.

[litemotiv]

This occurred again on the 5.4 upgrade.

[litemotiv]

This is still occurring, currently with the 5.4.2 update.

This is a pretty serious problem since it causes around 100 websites to lose their permalinks instantaneously on our installation on each core update. Hoping for a further review of the issue from the WordPress team.

[litemotiv]

Does not occur anymore with 5.5, marking this resolved.