Store widgets in a custom post type instead of options

[westonruter]

Widget instances are stored in options. For a multi-widget (WP_Widget) the widget instances of a given type (id_base) are stored in a serialized array of instance arrays. A widget ID is comprised of a widget's id_base followed by a number which is the array index for that widget instance. For example, the third-created Text widget would have the ID text-4 (note that multi-widget numbering starts at 2). Old single widgets do not include the numeric index after the id_base, and technically they could be stored anywhere (see #35656 for suggestion to deprecate old single widgets).

Issues

There are several problems with how widgets are currently stored as options.

Scalability: For sites with a large number of widget instances, the entire collection of widgets must be unserialized with each request to access only one widget of a given type. (Note #23909 for how all widget instances get registered with every request.) For sites that use Memcached as an external object cache where cache buckets have a 1MB limit, since all widget instances of a given type are stored in a single option, sites with a huge number of widgets will overrun this limit. What's more is that widget options get registered as autoloaded, so all widget options will get combined together in the alloptions key, making widgets even more liable to overrun the 1MB cache bucket limit in Memcached.

Concurrency: Since all widget instances of a given type are stored in a single option, if two users attempt to update two separate widgets at the same time, it is possible that one of the updates will get lost (see #31245). Additionally, the widgets admin page and widgets in the Customizer both get loaded with the max number (array index) for each widget type. When a new widget instance is created, this maximum number is incremented in memory and used in the new widget ID which is then passed to the server for saving. If two users have loaded the UI at the same time, when they both create a widget of a given type and save their widget changes, the one who saves last will overwrite the other user's widget since the two widgets would have the same ID. (See #32183 for more about the widget ID collisions, and see ​Customize Widgets Plus for a “Widget Number Incrementing” component which uses Ajax to generate new widget IDs in a more concurrency-safe manner.)

Addressability: As noted above, widget instance IDs are comprised of the widget type's id_base followed by the array index number. Two different widget instances can have the same number, such as search-3 and text-3, since the number is incremented in the scope of the instances of the given type. No other objects in WordPress are identified by strings in this way, that is as of now: taxonomy terms actually used to have to be addressed by a numeric term ID and taxonomy name until term splitting happened in 4.2 (see #5809). Now, however, a term can be uniquely identified by a single integer ID.

All of the above issues would be resolved by switching to store widget instances in a custom post type, where each widget instance has a single unique auto-incremented post ID.

Advantages

Storing widgets in custom post type has several benefits beyond fixing the above issues, including:

• widget authorship attribution
• revision history
• import/export
• querying
• widget drafts
• scheduled widgets

Data Migration

Migrating widgets from options to a custom post type would involve some tedious data migration to update all references to current id_base-number widget IDs to their new integer IDs. The old widget ID could actually be copied directly into the post_name field for the widget_instance posts. Backwards compatibility for the sidebars_widgets option containing the old-style IDs may be necessary. Newly created widget IDs could have post_name fields populated with the id_base followed by the post ID. This switch would also necessitate discontinuing to register all widget instances with every request (#23909).

Sidebars and Widget Groups

Perhaps out of scope for this ticket, but the way that widgets get associated with sidebars should also perhaps be changed to follow the pattern of how nav menu items are associated with a nav menu via a taxonomy term. The implementing of widget groups (#19912) could be the right opportunity to do this, where a widget_grouping taxonomy could be introduced, and when a grouping is assigned to a sidebar, the backwards-compatible widget IDs could be copied into the existing sidebars_widgets option. Otherwise, backwards compatibility might entail adding pre_option_sidebars_widgets filter.

REST API Impacts

For more on widgets and now they relate to nav menu items in the context of a harmonized interface via the REST API, see ​https://github.com/WP-API/wp-api-menus-widgets-endpoints/issues/10

Feature Plugin

See the ​Customize Widgets Plus feature plugin's “Widget Posts” module for an initial implementation of storing widgets in a widget_instance custom post type. This plugin depends on #32474 which facilitated plugins to store widgets in posts instead of options.

[iamntz]

You can add on the advantage list:

• Quickly pick a certain widget instance for a certain page (link them together);
• Custom UI for widgets that is not limited to the widgets settings page.

Looking forward to see this implemented!

[MattyRob]

First I need to admit that I'm obsessive about my WordPress databases and I like to keep them just so.

Reading this ticket sent me into a panic that more 'non-post' stuff is going to be stored in the wp_posts table.

If this does get implement please, please, please can you keep the current system as a fallback and implement some sort of filter so I can keep my Widgets out of my post table.

I think for me the issue is that the wp_posts table is now poorly named and used for stuff it was not originally intended for. (I actually implement my nav menus though code to keep entries out of the database!)

[westonruter]

@MattyRob a “post” in WordPress now just means “object”. If the wp_posts table and post_type column could be respectively renamed to wp_objects (or wp_items, etc) and type I'm sure it would, but it remains for backwards-compatibility. So it is actually the “WordPress way” to put non-“post” stuff in thewp_posts table (e.g. nav menu items).

[PeterRKnight]

Replying to MattyRob:

First I need to admit that I'm obsessive about my WordPress databases and I like to keep them just so.

Reading this ticket sent me into a panic that more 'non-post' stuff is going to be stored in the wp_posts table.

If this does get implement please, please, please can you keep the current system as a fallback and implement some sort of filter so I can keep my Widgets out of my post table.

I think for me the issue is that the wp_posts table is now poorly named and used for stuff it was not originally intended for. (I actually implement my nav menus though code to keep entries out of the database!)

To extend on the reality that the wp_posts table is effectively an alias for wp_objects, if you are a fan of the databases, you should be rejoiced. Currently widgets are stored in the options table, which is not a great fit. Widgets define content after all and most options are autoloaded on every page load. As custom post objects the widgets can take advantage of post type-like features, such as authorship, scheduling, revisions etc.

[MattyRob]

Thanks for trying but it's not really helping :) (I did tell you I was obsessive!)

I've disabled revisions too - I want 1 table row for each post on my site without anything else in there, essentially consecutive ID numbers and I actually have that on my site via some plugin changes (for nav-menus and disabling revisions) and a small hack to the post-new.php file. Heck, I've even written a plugin for myself the re-IDs all of the wp_post entries in consecutive number order dropping auto-drafts, revisions and nav-menu entries.

I guess my other option if this goes ahead without any fall back is to disable Widgets in my themes and hard code them in my sidebar.php file.

As I said above it'd be nice to keep a fall back but if that's not possible these changes still need to work on themes without Widget support enable.

[ocean90]

#41686 was marked as a duplicate.

[paulschreiber]

+1 this

It solves the problem where saving a subset of widgets will remove the other widgets.

[westonruter]

#45890 was marked as a duplicate.

[oglekler]

I wonder if this is the time to raise this question again. Right now, new widgets are stored as autoloaded options, and they have no revisions and no publish date or modified date. Disadvantage will be in additional request to the DB if the widget is needed, but if we don't need it, there will be no requests and no autoloaded option.

The New Widgets editor is quite flexible with blocks with several restrictions, like the inability to add patterns there (and this is not very convenient either), but due to this flexibility, each widget needs at least revisions (and better to have drafts) - it is so simple to ruin the work or publish a half-finished widget.