Export: Allow multiple post types to be selected

[hlashbrooke]

The export tool currently only allows for all post types to be exported, or for only one to be exported. I propose fixing this so that the selection is made via checkboxes and any number of post types can be exported. This will be beneficial in many use cases.

[hlashbrooke]

First pass at updating UI and export function

[hlashbrooke]

I have created a patch (attached) for this that *almost* solves the problem completely. I have updated the UI and handled all the processing of the submitted form. Everything works perfectly if you do not select a specific category for your posts - if you select a category then it will only return posts in that category and ignore all other post types. Unfortunately my MySQL skills aren't good enough to overcome this issue, so maybe someone else can help out here.

The problem is that by selecting a category it joins the term_relationships table which causes it to ignore any CPT posts that do not have an entry in that table as well as any CPT post with a different taxonomy ID (which will always be all of them). I've tried numerous IF statements and the like, but I haven't been able to work it out. Any help would be appreciated :)

[mordauk]

Yes please!

Aside from the category issue, the patch appears to work well, though I did notice there's an issue with the Posts post type getting two checkboxes: ​http://screencloud.net/v/krbl

[mordauk]

Nevermind, ignore my comment about the duplicated Posts, I had a custom post type without a label so it was showing up as Posts.

[hlashbrooke]

I'm going to work on the SQL a bit more, but if anyone else can work their magic on it then that would be awesome. Since I started working with WP every day, my SQL skills have diminished somewhat.

I actually might see if I can switch it to use standard WP queries instead of DB calls for as much as possible.

[SergeyBiryukov]

Replying to hlashbrooke:

I actually might see if I can switch it to use standard WP queries instead of DB calls for as much as possible.

That would be great.

[hlashbrooke]

Moving to WP_Query instead of raw DB queries

[hlashbrooke]

I've uploaded a new patch that switches the main export query to a WP_Query instead of the raw DB queries. This now includes all of the export functionality correctly and as far as I can tell it's all working just like it should. I also added menus in as a selectable option giving you more control of what you export.

The queries should all be spot on, but if you see something that isn't quite right (or that can be optimised) then please let me know :)

[netweb]

I like this, probably going to need some updated unit tests.

Ref: /trunk/tests/phpunit/tests/export & /trunk/tests/phpunit/tests/import

[hlashbrooke]

Replying to netweb:

I like this, probably going to need some updated unit tests.

Ref: /trunk/tests/phpunit/tests/export & /trunk/tests/phpunit/tests/import

How are unit tests handled? I haven't delved into the world of unit tests yet, so not sure where to start. Do core devs handle that on here or is that the committer's responsibility?

[danielbachhuber]

It's the developer's responsibility to produce a patch that includes unit tests. ​The handbook has a good introduction. Your tests can be included in the same patch file as your change.

[hlashbrooke]

OK cool - I've installed PHPUnit, but the section of the handbook title 'Writing Tests' is sadly unhelpful: ​https://make.wordpress.org/core/handbook/automated-testing/#writing-tests. I'll look further though and I'm sure I'll work it out - I've heard this stuff isn't too difficult to get right, so I'm sure I won't struggle too much :)

[mordauk]

I can probably help out and get some updated tests written.

[hlashbrooke]

Replying to mordauk:

I can probably help out and get some updated tests written.

That would be awesome. If you could write some tests for this it would be a great learning tool for me - knowing what the code is supposed to do will help me understand the tests better so I could learn for future patches. I know that's a pretty selfish reason, but I would really appreciate it :)

[netweb]

It looks like the tests in /tests/phpunit/tests/export/class-wp-export-query.php need to be expanded.

Easier said than done...

If you run phpunit --group export results in Tests: 33, Assertions: 0, Skipped: 33.

To forcibly run the skipped tests for an open ticket, you may specify that ticket as a group:

If you run phpunit --group 22435 results in Tests: 39, Assertions: 0, Skipped: 39.

If I run phpunit with the --verbose switch various errors occur:

WP_Export_Query class doesn't exist
wp_export function doesn't exist
WP_Export_Returner class doesn't exist
WP_Post_IDs_Iterator class doesn't exist

All the tests class-wp-export-query.php are attached to @group export & @ticket 22435

Ticket #22435 Export API.

Looking at r26370:

"When setting WP_TESTS_FORCE_KNOWN_BUGS to true, it is preferable that some forced tests are still skipped when they call classes or functions that do not exist, producing fatal errors."

This is all pretty much getting beyond me now and it hurts ;)

Some extra feedback and guidance would be extremely helpful please :)

[hlashbrooke]

The problem here is that the unit tests were rewritten for #22435 presumably assuming that it will be added to core in the next release. Does anyone have a copy of the old unit tests for the export function? Because they would probably only some very small modifications to work with this patch.

That being said, if #22435 is going to be included in core then that makes this patch redundant, so there'll be no need for further testing.

[helen]

Unfortunately a little late for this - we really should get some eyes on the general export API. We also still have problems like dropping attachments on filtered exports: #17379.

[hlashbrooke]

Replying to helen:

Unfortunately a little late for this - we really should get some eyes on the general export API. We also still have problems like dropping attachments on filtered exports: #17379.

Fair enough if this won't fit in 3.9 (although saying it's too late isn't really true as I submitted this patch a month ago), but I've actually solved the problem that #17379 mentions, so it might be worth considering again :)

In this patch all the post meta is now exported for all post types that are selected, so this would actually be a pretty neat improvement and bug fix until the export API is completely rewritten in a future version (as submitted in #22435).

[hlashbrooke]

I know this still needs unit tests, but like I said before - the core unit tests repo had the export function test rewritten for #22435, which has not actually been accepted yet. Because of this I don't have any way of running the correct unit tests for this patch. Any help would be appreciated.

[hlashbrooke]

Just FYI - I've added this new export tool as a plugin: ​https://wordpress.org/plugins/export-plus/

[jtsternberg]

Can we do this thing? :) What are the blockers at this point?

[hlashbrooke]

Replying to jtsternberg:

Can we do this thing? :) What are the blockers at this point?

See my comment above - we need unit tests, but someone rewrote the core tests for a different patch so now they no longer work for the current export tool or this patch of mine. My knowledge of unit testing is poor at best, so I'm going to need some help with that side of things.

[atomicadam]

Any chance this plugin will solve the featured image meta data not being exported when just exporting posts? I didn't see this a feature in the plugin features list, nor does it seem to do it (at least when I tried), but it would be a great thing to solve. Thanks for the work.

[sc0ttkclark]

@hlashbrooke If we're going to use get_posts to get the post IDs out, we should be using 'fields' => 'ids' to pull out only the IDs. If we're going to use get_posts at all, we should consider ripping out some logic down below to just utilize WP_Query and a ->have_posts() / ->the_post() loop to consolidate all of the logic and utilize object caching that WP_Query affords.

As for unit tests, we should indeed have a look at those and figure out what we need to do to get export_wp / this ticket covered under the tests again, or at least setup some new tests.

[sc0ttkclark]

I think splitting this up into #28146 for the export_wp updates would be better here, that way we can move forward with the export_wp updates including the new filter on query args.

[hlashbrooke]

Refreshed and updated for WP 4.4

[hlashbrooke]

This has been dormant for a little while, but I think it's worth looking at it again as I think this will be a hugely valuable enhancement to core.

I have refreshed the patch against the current trunk, so it's all up to date in 27048.2.diff.

To update everyone with where we're at with this patch:

• All export queries are switched from raw MySQL to WP_Query
• Post type selection is switched to checkboxes to allow any selection of post types to be selected (including attachments and nav menus)
• Author, date range and status filters now apply to all post types being exported

I have not included unit tests, partly because (rather ashamedly) I have not learnt how to write them yet, but also because I have merely modified existing functions (most notably export_wp() of course), so the existing unit tests for the export code will still cover the same ground as this patch leaving no need for brand new tests.

I would really like to see this land in core as it will solve a rather large usability issue with the exporter, in that you have to do multiple exports to get a specific set of post types (not to mention then featured image dropping bug that is also fixed in here).

[hlashbrooke]

Screenshot of updated export tool

[hlashbrooke]

Tweaks to bring back responsive classes

[hlashbrooke]

In addition to the updates I'm proposing here, the following tickets are also inadvertently fixed in this patch:

• #17379 - Featured images are no longer dropped from posts
• #28146 - I have included an export_wp_post_ids filter in the updated export_wp() function
• #19307 - The date queries apply to all post types now
• #26311 - The new screen looks just fine on mobile
• #28375 - The post type conditional is no longer necessary and has been removed

This patch will also make tickets like #34062, #32266, #20903, #31084 & #34414 much easier to fix.

[kirasong]

I don't think these efforts conflict, but wanted to be sure you were aware of @rmccue's work rewriting the importer:
​https://make.wordpress.org/core/2015/11/18/wordpress-importer-redux/

[hlashbrooke]

Replying to mikeschroder:

I don't think these efforts conflict, but wanted to be sure you were aware of @rmccue's work rewriting the importer:
​https://make.wordpress.org/core/2015/11/18/wordpress-importer-redux/

Yeah I'm aware of @rmccue's work there (and very excited for it) - this won't conflict because the exported file that is produced is no different to what the current export produces. The difference here is the control over what you export, but the actual exported content is identical.

[netweb]

@hlashbrooke I'm adding back the needs-unit-tests keyword, there aren't actually and existing tests, hence one of the reasons of importance for them. (They are only included in the Export API patch)

My biggest concern in your latest patch is the switch from raw MySQL queries to WP_Query.

An example of this was r36068 that shipped with WordPress 4.4.1 which reverted a change to WP-Query from raw MySQL queries in WordPress 4.4 for non-public post types, as such testing post types defined with exclude_from_search and custom post statuses will be required to be tested here.

My knowledge of units testing has come a long way since my last comment in this ticket 2 years ago so I'll try and find some time to see if I can get some tests up and running.

[hlashbrooke]

@netweb OK - that makes sense - I thought the unit tests must have existed somewhere from previous work on the exporter, but the tests obviously never covered this area.

In terms of switching to WP_Query instead of raw MySQL, I'm not sure I fully understand why that's a problem, but if it really won't work then I could remove that from this patch and just stick with the UI changes here. That wouldn't be ideal though as the MySQL queries get a lot more complex at that point and I know I struggled to get them working and performant before I switched to using WP_Query.

Is there no way to work around the issue where some content types won't be exported? I haven't experienced any issues with this yet myself, so it would be helpful to know exactly what the issue is and what doesn't get included so I could see how we can work around it in here.

[SergeyBiryukov]

#55765 was marked as a duplicate.