More hooks for WordPress Importer

[batmoo]

It would be nice to add a number of additional hooks to the WordPress Importer so that plugins and other tools (like CLI scripts) can hook and modify imports on-the-fly.

Patch adds a number of new filters:

• wp_import_categories (before categories are processed)
• wp_import_tags (before tags are processed)
• wp_import_terms (before terms are processed)
• wp_import_posts (before posts are processed)
• wp_import_post_data_raw (modify the raw post data pre-insert)
• wp_import_post_data_processed (modify the processed post data pre-insert)
• wp_import_post_terms (modify terms for a post)
• wp_import_post_comments (modify comments for a post)
• wp_import_post_meta (modify postmeta for a post)

New actions introduced:

• wp_import_post_exists (when a post already exists)
• wp_import_insert_post (when a post is successfully inserted)
• wp_import_insert_term (when a term is successfully inserted)
• wp_import_insert_term_failed (when a term insert fails)
• wp_import_set_post_terms (when terms for a post are added)
• wp_import_insert_comment (when a comment for a post is added)

Modified one existing filter:

• import_post_meta_key (gets the $post_id and $post params passed in)

Sample code that hooks in to these filters (excuse the PHP 5.3-ness).

Make the import more verbose:

add_action( 'wp_import_insert_post', function( $post_id, $original_post_ID, $postdata, $post ) {
 	if ( is_wp_error( $post_id ) )
 		echo "-- Error importing post: " . $post_id->get_error_code() . PHP_EOL;
 	else
 		echo "-- Imported post as post_id #{$post_id}" . PHP_EOL;
 }, 10, 4 );

Modify post data on the fly:

add_filter( 'wp_import_post_meta', function( $postmeta, $post_id, $post ) {
	$postmeta[] = array(
		'key' => '_imported_author',
		'value' => sanitize_user( $post['post_author'], true ),
	);
	return $postmeta;
}, 10, 3 );

Exporter hooks: #19863

I think this is a bit too much. Why can't you do all these things after the initial import step?

I guess these 3 are a bit redundant and can be handled using wp_import_post_data_processed:

• wp_import_post_terms
• wp_import_post_comments
• wp_import_post_meta

Replying to scribu:

I think this is a bit too much. Why can't you do all these things after the initial import step?

Do you mean using the wp_import_step* actions?

I don't know what wp_import_step* actions you're talking about.

I was referring to this workflow:

1. Import data.
2. Go through all data and make the necessary adjustments (delete unwanted posts, change the tags, etc).

Replying to scribu:

I was referring to this workflow:

1. Import data.
2. Go through all data and make the necessary adjustments (delete unwanted posts, change the tags, etc).

After-the-fact fixer scripts are super slow when you're dealing with hundreds of thousands of posts. It can literally take multiple days to run and the cache invalidation is not fun.

Doing the fixing in an automated fashion during the import process really is the fastest way.

Can you really have too many hooks?

Each call to apply_filters() takes a bit of time, which adds up "when you're dealing with hundreds of thousands of posts".

But more importantly, each hook needs to be maintained as the code around it changes, which gets really tricky sometimes.

So, I'm not saying that more hooks aren't useful here; just that each one needs to be considered carefully.

Not to derail this ticket, but apply_filters() is insanely fast. Like you can run 250,000 calls a second no joke.

But anyway, maintaining filters is understandable but I don't think a blocker. These are power user filters and actions and it would be understandable if they broke.

But right now the alternative is not pretty.

Unless I'm mistaken, the WordPress.org milestone is for things affecting the actual WordPress.org website. This ticket is about WordPress the software, not WordPress the website.

Actually, the WordPress.org milestone is used for ​plugins supported by the core team as well.

We could have a separate milestone just for those plugins], but what should we call it?

• "WordPress.org plugins"
• "Core plugins"
• "Canonical plugins"

You see the difficulty. :)

Currently it's very difficult to import custom tables where one of the columns references a core WordPress table. Hooks like wp_import_insert_post would help a lot in keeping check of how posts have been imported (i.e. Old ID -> New ID).

In fact, all that would be required is one action, preferably passing the $wp_import object (as this tracks ID changes), just before wp_import_cleanup() is called in the import_end() method.

At the risk of adding more hooks - another one that I think is needed is an action inside WXR_Parser_SimpleXML::parse to allow the imported file to be parsed for custom data, rather than having to use import_start, access the global $wp_import and re-parse the file.

Couldn't some of the above suggested filters be made redundant by passing $wp_import by reference to import_start?

Just to echo the original request — I've been actively using these filters for over six months now and they're indispensable. I look forward to seeing them in the actual plugin so I don't have to maintain the fork :)

Replying to danielbachhuber:

Just to echo the original request — I've been actively using these filters for over six months now and they're indispensable. I look forward to seeing them in the actual plugin so I don't have to maintain the fork :)

We've had the need for these recently too. Do you maintain the fork someplace public?

I just went ahead and committed to wpcom trunk :) Of all the ways we've hacked core, I think this is the least egregious

+1 to add the hooks as discussed in IRC: ​https://irclogs.wordpress.org/chanlog.php?channel=wordpress-dev&day=2013-01-28&sort=asc#m542294

Committed to the plugin in ​http://plugins.trac.wordpress.org/changeset/688911