Exporting in high data instances results in empty file download

[bor0]

When attempting to export products or documentation on a large WP instance with a lot of data, an empty .xml file is downloaded. This causes problems when we need to export updated content from production for use on our dev environments.

In our investigation, we found out that the query gets stopped because it runs for too long.

It comes from WP core: ​https://github.com/WordPress/WordPress/blob/2879b85136044a56fdd908a56c79c85a0ed7a39e/wp-admin/includes/export.php#L415. The query is also super long as it contains 23794 entries in the $post_ids array, though:

$ time vip wp -a woocommerce -e production -y -- db query "SELECT count(*) FROM wp_posts WHERE post_type = 'product'"
+----------+
| count(*) |
+----------+
|     2105 |
+----------+

It likely happens because it also includes post attachments here: ​https://github.com/WordPress/WordPress/blob/2879b85136044a56fdd908a56c79c85a0ed7a39e/wp-admin/includes/export.php#L190).

In that code block there's chunking, which is why it works, however, in the code where it fails there's no chunking.

[bor0]

@SergeyBiryukov another one when you get the chance!

I owe you a ton of beers or coffees (whichever you prefer). On some WordCamp... :)

[SirLouen]

@bor0 can you provide reproduction steps?

[bor0]

Yup, testing instructions are quite simple, though hard to reproduce on a blank install. Our WordPress instance (WooCommerce.com) has a ton of posts.

1. Navigate to wp-admin/export.php
2. Select All content
3. Click Export

After some time, observe logs:

08:47:06 PM	
PHP message: PHP Fatal error: Allowed memory size of 805306368 bytes exhausted (tried to allocate 33554440 bytes) in /var/www/wp-admin/includes/export.php on line 590

08:47:06 PM	
PHP message: Fatal error: Allowed memory size of 805306368 bytes exhausted (tried to allocate 33554440 bytes) in /var/www/wp-admin/includes/export.php on line 590 [woocommerce.com/wp-admin/export.php?download=true&content=all&cat=0&post_author=0&post_start_date=0&post_end_date=0&post_status=0&page_author=0&page_start_date=0&page_end_date=0&page_status=0&attachment_start_date=0&attachment_end_date=0&submit=Download+Export+File] []

[SirLouen]

Replying to bor0:

Yup, testing instructions are quite simple, though hard to reproduce on a blank install. Our WordPress instance (WooCommerce.com) has a ton of posts.

@bor0 can you define "a ton"?
Do you think that maybe 10,000 posts is sufficient to reproduce?

[bor0]

Let me know if this gives a general idea:

$ vip wp -a woocommerce -e production -y -- db query "SELECT count(*) FROM wp_posts WHERE post_status != 'auto-draft'"
count(*)
13382311
$ vip wp -a woocommerce -e production -y -- db query "SELECT count(*) AS cnt FROM wp_posts WHERE post_status != 'auto-draft' GROUP BY post_type ORDER BY cnt DESC LIMIT 10"
cnt
12463550
703227
90765
81500
17002
8845
4208
2326
2189
1560

[bor0]

Forgot to add, 81500 is the count of attachment post type.

[SirLouen]

That will be epic to test.

Why have you chosen 20 units for the chunks specifically?

[bor0]

As noted in the description, there is already chunking in other parts, e.g. ​here and ​here.

I chose 20 for consistency, but we could maybe use filters to allow for different sizes.

[SirLouen]

Test Report

Description

✅This report validates that the indicated patch works as expected.

Patch tested: https://core.trac.wordpress.org/attachment/ticket/63503/63503.patch

Environment

• WordPress: 6.9-alpha-60093-src
• PHP: 8.2.28
• Server: nginx/1.27.5
• Database: mysqli (Server: 8.4.5 / Client: mysqlnd 8.2.28)
• Browser: Chrome 137.0.0.0
• OS: Windows 10/11
• Theme: Twenty Twenty 2.9
• MU Plugins: None activated
• Plugins:
  • Test Reports 1.2.0
  • WP Dummy Content Generator 4.0.0

Patch Testing Instructions

1. Add some dummy posts
2. Export data
3. Save the xml file
4. Apply the patch
5. Export data again
6. Compare resulting XML file with the previous XML saved file

Expected Result

• Both XML files are identical (except for timestamps)

Actual Results

1. ✅ Issue resolved with patch.

Additional Notes

• ✅ Passing export group unit-tests (although they are a little Spartan, ngl)

• @bor0 suggestion: the patch feels pretty straightforward, but it is a good idea to add a GB PR with your patch, just to double-check that is passing all the CI tasks and ready to ship.

[bor0]

add a GB PR

I assume you mean opening a PR in ​https://github.com/WordPress/wordpress-develop instead?

[SirLouen]

Replying to bor0:

add a GB PR

I assume you mean opening a PR in ​https://github.com/WordPress/wordpress-develop instead?

Yes, that. The good thing is that they run the full CI protocol. Some committers (not all) will do this if you don't do it and then bring the patch from GB to commit it into SVN. It's like a step you save. Unless it's a patch that changes like 3 characters, it's a good practice nowadays.

Creating this PR just to test CI checks, as suggested per comment on the trac ticket.

[bor0]

Created the ​PR. This was actually a great idea - it caught some lint issues :) attaching new patch.