noindex, wp_die and admin-ajax.php

[harryfear]

Hi folks,

I've found what appears to be a bug that affects fresh WordPress installs, so I am sure it's about the core and not themes or plugins.

If one browses directly to "admin-ajax.php", the URL dies with code 400 as a result of the:

wp_die( '0', 400 );

Further down in the file, there is the noindex header:

@header( 'X-Robots-Tag: noindex' );

My issue arose due to Google reporting a "robots blocked but indexed" issue ​https://webmasters.stackexchange.com/questions/123439/how-to-allow-only-admin-ajax-php-to-be-crawled.

But when it dies it seems not to send the noindex header, at least on my hosting environment. I don't have any other issues with this host. My host is a basic shared Cpanel host with a bunch of different WordPress sites/domains running on it. I think this is something that could be looked at from the WordPress side as a possible bug?

On some others' publicly-browsable sites, I can see that the noindex header *is* sent along with the 400 error status.

I don't myself have access to multiple different hosting environments to test this with, so I'm throwing it out here for advice/expertise/support/alarm-raising.

The noindex header instruction is set explicitly lower down in the admin-ajax.php file. If I copy that line up in way of a hot fix tweak on the code, the issue is resolved for me with Google:

// Require an action parameter
if ( empty( $_REQUEST['action'] ) ) {
	@header( 'X-Robots-Tag: noindex' ); // I moved this up and it solves the issue
	wp_die( '0', 400 );
}

I also contacted my host and we found that a .htaccess directive hot-fixes my issue on all my domains for now:

<FilesMatch "admin-ajax.php">
Header set X-Robots-Tag "noindex"
</FilesMatch>

[SergeyBiryukov]

Hi @harryfear, welcome to WordPress Trac! Thanks for the report.

Yes, apparently the header should be sent earlier, after send_origin_headers(), so that it applies before wp_die() as well.

For reference, the current line was introduced in [20288].

[harryfear]

Great, glad I'm not going mad. I'll watch this ticket.

[robi-bobi]

I've submitted a patch for this.

[SergeyBiryukov]

Thanks for the patch, @robi-bobi!

Looks like it only updates the package-lock.json file, but not wp-admin/admin-ajax.php. Could you update it with the correct changes?

[robi-bobi]

Thanks @SergeyBiryukov
Yeah, this is my first patch for this project, so a help from someone more familiar with WordPress processes would be great:

I used grunt upload_patch:47711, which added the patch, but in it I see:

• package-lock.json file changes, isn't this supposed to be ignored?
• my changes from wp-admin/admin-ajax.php are indeed not in there. I do see them if I use svn diff command (where I also see the package_json stuff, again, should I ignore this locally)

[garrett-eclipse]

Hi @robi-bobi

There's some great assistance here - ​https://make.wordpress.org/core/handbook/tutorials/working-with-patches/#creating-a-patch-with-the-command-line

I followed that when I started. And one of the things I learned when doing a patch from SVN is to either specify the file or the folder, in my case I just always run svn diff on the src folder;
svn diff src > #####.diff

You'll want to create the patch file before running the upload_patch command. And I assume it picked up that file as you weren't specific as to what folder (src) to diff.

In your case I suggest the following commands in this order;
svn diff src > 47711.2.diff
grunt upload_patch:47711
*The only time I personally don't focus my diff on src is when there's unit tests and then I include the tests folder like;
svn diff src tests > #####.diff

Hope that helps,
P.S. Another tip that took me a while to realize is if your patch introduces new files you first have to do an svn add before creating the patch file;
svn add src/path/to/file.txt

[robi-bobi]

Thanks @garrett-eclipse !
I did follow the tutorial you linked and did exactly as it instructs (did not target the src because of that).

I followed your suggestion, but it still uploaded the same file as before.

Finally I did svn diff while targeting src, and uploaded the diff manually. Hope that this works.

Please use only the 47711.3.diff when merging/reviewing.

[robi-bobi]

47711.3.diff

[garrett-eclipse]

Glad the manual approached worked for you @robi-bobi the patch 47711.3.diff looks good to me.

Weird that the grunt task isn't working for you though. Feel free to find me on ​Slack (@garrett-eclipse) and I can help you troubleshoot. Or come join the new contributor meeting on Slack in the #core channel on the 2nd and 4th Wednesdays at 19:00 UTC. Everyone is quite helpful there.

[SergeyBiryukov]

In 45650:

Administration: In admin-ajax.php, send X-Robots-Tag header earlier, so that it applies before wp_die() when no action parameter was provided.

Props robi-bobi, harryfear, garrett-eclipse.
Fixes #47711.

[robi-bobi]

Thank you @SergeyBiryukov and @garrett-eclipse for helping me complete my first patch.
Thanks @harryfear for describing the issue so well.