When visiting Wordpress sites with CPTs via Firefox the sites return rndmly: User-agent: * Disallow: /wp-admin/

[Jyria]

Hello!

I am experiencing a weird issue when using Firefox (Mac as well as Windows machines) and visiting a WordPress site that is built using custom post types.

The issue occurs randomly yet consistently and is usually solved when clearing firefox's browser cache - at least until for a few reloads:

1. Fire up firefox
2. Visit for example ​https://saskialund.de
3. Navigate to "Referenzen"
4. Hope to see the Referenzen page... if not it will show

User-agent: *
Disallow: /wp-admin/
instead

5. keep browsing the page. The above text will show eventually, once firefox starts to build the cache.

Questions:

1. Why is the robots.txt being displayed instead of the oage content?
2. Did I do something wrong when setting up the installation?

On all pages that have this issue the following plugins are installed:
Jetpack
Yoast Seo

I am experiencing this issue with all of my wordpress sites that have custom post types registered and in use.
Even in a page that is using a twentyfifteen child (see here: ​https://luciaschwalenberg.de ).

I am aware of this issue since WP 4.1 however I am not sure if it was there before 4.1 or not.

What could be the issue? Is this a bug?

Thanks for looking into this!

[swissspidy]

Hi Saskia

Thanks for your report. However, this place is for reporting bugs in WordPress core and not incompatibilities between two plugins. Therefore your question is better suited for the ​support forums. Consider opening a support thread in the respective plugin's support forums. As a little starting help, try disabling one plugin and Jetpack module after the other to find the culprit.

[atomicjack]

Hi,

It would be best to disable each plugin one-by-one and testing, to see which, if any, plugin is causing this.

We can't really look any further into it until it's ruled out that it is a plugin issue.

[swissspidy]

Also worth noting that the same error was already reported in #32518 and the support forums were already recommended.

[Jyria]

Thanks for your replies,
indeed I had already opened a ticket concerning this issue. I eventually gave up on finding a solution and thought it might get solved with some WP update.

Well.. here I am again.
Tried all the plugin deactivationing standard process to no avail.
The issue stayed.

I just posted in the Jetpack support forum, hoping to find a solution.
​https://wordpress.org/support/topic/when-visiting-my-site-via-firefox-the-sites-return-rndmly-user-agent-disallo?replies=1#post-7414728

[atomicjack]

I've read over both tickets, and it seems to be related to your particular installation, as I am unable to find reports of other users experiencing the same issue, and you mentioned it also takes place on single.php, so lots of people would have noticed it.

[Jyria]

Yes you are right @atomicjack.

Thats what's driving me nuts. I havent done any weird things to my installations. So the last possible culprit would be my vserver.

I am currently moving my sites to a freshly installed vserver with another hoster. I will report back, if the issue stays or is solved after this.

[netweb]

Trac is not for support, please use the forums, @Jyria has created a new topic on the forums, please continue discussion there rather than here :) ​https://wordpress.org/support/topic/2534411

[Jyria]

Update:
I found the culprit!

On my server was an nginx directive snippet in place which caused firefox to stumble over its feet while interpreting Cache-Control which was sent by my nginx.
The problem was wrong quote marks used for " wrapping "public" in the following code snippet: ( these were the wrong ones: ″ (forum interpreter shows the correct ones, but when viewing the post in edit mode I see the other quote marks ) )
`location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {

expires 30d;
add_header Pragma public;
add_header Cache-Control "public";
try_files $uri @fallback;

}`

Firefox showed broken Cache-Control definitions instead of these correct ones:
Cache-Control: "max-age=2592000, public"
this was shown
Cache-Control: "max-age=2592000, âpublicâ"

After correcting the nginx directive (exchanging the wrong quote marks for the correct ones), the error did not show up in ff anymore.
Chrome didn't have a problem with the other quote marks and therefore never showed that behaviour.

Also did I find that mod_pagespeed which was activated on that server was causing caching problems that I could not narrow down specifically though.

Funny thing is, I installed a site on a client's webhosting which is just a shared webhosting with german STRATO. Same caching problem occurs over there.
They might have this very nginx directive in place also with those wrong quotes that can't be interpreted correctly by firefox. Maybe even also installed mod_pagespeed which is not fully configured.

This can be considered as fixed.