WordPress.org

Make WordPress Core

Opened 9 months ago

Closed 9 months ago

Last modified 9 months ago

#53016 closed defect (bug) (invalid)

Robots API

Reported by: kraevit Owned by:
Milestone: Priority: normal
Severity: normal Version: 5.7
Component: Sitemaps Keywords: has-screenshots reporter-feedback
Focuses: Cc:

Description (last modified by dd32)

Hello,
My name is Krasimir Kraev, I have frustrating issue with my wordpress installation version 5.7 using yoast seo plugin. My google search console reported that i have 1600 invalid url's due to having "noindex" in the <meta robots tag> of my pages/posts. Everything in the admin panel is set correctly(eg settings reading). I found that in this particular file of wordpress core files - /wp-includes/robots-template.php in the lines from 153-165 there is a function:

function wp_robots_no_robots( array $robots ) {

    // $robots['noindex'] = true;  THIS DEFAULT LINE COUSING ISSUES INDEXING MY PAGES ON GOOGLE SEARCH CONSOLE so i changed to false
    $robots['noindex'] = false;

	if ( get_option( 'blog_public' ) ) {
		$robots['follow'] = true;
	} else {
		$robots['nofollow'] = true;
	}

	return $robots;
}

Is this a bug or it is a normal wordpress 5.7 behavior?

Please respond.

website: https://webnavigator.bg
template: Listify 2.13.7
yoast seo 16.1.1

Regards!

Attachments (1)

google-search-console-invalid-urls.jpg (132.3 KB) - added by kraevit 9 months ago.
this tag has been automatically added to my header <meta name="robots" content="index, noindex, follow> which is what causing the issue.

Download all attachments as: .zip

Change History (8)

@kraevit
9 months ago

this tag has been automatically added to my header <meta name="robots" content="index, noindex, follow> which is what causing the issue.

#1 @dd32
9 months ago

  • Description modified (diff)
  • Keywords reporter-feedback added; needs-testing removed
  • Milestone Awaiting Review deleted
  • Resolution set to invalid
  • Status changed from new to closed

Hi @kraevit,

The function in question should only be called when the blog visibility option is enabled, or another plugin is causing it to be called.

The functionality is controlled through the following setting under Settings -> Reading:

Search engine visibility	Discourage search engines from indexing this site

If that option is disabled, you should check where the function is being called from (It's new in 5.7), as I suspect it'll be being called from a plugin. The only place that calls it in WordPress that would affect most urls is wp_robots_noindex().

Without knowing which URLs are affected or why it's being called, it's not possible to determine if this is a WordPress issue or not, but it doesn't appear that it's likely to be a core bug due to the newness of the functionality and that more people aren't affected by it.

Unfortunately I'm closing this as invalid as there's no information which can be used to replicate it or point to the cause being WordPress itself.

#2 @kraevit
9 months ago

The entire website is affected all the links from pages to posts etc. Discourage is not enabled also. I commented the line as i said before and the site is back to normal. So it is definetely a bug.

#3 @kraevit
9 months ago

  • Resolution invalid deleted
  • Status changed from closed to reopened

The entire website is affected all the links from pages to posts etc. Discourage is not enabled also. I commented the line as i said before and the site is back to normal. So it is definetely a bug.

#4 @kraevit
9 months ago

I checked the following: .htaccess file is ok, robots.txt is ok, all plugins are disabled. Caches of the website and browser are cleared, Discourage in settings ITS NOT enabled. The problem persist, and i have noindex in meta tag in the header on the entire website. When i comment the line in the wordpress core file all seems to be ok and the noindex disappears. So....

#5 follow-up: @adamsilverstein
9 months ago

@kraevit it is possible your theme is causing the issue, or perhaps some other code running in your environment (maybe an mu plugin)?

Can you try switching temporarily to a core default theme, disabling all plugins and see if the issue persists? If possible can you enable logging and see if there are any errors in your logs?

As Dion points out above, this is very unlikely to be a widespread core bug or we would have had many reports of it already - it is somehow triggered by your setup.

#6 in reply to: ↑ 5 @kraevit
9 months ago

  • Resolution set to invalid
  • Status changed from reopened to closed

Replying to adamsilverstein:

@kraevit it is possible your theme is causing the issue, or perhaps some other code running in your environment (maybe an mu plugin)?

Can you try switching temporarily to a core default theme, disabling all plugins and see if the issue persists? If possible can you enable logging and see if there are any errors in your logs?

As Dion points out above, this is very unlikely to be a widespread core bug or we would have had many reports of it already - it is somehow triggered by your setup.

#7 @kraevit
9 months ago

I am really sorry for disturbing you. The problem was actually caused by my paid wp theme - listify. I didn't expect that latest theme update will cause this issue. Also I was frustrated since my client was angry because the bug affected his entire business. Now it's solved and all works fine. Sorry again and thanks for the provided information. My bad!

Note: See TracTickets for help on using tickets.