Site health: robots.txt and WordPress in a subdir

[hupe13]

See file wp-admin/includes/class-wp-debug-data.php line 571 and following

// Check if a robots.txt file exists.

This code prevents robots.txt from being determined correctly, if the installation is located in a subdirectory.
Site health reports: 'Your site is using the dynamic robots.txt file which is generated by WordPress.' This is not correct.

A robots.txt file should be located in $_SERVER["DOCUMENT_ROOT"] and, in this case, it cannot be handled by WordPress. get_home_path() and $_SERVER["DOCUMENT_ROOT"] return different values.

How about this code:

<?php
        $site       = wp_parse_url( get_home_url() );
        $serverroot = $site['host'];
        if ( isset( $site['path'] ) ) {
            // cannot handled by WP
        }

The user should provide a robots.txt in $_SERVER[“DOCUMENT_ROOT”] or relevant directives in $_SERVER[“DOCUMENT_ROOT”]/.htaccess.

[sainathpoojary]

Great catch, @hupe13! I looked into this and your diagnosis is completely right.

Since search engines only look for robots.txt at the main root of a domain, WordPress cannot serve one if it is installed in a subfolder. Right now, Site Health misses this check and gives users the wrong information.

Your idea to check the home URL path is the perfect way to solve this. If everyone agrees with this approach, I would be happy to put together a patch and add the tests for it. Let me know!

[khushipatel15]

Can you check this code snippet, in wp-admin/includes/class-wp-debug-data.php

Check if a robots.txt file exists.
$robots_txt_paths = array();

1. Path relative to ABSPATH (useful for single site installs directly in webroot).
$robots_txt_paths[] = ABSPATH . 'robots.txt';

2. Path relative to get_home_path() (most common for WordPress site root).
$robots_txt_paths[] = get_home_path() . 'robots.txt';

3. Path relative to $_SERVERDOCUMENT_ROOT (important for subdirectory installs where
the web server serves a robots.txt from a higher level than get_home_path()).
if ( isset( $_SERVERDOCUMENT_ROOT ) && ! empty( $_SERVERDOCUMENT_ROOT ) ) {

$document_root = wp_normalize_path( $_SERVERDOCUMENT_ROOT );
Add DOCUMENT_ROOT path only if it's distinct from get_home_path() to avoid redundant checks.
if ( wp_normalize_path( get_home_path() ) . '/' !== $document_root . '/' ) {

$robots_txt_paths[] = $document_root . '/robots.txt';

}

}

$has_robots_txt = false;
$robots_txt_file_path_to_read = ;
$robots_txt_source_description = ;

foreach ( $robots_txt_paths as $path ) {

if ( file_exists( $path ) ) {

$has_robots_txt = true;
$robots_txt_file_path_to_read = $path;

Provide a more specific description based on the detected location.
$normalized_path_dirname = wp_normalize_path( dirname( $path ) ) . '/';
$normalized_abspath = wp_normalize_path( ABSPATH );
$normalized_home_path = wp_normalize_path( get_home_path() );
$normalized_doc_root = isset( $_SERVERDOCUMENT_ROOT ) ? wp_normalize_path( $_SERVERDOCUMENT_ROOT ) . '/' : ;

if ( $normalized_abspath === $normalized_path_dirname ) {

$robots_txt_source_description = ( 'Your site is using a static robots.txt file located in the WordPress installation directory.', 'wordpress' );

} elseif ( $normalized_home_path === $normalized_path_dirname ) {

$robots_txt_source_description = ( 'Your site is using a static robots.txt file located at the WordPress site root.', 'wordpress' );

} elseif ( $normalized_doc_root === $normalized_path_dirname && $normalized_doc_root !== ) {

$robots_txt_source_description = ( 'Your site is using a static robots.txt file located at the web server\'s document root.', 'wordpress' );

} else {

$robots_txt_source_description = ( 'Your site is using a static robots.txt file.', 'wordpress' ); Fallback for unusual setups.

}
break; Found one, no need to check further.

}

}

if ( $has_robots_txt ) {

$has_robots_txt_content = file_get_contents( $robots_txt_file_path_to_read );
$robots_txt_details = array(

'label' => ( 'robots.txt file', 'wordpress' ),
'value' => $robots_txt_source_description,

);

} else {

$has_robots_txt = false;
$has_robots_txt_content = false;
$robots_txt_details = array(

'label' => ( 'robots.txt file', 'wordpress' ),
'value' => ( 'Your site is using the dynamic robots.txt file which is generated by WordPress.', 'wordpress' ),

);

}

[hupe13]

I did not patched the file, I made a test page in backend. I tested it with all my installations, but one of them has a very special configuration. Then this code does not detect right the configuration, but the user knows what they're doing.

<?php

echo '<h4>Robots Patch</h4>';

// Check if a robots.txt file exists.
$robots_txt_paths = array();

// 1. Path relative to ABSPATH (useful for single site installs directly in webroot).
$robots_txt_paths[] = ABSPATH . 'robots.txt';
//
// 2. Path relative to get_home_path() (most common for WordPress site root).
$robots_txt_paths[] = get_home_path() . 'robots.txt';
//
// 3. Path relative to $_SERVER['DOCUMENT_ROOT'] (important for subdirectory installs where
// the web server serves a robots.txt from a higher level than get_home_path()).
//
if ( isset( $_SERVER['DOCUMENT_ROOT'] ) && ! empty( $_SERVER['DOCUMENT_ROOT'] ) ) {
        $document_root = trailingslashit( wp_normalize_path( $_SERVER['DOCUMENT_ROOT'] ) );
        // Add DOCUMENT_ROOT path only if it's distinct from get_home_path() to avoid redundant checks.
        if ( trailingslashit( wp_normalize_path( get_home_path() ) ) !== $document_root ) {
                        $robots_txt_paths[] = $document_root . 'robots.txt';
        }
}

$path = end( $robots_txt_paths );

echo '<pre>';
var_dump( $path );
echo '</pre>';

$has_robots_txt                = false;
$robots_txt_file_path_to_read  = '';
$robots_txt_source_description = '';

if ( file_exists( $path ) ) {
        $has_robots_txt               = true;
        $robots_txt_file_path_to_read = $path;

        // Provide a more specific description based on the detected location.
        $normalized_path_dirname = trailingslashit( wp_normalize_path( dirname( $path ) ) );
        $normalized_abspath      = wp_normalize_path( ABSPATH );
        $normalized_home_path    = wp_normalize_path( get_home_path() );
        $normalized_doc_root     = $document_root;

        if ( $normalized_abspath === $normalized_path_dirname ) {
                $robots_txt_source_description = sprintf( __( 'Your site is using a static robots.txt file located in the WordPress installation directory.', 'WordPress' ) );
        } elseif ( $normalized_home_path === $normalized_path_dirname ) {
                        $robots_txt_source_description = sprintf( __( 'Your site is using a static robots.txt file located at the WordPress site root.', 'WordPress' ) );
        } elseif ( $normalized_doc_root === $normalized_path_dirname && $normalized_doc_root !== '' ) {
                        $robots_txt_source_description = sprintf( __( 'Your site is using a static robots.txt file located at the web server\'s document root.', 'WordPress' ) );
        } else {
                        $robots_txt_source_description = sprintf( __( 'Your site is using a static robots.txt file.', 'WordPress' ) ); // Fallback for unusual setups.
        }
}
if ( $has_robots_txt ) {
        $has_robots_txt_content = file_get_contents( $robots_txt_file_path_to_read );
        $robots_txt_details     = array(
                'label' => sprintf( __( 'robots.txt file', 'wordpress' ) ),
                'value' => $robots_txt_source_description,
        );
} elseif ( trailingslashit( wp_normalize_path( get_home_path() ) ) !== $document_root ) {
        $has_robots_txt                = false;
        $has_robots_txt_content        = false;
        $robots_txt_source_description = sprintf( __( 'Your site does not have a robots.txt file in web server\'s document root.', 'WordPress' ) );
        $site                          = wp_parse_url( get_home_url() );
        $path                          = isset( $site['path'] ) ? $site['path'] : '';

        $robots_txt_source_description .= ' ' . sprintf( __( 'Thats why you should write in document roots .htaccess', 'WordPress' ) );
        $robots_txt_source_description .= '<pre>RewriteRule ^robots.txt$ ' . $path . '/index.php?robots=1 [L]</pre>';
        $robots_txt_details             = array(
                'label' => sprintf( __( 'robots.txt file', 'wordpress' ) ),
                'value' => $robots_txt_source_description,
        );
} else {
        $has_robots_txt         = false;
        $has_robots_txt_content = false;
        $robots_txt_details     = array(
                'label' => sprintf( __( 'robots.txt file', 'wordpress' ) ),
                'value' => sprintf( __( 'Your site is using the dynamic robots.txt file which is generated by WordPress.', 'wordpress' ) ),
        );
}

echo '<pre>';
var_dump( $has_robots_txt_content, $robots_txt_details );
echo '</pre>';

echo $robots_txt_details['value'];

[hupe13]

I tested it with all my installations,

I have single site installions with and without subdir.
In my multisite installation I have 2 domains and sites with and without subdir.
I don't have a multisite installation with subdomains.
I don't have an installation where siteurl and home are different.

[hupe13]

Check if a robots.txt file exists.
$robots_txt_paths = array();

1. Path relative to ABSPATH (useful for single site installs directly in webroot).
$robots_txt_paths[] = ABSPATH . 'robots.txt';

2. Path relative to get_home_path() (most common for WordPress site root).
$robots_txt_paths[] = get_home_path() . 'robots.txt';

This is not needed. You should check, if a robots.txt exists in $_SERVER['DOCUMENT_ROOT'] and then compare this with other paths.

This PR improves Site Health's robots.txt detection for installations where WordPress is installed in a subdirectory or where the web server document root differs from the WordPress home path.

Previously, Site Health checked get_home_path() . 'robots.txt', which could incorrectly report that WordPress was dynamically generating robots.txt when a static file existed at the web server's document root.

The robots.txt detection is moved into a dedicated helper and now considers the server document root when determining whether a static file exists at the root of the site's domain.

When DOCUMENT_ROOT and get_home_path() differ and the public robots.txt location cannot be determined reliably, Site Health reports that it is unable to determine how the file is served instead of incorrectly reporting a dynamic robots.txt.

The existing messages for static robots.txt, dynamic robots.txt, and unavailable rewrite support remain unchanged.

Testing

Added regression tests for robots.txt reporting across root and subdirectory URLs, file presence, permalink settings, simulated rewrite support, and differing filesystem roots. Tests run in both single-site and multisite modes, including shared physical-file detection for a child site.

Validation: 84 single-site and 85 multisite Site Health tests passed, along with PHP syntax, coding standards, and whitespace checks.

Public /robots.txt responses and custom web-server mappings were not verified. No line or branch coverage percentage was measured.

Trac ticket: https://core.trac.wordpress.org/ticket/66060

Use of AI Tools

AI assistance: Yes
Tool(s): ChatGPT
Model(s): GPT-5.6 Sol
Used for: Reviewing the proposed approach, discussing edge cases and test scenarios. The implementation and tests were reviewed and validated by me.