WordPress.org

Make WordPress Core


Ignore:
Timestamp:
09/02/2019 02:26:55 AM (13 months ago)
Author:
peterwilsoncc
Message:

#43590: Use robots meta tag to better discourage search engines.

This changes the "discourage search engines" option to output a noindex, nofollow robots meta tag. Disallow: / is removed from the robots.txt to allow search engines to discover they are requested not to index the site.

Disallowing search engines from accessing a site in the robots.txt file can result in search engines listing a site with a fragment (a listing without content).

Props donmhico, jonoaldersonwp.
Fixes #43590.

File:
1 edited

Legend:

Unmodified
Added
Removed
  • trunk/src/wp-includes/functions.php

    r45926 r45928  
    15831583 *
    15841584 * @since 2.1.0
     1585 * @since 5.3.0 Remove the "Disallow: /" output if search engine visiblity is
     1586 *              discouraged in favor of robots meta HTML tag in wp_no_robots().
    15851587 */
    15861588function do_robots() {
     
    15961598    $output = "User-agent: *\n";
    15971599    $public = get_option( 'blog_public' );
    1598     if ( '0' == $public ) {
    1599         $output .= "Disallow: /\n";
    1600     } else {
    1601         $site_url = parse_url( site_url() );
    1602         $path     = ( ! empty( $site_url['path'] ) ) ? $site_url['path'] : '';
    1603         $output  .= "Disallow: $path/wp-admin/\n";
    1604         $output  .= "Allow: $path/wp-admin/admin-ajax.php\n";
    1605     }
     1600
     1601    $site_url = parse_url( site_url() );
     1602    $path     = ( ! empty( $site_url['path'] ) ) ? $site_url['path'] : '';
     1603    $output  .= "Disallow: $path/wp-admin/\n";
     1604    $output  .= "Allow: $path/wp-admin/admin-ajax.php\n";
    16061605
    16071606    /**
Note: See TracChangeset for help on using the changeset viewer.