Opened 8 years ago
Closed 8 years ago
#36390 closed defect (bug) (fixed)
Disallow /wp-json/ crawling
Reported by: |
|
Owned by: |
|
---|---|---|---|
Milestone: | 4.6 | Priority: | normal |
Severity: | normal | Version: | 4.4 |
Component: | REST API | Keywords: | has-patch commit |
Focuses: | Cc: |
Description (last modified by )
Since 4.4, I'm seeing support topics about /wp-json/
being crawled by search engines, apparently leading to penalties for duplicate content in some cases, caused by wp_oembed_add_discovery_links()
, or higher server load in other.
Some blog posts suggest disallowing /wp-json/
in robots.txt
or even disabling REST API altogether.
Should we add Disallow: /wp-json/
to do_robots()
?
Attachments (2)
Change History (12)
This ticket was mentioned in Slack in #core-restapi by joehoyle. View the logs.
8 years ago
#7
@
8 years ago
- Keywords has-patch added; needs-patch removed
- Milestone changed from Awaiting Review to 4.6
#8
@
8 years ago
- Keywords commit added
Thanks for the patch @m_uysl, I just moved the line up a bit.
@rmccue you okay with 36390.2.diff?
Note: See
TracTickets for help on using
tickets.
Google, etc, need to crawl it for sites loading content via the API using JavaScript. This prevents it from being included in robots.txt.
Perhaps an
X-Robots-Tag: noindex
header on the API end-points to prevent the duplicate content penalties.