WordPress.org

Make WordPress Core

Opened 3 years ago

Closed 18 months ago

Last modified 2 weeks ago

#16416 closed defect (bug) (fixed)

Settings -> Privacy "block search engines" option is not accurate

Reported by: designsimply Owned by: westi
Milestone: 3.5 Priority: normal
Severity: normal Version:
Component: Administration Keywords: has-patch ux-feedback dev-feedback
Focuses: Cc:

Description

There is an option on the Settings -> Privacy page says it will "block search engines", but that does not always happen because checking the option does not block posts already established in the search engine. Proposing updating the language so it's more accurate.

Current:

"I would like to block search engines, but allow normal visitors"

Proposed:

"I would like to ask search engines not to index this site, but allow normal visitors"

Attachments (15)

options-privacy.diff (959 bytes) - added by designsimply 3 years ago.
options-privacy.2.diff (1.3 KB) - added by designsimply 3 years ago.
RightNowBlocked.patch (863 bytes) - added by dougwrites 3 years ago.
s/Search Engines Blocked/Search Engines Discouraged
16416.4.diff (4.1 KB) - added by mbijon 3 years ago.
Renames: Privacy -> Search Indexing. Updated Help tab
16416.5.diff (7.1 KB) - added by mbijon 3 years ago.
Moves private/public setting to Reading Options
16416.RightNowBlocked.2.patch (884 bytes) - added by SergeyBiryukov 2 years ago.
16416.general.diff (10.5 KB) - added by nacin 20 months ago.
16416.reading.diff (14.6 KB) - added by nacin 20 months ago.
16416.general.2.diff (12.7 KB) - added by mbijon 20 months ago.
Like this
16416.diff (2.6 KB) - added by nacin 19 months ago.
16416.2.diff (2.6 KB) - added by lessbloat 19 months ago.
16416.3.diff (2.6 KB) - added by nacin 19 months ago.
16416.4-help.diff (1.7 KB) - added by DrewAPicture 19 months ago.
Help text fixes
16416.5.2.diff (1.6 KB) - added by DrewAPicture 18 months ago.
install / signup
16416.6.patch (4.4 KB) - added by ocean90 18 months ago.
Removes search engine stuff from signup/install

Download all attachments as: .zip

Change History (76)

comment:1 lloydbudd3 years ago

I'm not keen on the use of the term "ask", nor the term "normal". What about some thing like:

"I don't want search engines to index this site. This does not block visitors, and unfortunately, this may not affect what content search engines have already indexed."

comment:2 scribu3 years ago

checking the option does not block posts already established in the search engine.

I'm not sure what you mean. If a robot is blocked from re-visiting a URL, the search engine will eventually remove that URL from it's index.

Therefore, I think the text is accurate.

comment:3 scribu3 years ago

It says "block robots", not "remove this site from search engines".

comment:4 scribu3 years ago

Ah, right, it says "search engines" instead of "robots".

comment:5 follow-up: westi3 years ago

  • Component changed from General to Administration
  • Keywords 3.2-early added
  • Milestone changed from Awaiting Review to Future Release
  • Owner set to westi
  • Status changed from new to accepted

Ask and normal sound fine to me.

robots.txt is only a request and only obeyed by good citizen search engines - in some cases bad citizens of the web will use them to find hidden content.

Normal was already in the string for a long time and sounds fine to me.

comment:6 in reply to: ↑ 5 mikeschinkel3 years ago

Replying to westi:

Ask and normal sound fine to me.

robots.txt is only a request and only obeyed by good citizen search engines - in some cases bad citizens of the web will use them to find hidden content.

Normal was already in the string for a long time and sounds fine to me.

FWIW, I've recently had several test domains be indexed by Google after implementing analytics on them, even though I had robots.txt in full disallow mode. I had to request removal to get the pages out of the index, and I had to write a plugin that returned 403 for any page Googlebot visited on those sites to keep it out of the index. YMMV.

comment:7 RanYanivHartstein3 years ago

Issues with robots.txt are not limited to "misbehaving" search engines - pages from sites blocked with robots.txt can show up in Google or any other search engine.

I like "ask" since it's technically accurate while still being human, but I don't think "normal" is necessary since search engines should not really be thought of as visitors anyway. Instead, how about:

I would like to ask search engines not to index this site, but allow direct visitors

Also, hope this would not be considered hijacking the thread, but I think the "I would like" part is unnecessary and can be removed - those three words are wasting too much valuable user concentration that can be used on figuring out this not very intuitive concept.

Last edited 3 years ago by RanYanivHartstein (previous) (diff)

comment:8 scribu3 years ago

+1 on dropping "I would like". All other setting texts are direct.

comment:9 markjaquith3 years ago

There are a lot of text issues on this screen. "Site Visibility" isn't correct. It's visible either way. It's about what you're asking robots (Search Engines is probably a fine colloquial term for robots) to do.

"I would like..." "I would like..." lose those.

Maybe frame the whole thing in terms of search engines.


Search Engine Visibility:

  • Allow search engines to index this site.
  • Ask search engines to not index this site.

Note: neither of these options physically blocks access to your site — it is up to search engines to honor your request.


comment:10 designsimply3 years ago

Changed "physically blocks access" to "blocks 100% access". Updated the patch.

comment:11 follow-up: markjaquith3 years ago

Nevermind "100%" access. It doesn't block "1%" access. Maybe just "Neither of these options forcibly blocks access"

comment:12 nacin3 years ago

In [18554]:

First pass on clarifying Settings > Privacy. props designsimply, markjaquith. see #16416.

comment:13 dougwrites3 years ago

  • Cc heymrpro@… added

I don't think [18554] creates a need to change anything in the help tab for this screen, but someone might want to double-check that. The Right Now box in the Dashboard still says "Search Engines Blocked"; about to upload a patch for that.

dougwrites3 years ago

s/Search Engines Blocked/Search Engines Discouraged

comment:14 in reply to: ↑ 11 lloydbudd3 years ago

I like where this has gone!

Replying to markjaquith:

Nevermind "100%" access. It doesn't block "1%" access. Maybe just "Neither of these options forcibly blocks access"

What is the thought behind using the term "neither"? It makes me think that *both* are requests not to index.

What is lost if the options were more vertically separated and the note reads something like:

Note: This option doesn't block access to your site — it is up to search engines to honor your request.

comment:15 RanYanivHartstein3 years ago

+1 to Mark Jaquith's suggestion to rebuild the whole thing around search engines, and remove "normal visitors" out of the equation.

However, once you go down that road, I think it makes the most sense to just remove this whole page and replace it with a single toggle box someplace: "Prevent search engines from indexing my site".

comment:16 scribu3 years ago

+1 on tranforming it into a "Privacy" section with a single checkbox on the General Settings page.

comment:17 scribu3 years ago

  • Keywords needs-patch added; has-patch ux-feedback 3.2-early removed

comment:18 DrewAPicture3 years ago

+1 on tranforming it into a "Privacy" section with a single checkbox on the General Settings page.

Might it not be more aptly included in the Reading settings? I can give a patch for this a go, but I think it might be more suited for inclusion in the Reading settings.

comment:19 scribu3 years ago

Yeah, I guess putting it under Reading makes more sense.

comment:20 dd323 years ago

See also: #18841 - Privacy settings are useless when Default permalinks are enabled

mbijon3 years ago

Renames: Privacy -> Search Indexing. Updated Help tab

comment:21 mbijon3 years ago

  • Cc mike@… added

I started working on #18841 and couldn't help but take a shot at renaming 'Privacy' to 'Search Indexing.'

Language changes are always hard:

  • I choose this to try to reflect what is happening while leaving the door open for search-related items like sitemaps or SEO options (knowing those are usually handled by plugins)

Note, this patch also updates the Help menu to the new format from #18690. With those upgraded Help menus, moving this single setting to the Reading settings seems reasonable. The Reading settings could have the current Privacy help-info added as an option-specific tab.

mbijon3 years ago

Moves private/public setting to Reading Options

comment:22 mbijon3 years ago

That latest .5 diff moves the setting from the Privacy Options page to the Reading Options page.

I think I prefer keeping it on its own page. That's so SEO & Sitemap plugins might use that existing page with add_settings_section()/add_settings_field() instead of always building their own settings pages.

comment:23 dougwrites3 years ago

Hope it's OK to put in a reminder that the related UI text on the Dashboard screen, which links to this, needs re-wording too.

comment:24 mbijon3 years ago

Good reminder. One of my own: My patches don't affect the main navigation in WP-Admin, just starting points right now.

comment:25 follow-up: dougwrites2 years ago

Is this ticket still "milestone: future release" given that [18554] is in the beta? I believe there's still an incompatibility bug with the related phrasing on the Dashboard screen.

comment:26 in reply to: ↑ 25 SergeyBiryukov2 years ago

Replying to dougwrites:

Is this ticket still "milestone: future release" given that [18554] is in the beta?

Moving Privacy settings to General/Reading settings is definitely a Future Release material, but RightNowBlocked.patch would probably make sense for 3.3 in terms of consistency. 16416.RightNowBlocked.2.patch is a slight variation of the text.

comment:28 mbijon2 years ago

Related: #18605

nacin20 months ago

comment:29 nacin20 months ago

  • Milestone changed from Future Release to 3.5

I'd like to eliminate options-privacy.php from 3.5. I was originally moving it to options-general.php. I understand why options-reading.php is enticing. Given core's usage, reading makes sense. If however WP.com were to dissolve options-privacy, they'd probably move it to general. That is because WP.com (like some plugins) add an actual "private site" option here.

Technically, a site being private only applies to the frontend, though, so reading is still not incorrect. I'm also not too concerned about redirecting plugins' privacy sections and fields to reading.

16416.general.diff moves it to general and adds a "Date and Time" group to split things up over there. Moving it to options-reading will be a bit cleaner, but does require more than 16416.5.diff.

nacin20 months ago

comment:30 follow-up: nacin20 months ago

  • Keywords has-patch ux-feedback added; needs-patch removed

16416.reading.diff shifts blog_public to options-reading instead. It is also a bit more inclusive, catching options-privacy.php links on the dashboard, options-writing, and the welcome panel. (It was removed from the welcome panel because they just got through installing WP, where they were asked, and I don't think "search engine visibility" is that important otherwise, such as for multisite welcome's.)

It also toys with an idea for how to present blog_public. We can call it 'Search Engine Visibility' unless someone is using the blog_privacy_selector action, in which case it can revert to 'Site Visibility' automatically.

Additionally, when the action isn't used, we can get away with a checkbox only. While checkboxes are generally cleaner, the radio buttons do give you more allowances with the text; specifically, "Ask search engines not to...". It's easier to describe the negative "ask not" rather than the positive "allow", and it's more accurate, which is not something you can do with a checkbox.

I like how the label changes based on the action usage; not sold on the checkbox.

comment:31 in reply to: ↑ 30 lessbloat20 months ago

Replying to nacin:

16416.reading.diff shifts blog_public to options-reading

I'd prefer this over settings->general. Reading is the section I'd look to first (if I didn't know where to look). Plus, I'm not a huge fan of the "general" section, as it's not super transparent what I'll find there without clicking the link.

comment:32 follow-up: johnbillion20 months ago

Note that there's some text and link under Update Services on the Settings -> Writing screen which relates to the same setting and links to the Privacy Settings screen.

comment:33 in reply to: ↑ 32 nacin20 months ago

Replying to johnbillion:

Note that there's some text and link under Update Services on the Settings -> Writing screen which relates to the same setting and links to the Privacy Settings screen.

Yep, that's in 16416.reading.diff.

comment:34 mbijon20 months ago

+1 for 16416.general.diff and the label changes. Also undecided on the checkbox

In .5 I put these into Reading because it seemed easily justified. But I consider General a better place:
Search indexing is a core consideration when you're running any site, so these should be important enough to go front & center in Options.

(Have to admit that using WP.com as an argument for putting these into General almost made me vote for Reading. Petty jealousy though, I wish .org had more in common with .com at times)


What do you think about adding a hook to stop output of these 'Site Visibility' settings?

If a plugin like 'Registered Users Only' or WP.com creates a level of site privacy that overrides these settings, then hiding them from the UI would simplify Options & reduces the chance of users' confusion.

mbijon20 months ago

Like this

comment:36 nacin19 months ago

In [21838]:

Fold Privacy Settings into Reading Settings, moving blog_public (search engine/robots) to options-reading and removing options-privacy.

When blog_public only has two values (as judged by whether the blog_privacy_selector action is used), convert from radio buttons to a checkbox, and rename from 'Site Visibility' to a more specific 'Search Engine Visibility'.

The text and implementation may change a bit. see #16416.

comment:37 nacin19 months ago

I wanted to get this in. What still needs review after [21838]

  • The text/language in general.
  • Whether we can invert the checkbox, as the options for language is better when inverted. I think we can get away with this if we set value="0" on the box and then in sanitize_option() convert === null to 1. (As options.php uses 'null' when nothing is posted... The fix in [21292] makes this actually distinguishable now.)
  • Whether we should register this whole option using add_settings_field(), to hypothetically allow it to be unhooked. Note that there's no actual API for removing fields and sections, but I'd consider that to be good enough for now. Note also #21507.

nacin19 months ago

comment:38 nacin19 months ago

16416.diff inverts the checkbox, with the label "Ask search engines to <strong>not index</strong> this site".

options.php sets $value to be null by default. It then pulls the value from $_POST (which can only be a string or array), trims that value if it is not an array, and calls stripslashes_deep() on it (which only affects strings, and strings inside arrays).

The problem is that the trim occurs even if the value was not pulled from POST. That means null is converted to a string, and my trick doesn't work. A slight tweak means null can be passed to update_option(), which converts it to an empty string anyway, so there's no change in functionality — but, it provides some extra context in sanitize_option().

How it looks: http://cl.ly/image/2C0p3M1y2f41

comment:39 mbijon19 months ago

+1 registering this with add_settings_field()
(but I still prefer this in Options > General)

Only slightly off topic:
Seems like an API for hiding sections and/or fields is a good idea. One of the top requests from site owners is to hide individual settings or to re-mix the settings pages. An API for that would enable this instead of hiding whole admin pages based on user levels.

lessbloat19 months ago

comment:40 lessbloat19 months ago

in 16416.2.diff I made 2 minor copy tweaks:

http://f.cl.ly/items/3b3C3Y0g2H2j0T3i1c3v/privacy.jpg

Version 0, edited 19 months ago by lessbloat (next)

comment:41 nacin19 months ago

In [21849]:

If an option was not sent to options.php, pass null to update_option(), rather than trimming the null and converting it to an empty string. This provides better context for sanitize_option() while still storing what ends up being an empty string either way. see #16416.

nacin19 months ago

comment:42 nacin19 months ago

  • Resolution set to fixed
  • Status changed from accepted to closed

In [21851]:

"[ ] Discourage search engines from indexing this site". fixes #16416.

comment:43 nacin19 months ago

  • Resolution fixed deleted
  • Status changed from closed to reopened

Re-opening to discuss hook-ability.

comment:45 in reply to: ↑ 44 ocean9019 months ago

Replying to SergeyBiryukov:

Do we want to apply [21851] to wp-admin/install.php and wp-signup.php as well?
http://core.trac.wordpress.org/browser/tags/3.4.2/wp-admin/install.php#L132
http://core.trac.wordpress.org/browser/tags/3.4.2/wp-signup.php#L95

I think we should remove the "privacy" phrases from these pages to speed up the install/sign-up process. A new user shouldn't faced with such a special option.

comment:46 DrewAPicture19 months ago

#22007 was marked as a duplicate.

DrewAPicture19 months ago

Help text fixes

comment:47 DrewAPicture19 months ago

  • Cc xoodrew@… added

16416.4-help.diff Updates the help text from 'radio button' to 'checkbox' and the new option label from r21851

comment:48 RanYanivHartstein18 months ago

This ticket was reopened to discuss hook-ability, I hope it's OK to still comment on copy.

"Discourage search engines from indexing this site"

  • The term "discourage" doesn't seem right for a setting. It's ambiguous about what the setting actually does, whereas something like "ask" that was suggested before is a more accurate description. Also, the note "it is up to search engines to honor your request" doesn't make as much sense in the context of "discourage".
  • "indexing this site" requires you to understand how search engines work. Something like "ask search engines to exclude your site from search results" might be less accurate but more helpful to users.

It might even be worth considering something like "Ask Google, Bing and Yahoo to exclude your site from search results".

comment:49 nacin18 months ago

This was discussed nearly ad nauseam during an IRC meeting: https://irclogs.wordpress.org/chanlog.php?channel=wordpress-dev&day=2012-08-29&sort=asc#m447025.

That said, I could re-consider based on "Ask search engines to exclude your site from search results".

comment:50 DrewAPicture18 months ago

  • Keywords dev-feedback added

From what I see, this is what's left:

  • 16416.4-help.diff Updates the help text to match [21851], but could be adjusted if the language changes again.
  • Decide whether to remove the checkbox from wp-signup and/or install
  • Decide on hook-ability

comment:51 ryan18 months ago

In 22388:

Update contextual help for 'Search Engine Visibility' to reflect UI changes. Props DrewAPicture. see #16416

comment:52 ryan18 months ago

We might as well propagate the updated text to signup and install, although I'd be fine with punting that.

DrewAPicture18 months ago

install / signup

comment:53 DrewAPicture18 months ago

16416.5.diff changes the install/signup language. It's flipped to the positive because the box is checked by default.

ocean9018 months ago

Removes search engine stuff from signup/install

comment:54 follow-up: ocean9018 months ago

As mentioned in comment:45, 16416.6.patch removes the privacy search engines option from the signup/install page.

comment:55 in reply to: ↑ 54 DrewAPicture18 months ago

Replying to ocean90:

As mentioned in comment:45, 16416.6.patch removes the privacy search engines option from the signup/install page.

Good call. If we remove it, the option still defaults to 'on', correct?

comment:56 nacin18 months ago

I'm okay with removing this on install. I'm not sure I'd like to mess with wp-signup.php at all - we know basically nothing about how this page is used in the wild.

I'm also quite okay with closing this as fixed.

comment:57 nacin18 months ago

In 22473:

Add a class. see #16416.

comment:58 nacin18 months ago

  • Resolution set to fixed
  • Status changed from reopened to closed

Let's close this for 3.5. We can revisit install.php and wp-signup.php in 3.6, as well as any pluggable or language changes. Nice work, everyone.

comment:59 SergeyBiryukov17 months ago

#17845 was marked as a duplicate.

comment:60 nacin17 months ago

In 23050:

Remove options-privacy.php, as intended by [21838]. see #16416. see #22349.

comment:61 in reply to: ↑ 44 SergeyBiryukov2 weeks ago

Replying to SergeyBiryukov:

Do we want to apply [21851] to wp-admin/install.php and wp-signup.php as well?

Follow-up: #27628

Note: See TracTickets for help on using tickets.