Make WordPress Core

Opened 17 years ago

Closed 15 years ago

Last modified 3 years ago

#5007 closed defect (bug) (fixed)

Email notifications fail on hosted sites that check sender address

Reported by: jlwarlow's profile jlwarlow Owned by: pishmishy's profile pishmishy
Milestone: 2.9 Priority: normal
Severity: normal Version: 2.8.4
Component: Mail Keywords: needs-patch
Focuses: Cc:

Description

I had wordpress 2.2.2 hosted on FastHosts and wasn't getting any email notifications, everything was landing in dead.letter. After some investigation I found it was because the phpmailer->Sender wasn't being set. Fasthosts check the sender email address to make sure it's a valid email address for an account in your domain as part of their spam filtering rules.

If you add the line

$phpmailer->Sender = "wordpress@" . preg_replace('#^www\.#', '', strtolower($_SERVER['SERVER_NAME']));

to pluggable.php on wp-includes before the line

$phpmailer->FromName = "WordPress";

I found this fixed the issue.

Attachments (4)

patch.diff (489 bytes) - added by mattyrob 17 years ago.
Patch
patch.2.diff (489 bytes) - added by mattyrob 17 years ago.
5007.patch (975 bytes) - added by pishmishy 16 years ago.
Standardizes sender address to admin_email
5007.2.patch (1.6 KB) - added by pishmishy 16 years ago.
Document the problems in the code

Download all attachments as: .zip

Change History (50)

#1 @Viper007Bond
17 years ago

  • Keywords has-patch 2nd-opinion added; phpmailer removed
  • Milestone changed from 2.2.3 to 2.3
  • Version changed from 2.2.3 to 2.2.2

@mattyrob
17 years ago

Patch

@mattyrob
17 years ago

#2 @mattyrob
17 years ago

Working from revision 6159 it still looks like $phpmailer->Sender is not being used. I've attached (hopefully) a possible patch that makes use of the existing filters and variables instead of using a preg_replace as suggested above.

#3 @mattyrob
17 years ago

  • Milestone changed from 2.4 to 2.3.1

Can't we get this in for 2.3.1?

#4 follow-up: @ryan
17 years ago

I'm not sure how that would help unless you happen to have a wordpress@ address for your domain. Regardless, the patch makes Sender and From consistent, which seems a good thing to do. I'll +1.

#5 @westi
17 years ago

  • Resolution set to fixed
  • Status changed from new to closed

(In [6265]) Set the Sender on emails as well as from. Fixes #5007 for trunk props mattyrob

#6 @westi
17 years ago

  • Resolution fixed deleted
  • Status changed from closed to reopened

#7 @westi
17 years ago

  • Resolution set to fixed
  • Status changed from reopened to closed

(In [6266]) Set the Sender on emails as well as from. Fixes #5007 for 2.3.1 props mattyrob

#8 in reply to: ↑ 4 ; follow-up: @foolswisdom
17 years ago

  • Keywords regression added; has-patch 2nd-opinion removed
  • Milestone changed from 2.3.1 to 2.3.2
  • Resolution fixed deleted
  • Status changed from closed to reopened

Replying to ryan:

I'm not sure how that would help unless you happen to have a wordpress@ address for your domain. Regardless, the patch makes Sender and From consistent, which seems a good thing to do. I'll +1.

Who would have known, turns out this is bad mojo, and doesn't play nice with other configurations, causing regressions #5273 and 5294 . Seems that in some environments if you are smart enough to set the Sender, you are dumb to be a spammer, and so they run it through "callout verification".

#9 in reply to: ↑ 8 ; follow-up: @mattyrob
17 years ago

Who would have known, turns out this is bad mojo, and doesn't play nice with other configurations, causing regressions #5273 and 5294 . Seems that in some environments if you are smart enough to set the Sender, you are dumb to be a spammer, and so they run it through "callout verification".

That's bad news - seems we're damned if we do set Sender and we're damend if we don't depending on the server config :-(

I think we should set it to be honest as it's consistent and valid. Any host dropping email that sets Sender is being overly strict IMO.

#10 in reply to: ↑ 9 @lloydbudd
17 years ago

Replying to mattyrob:

consistent and valid.

That's the problem, for most members it isn't valid. wordpress@[domain] doesn't exist.

#11 @westi
17 years ago

I guess the solution is going to be:

#12 follow-up: @lloydbudd
17 years ago

westi, sounds good. A longer term project may be to evaluate using the admin email address or helping set and ensuring a valid email is used (maybe using this same "callout verification" technique, hehe) -- its own ticket, for another day.

#13 in reply to: ↑ 12 ; follow-up: @westi
17 years ago

Replying to lloydbudd:

westi, sounds good. A longer term project may be to evaluate using the admin email address or helping set and ensuring a valid email is used (maybe using this same "callout verification" technique, hehe) -- its own ticket, for another day.

I'm pretty sure we used to use the admin address at one point - I can't remember exactly why it changed to the current method.

#14 in reply to: ↑ 13 @mattyrob
17 years ago

Replying to westi:

I'm pretty sure we used to use the admin address at one point - I can't remember exactly why it changed to the current method.

My plugin tries to pull the admin details including the address - I often run into support issues because users have deleted the admin entry (ID = 1) from the database. This may be why that method is no longer used. It may also be because the supplied email could be on a different domain than the blog.

I think future work on using a valid email would be good. In the meantime I guess we'll have to reverse this change. :-(

#15 @torbens
16 years ago

  • Priority changed from low to normal

I'm also running into this issue. No mails due to nonexistent wordpress@... address. No documentation on this either in the upgrade guides. Occurs on a major German hoster. Therefore I'm boosting the priority.

I don't quite understand why you guys not simple use the email address specified at '/wp-admin/options-general.php'. It even says "This address is used only for admin purposes."

#16 @pishmishy
16 years ago

  • Owner changed from anonymous to pishmishy
  • Status changed from reopened to new

There doesn't appear to be any consistency as to when WordPress uses get_option('admin_email') or "wordpress@" . $_SERVER['SERVER_NAME'] to send mail.

Since the second case is only used twice I suggest standardizing on the first form. We can't guarantee it's a deliverable address, but it is almost certain to be. Simple patch attached.

@pishmishy
16 years ago

Standardizes sender address to admin_email

#17 @pishmishy
16 years ago

  • Keywords has-patch added

#18 @pishmishy
16 years ago

  • Status changed from new to assigned

#20 @lloydbudd
16 years ago

  • Milestone changed from 2.3.2 to 2.6

#21 @lloydbudd
16 years ago

  • Keywords regression has-patch removed

#22 @pishmishy
16 years ago

Why did the has-patch keyword get deleted? Is there a problem with the patch?

#23 @ryan
16 years ago

  • Keywords has-patch added

Added back has-patch, although I don't think we can use the patch because of #2053 and #1532. Using admin_email has its own problems.

#24 follow-up: @westi
16 years ago

(In [6599]) Revert #5007 as it causes more trouble than it solves. Fixes #5273 for trunk.

I have reverted this for 2.5

#25 @pishmishy
16 years ago

westi, despite owning the ticket I've not really been following it.

I'm a little confused by the relationship between this bug and #5273. Both seem to be problems caused by sending e-mail from "wordpress@" . $_SERVERSERVER_NAME? when $_SERVERSERVER_NAME? isn't necessarily a valid domain for sending mail. I guess there are also circumstances where it's not valid to use admin_email either. I think that makes solving these two tickets mutually exclusive.

What I might be tempted to suggest is standardizing on one address or the other and pointing out in the install process that the address must be valid for that server. Perhaps I'm just confused though =)

#26 in reply to: ↑ 24 @lloydbudd
16 years ago

Replying to westi:

(In [6599]) Revert #5007 as it causes more trouble than it solves. Fixes #5273 for trunk.

I have reverted this for 2.5

Ryan reverted in branch/2.3 today in preparation for a security release:
(In [6706]) Revert #5007 as it causes more trouble than it solves. Fixes #5273

#27 @pishmishy
16 years ago

I'd like to be able to close this particular ticket. It'd be nice to standardize on using get_option('admin_email') or "wordpress@" . $_SERVERSERVER_NAME? throughout but there'll be difficulty getting this right.

People's hosts might refuse to send mail from the first because it's not a domain they're aware of, hosts may refuse to send mail from the second because the wordpress@ address doesn't exist.

Could we document this in the code? If the issue appears again then this ticket can be found from the code. I'd be happy to close this ticket then.

#28 @Viper007Bond
16 years ago

Let's just pick something and then filter it. Plugins can take care of fringe cases.

#29 @pishmishy
16 years ago

I thought we did that, with my patch? It caused far more problems than the current situation does ;-)

#30 @pishmishy
16 years ago

  • Keywords has-patch removed

I think we could safely remove line 118 of wp-admin/includes/upgrade.php which would simplify any change.

#31 @Viper007Bond
16 years ago

Oh, well I guess I should pay attention then, huh? lol

@pishmishy
16 years ago

Document the problems in the code

#32 @janbrasna
15 years ago

  • Cc janbrasna added
  • Keywords e-mail notification sender spam added
  • Milestone changed from 2.9 to 2.8

Since both admin_email and wordpress@SERVER_NAME won't do for 100% of installations, another configuration field might be appropriate to set just the e–mail sender for notifications and other system e–mails.

Maybe just at the time of installation as not to clutter the blog settings?

#33 @jacobsantos
15 years ago

  • Milestone changed from 2.8 to 2.9

Documentation patch should be committed.

#34 @westi
15 years ago

(In [10575]) Add documentation to detail reasoning for default from address. See #5007 props pishmishy

#35 follow-up: @westi
15 years ago

I think we should close this ticket now we have that documentaion and there is a filter available for the % of sites where the current method doesn't work.

#36 in reply to: ↑ 35 @tigertech
15 years ago

Replying to westi:

I think we should close this ticket now we have that documentaion and there is a filter available for the % of sites where the current method doesn't work.

I'd like to object to closing this.

Having WordPress make up a fake address to send from, and then documenting that it doesn't work some of the time, is not a reasonable resolution to this problem.

Other scripts don't work this way. They either ask the user for a working address that the script is allowed to send from, or they just use the default setting without specifying any address, like WordPress originally did.

I run the mail system for a Web hosting company, and I can report that many extremely popular scripts send automated mail with no particular address set, allowing the system to fill in a valid address. The authors of those scripts expect it to work properly on properly configured servers, and it does; our mail logs show over 50,000 successful outgoing messages per day like this.

Because of that, I find it hard to believe that the original FastHosts problem (where they were blocking mail from scripts that didn't set a specific address) is widespread. The original bug here was not a WordPress problem, but rather a spam filtering bug at FastHosts that would prevent mail from being sent from many scripts. Heck, that system would prevent 3 of the 4 documentation examples for PHP's mail() function from working.

If the default address set on a server for script sending isn't allowed to send mail, that's a serious misconfiguration on the part of the hosting company, not something that WordPress should try to "fix" by making up a (probably nonexistent) address to send from.

The right solution is either to either ask the user for a working address that the script is allowed to send from, or to revert the original change so that WordPress uses the default address.

#37 @Denis-de-Bernardy
15 years ago

  • Keywords needs-patch added; e-mail notification sender spam removed
  • Severity changed from minor to normal

#38 @Denis-de-Bernardy
15 years ago

  • Component changed from General to Mail

#39 @aiusepsi
15 years ago

  • Version changed from 2.2.2 to 2.8.4

I'd also like to object to closing this. I just ran into this problem. The domain which the majority of things like my users' self-registration emails are going to be sent to is automatically spam-blocking the fake address Wordpress makes up here.

The way things work right now, the symptom from the user's side is that Wordpress inexplicably makes up an address, even when there's a setting in the admin interface. A filter makes it relatively simple to patch yourself, but you've got to dig though PHP to find the problem first.

#40 @westi
15 years ago

  • Resolution set to fixed
  • Status changed from accepted to closed

I closing this as fixed.

Please open a new ticket for any outstanding issue that you see.

#41 follow-up: @hakre
15 years ago

  • Resolution fixed deleted
  • Status changed from closed to reopened

I reopened this. The arguements given are totally resonable and should not be canceled that way. It's a serious issue.

#42 in reply to: ↑ 41 @westi
15 years ago

  • Resolution set to fixed
  • Status changed from reopened to closed

Replying to hakre:

I reopened this. The arguements given are totally resonable and should not be canceled that way. It's a serious issue.

Please open a new ticket for any outstanding issue that you see.

This ticket was mentioned in Slack in #cli by ipstenu. View the logs.


8 years ago

This ticket was mentioned in Slack in #core-editor by alexander-botteram. View the logs.


6 years ago

#46 @GwynethLlewelyn
3 years ago

Let me add my voice to those who are opposed to closing this ticket.

First, I fully understand why a 14-year-old ticket was closed. All teams using some sort of time-tracking software are under constant pressure from their supervisors to close as many tickets as possible. Your quarterly evaluation might be tied to the number of tickets being closed. I have never worked at Automattic, but I can fully understand if this is the case. An open ticket means a lazy developer. A closed one is work being done. I know the drill.

That said, it is good business practice to actually provide a concise and reasonable explanation about why the ticket was closed. "I closing this as fixed.
Please open a new ticket for any outstanding issue that you see." is not a valid reason.

The issue is not fixed. It's documented as not working, with a link to this ticket. Sure, writing good documentation is hard work, and my thanks for that, but the original message was about broken functionality, not about bad documentation.

Let me reinforce that again:

The original issue is not fixed. It wasn't fixed 12 years ago, and it is still broken today.

The argument that this issue only affected 'edge cases' 14 years ago (such as Fasthosts) is a fallacy. In fact, Fasthosts was just enforcing a very good practice, namely, checking for fake email addresses created from within a mail server. Indeed, these days, spammers don't just send out emails with fake addresses — these will very easily be discarded by anti-spam mechanisms. Rather, spammers prefer to infect systems (such as WordPress — or, more likely, one of the popular plugins), take advantage of the lack of strict checking inside the server, and just spam others with what is perceived as a legitimate address coming from a legitimate server.

That was 14 years ago, when people were much more worried about incoming spam than outgoing spam. There was a certain neglect in that area. Not so any more. These days — in 2021, at the date of writing — sending email from server to server, in a desperate attempt to curb spam/scams/malware, has become increasingly more sophisticated, with several layers of digitally encrypted signatures, and a plethora of authetication mechanisms (using DNS, for instance), in the attempt of making the lives of spammers more and more difficult.

It's not just Fasthosts that bothers with these things. Every email provider, in 2021, is required to do extra-strict checking on all inbound and outbound emails. Failing to do so results almost immediately into being placed into a blacklist.

Also, these days, 'being on a blacklist' is not just about spam. Malicious hacker activity is routinely monitored by many automated systems, and recurring culprits are flagged on global databases. Because one of the reasons for hacking a system is getting the ability to send spam out of a legitimate server, databases of hacking origin servers are mixed with those of spammers; in other words, attempt to hack another system, and your own server will not only be blocked for intrusion attacks but it will be blocked as a source of email, too. The reverse is also true: send some spam — even involuntarily (because you've been a victim of a hacking attack) — and your IP address gets blocked. Worldwide. In minutes, if not in seconds.

That means no more WordPress.

While 14 years ago such a scenario would probably be restricted to a few security-conscious hosting providers, today it's the norm, not the exception — at least on the major hosting providers: those that host millions of WordPress websites. And these large, popular providers are even more vulnerable to this issue — that's because, these days, most blacklists do not restrict themselves to a single IP address. They collect data and do statistical analysis — if a significant number of IP addresses, all belonging to the same provider, are sending emails that get labelled as not legitimate (worse: all of them coming from the same fake mailbox, wordpress@..., it's highly probable that all IP addresses in that block will also be blacklisted, or, at least, that email coming from those addresses get tagged as coming from a 'suspicious' provider. Get several providers exhibiting the same behaviour, and you start to see country-wide blacklisting. Again, this is not just about email. These days, an organisation may start just having their email being blacklisted, but eventually, their whole IP block may get cut at the source — packets getting dropped by routers world-wide. Or at least these packets will get a lower quality of service; this is left to the discretion of those who manage the backbones of the Internet. But they're becoming more aggressive and restrictive.

In fact, even organisations with impeccable track record get blocked occasionally. Google, for instance, has one of the best anti-spam/anti-scam/anti-malware solutions implemented on Gmail. Nevertheless, every other month or so, some of their outbound email relayers get blacklisted (at least temporarily — say for 48 hours, or a whole week). I'm just aware of that because users of my own server, having local mailboxes hosted there, start complaining that 'they cannot get any email from Gmail'. Obviously, they complain to me. But even if they complained to Google instead and were taken seriously (which is not a realistic scenario), there is next to nothing that Google can do about that. If one of their outbound servers has been blacklisted, it means that spammers have found a way to send at least a few spam emails out. Often they do that by creating a multitude of accounts with randomly generated usernames, then send email out of all of them simultaneously — staying just one step ahead of Google's automated account cancelling software — many of which, since they coming from a highly reputable organisation, will go through and eventually reach their victims. Not, however, without ringing a few alarms all the way. Google's own servers might just get preventively banned for a while, giving Google a few days to handle the issue on one particular server caught at sending out spam. Eventually, service will be restored...

My point is that not even all-powerful Google is immune to being blacklisted. They do send spam occasionally — which come from 'fake' accounts. Google is powerless to prevent the blacklisting services to list them as spammers; all they can do is, like everybody else, request removal from a specific blacklist (which is usually granted but can take hours/days...)

Also, knowing that all WordPress installations will generate a wordpress@... address, which the server's sysadmin has to make sure that it gets accepted as legitimate (even if it isn't being read/monitored by any human), spammers will know, after hacking into a WordPress website, that at least one address coming from that site will be legitimate. Granted, the lack of this knowledge hasn't stopped spammers in the past (or the present). But it's definitely something to be taken into account — especially because it's an open invitation available on all WordPress sites since the dawn of history. Obviously, hackers might simply get email addresses directly from the user table; but such emails may not be local. A good fall-back option for the hackers, therefore, is just to try sending emails out of wordpress@local.server.name.domain.tld — it's likely that these emails will at least go through the local MTA's anti-spam measures.

Anyway... this message is already way too long; my point is that there is still a lot to be discussed on this particular issue and that it is by no means 'resolved' or 'fixed'. Worse, due to the extra layers of security imposed by today's email servers, the situation has become potentially much more problematic than it was in 2007.

Paraphrasing myself from a discussion tread on the WordPress support forums, I still maintain that using the Administrator Email instead of a 'fake' address is a much better choice because:

  1. Usually, the admin email gets verified before being used, so it’s more likely to be a valid email;
  2. Admins get periodically notified by WordPress itself, when logging in, to re-check if the email address for the site administration is still valid;
  3. It’s far more likely for a site admin to check their own email address periodically rather than a ‘fake’ address ‘invented’ by WordPress (which, if it exists at all, may be routed to nowhere);
  4. It’s also reasonable to expect that the site admin’s own email address is less likely to be filtered out by their own mail servers’ anti-spam/anti-scam/anti-malware tools than a ‘fake’ address which is neither registered nor whitelisted anywhere (unless, of course, one is aware of the need of doing so a priori).

That said, the best solution, of course, is to refrain from creating 'fake' email addresses. At the very least, I would expect that such a critical option/decision was under user control — an option somewhere on the settings panel or at least on wp-config.php. The default behaviour, however, ought to be to use the admin email for all communications and notifications, giving users the option to select a different email if desired.

Fortunately, there is currently a workaround that can limit the damage: the notifications email address (and user name) can be filtered out. The simplest way of doing so is to add a few lines to the theme's functions.php, but an alternative is to add it as a plugin (so that it carries on between theme changes), or, even better, as a Must-Use plugin, installed by the sysadmins by default on all WordPress sites. On large-scale hosting providers this can be easily achieved by the already-existing automated tools; all that is required is to make these organisations (and their sysadmins) aware of the issue.

It's a suboptimal solution, but at least it works today. Nevertheless, there is still a pressing need for a permanent fix: it's counter-intuitive to have to add additional functionality (a theme, plugin...) to remove what could otherwise be described as a security flaw (at the very least, 'a bug' or 'an issue') just because some people do not see it as such. Instead, everything should work the other way round: using the admin email ought to be the default, and any other exception to the rule should be made optional (possibly even delegating it to a plugin, if the user so desires).

Please stick to the Principle of Least Surprise.

A concerned system administrator (who also does a bit of WordPress administration)

Note: See TracTickets for help on using tickets.