Unicode Email Addresses

[ysalame]

Tested against trunk (2015-04-16)

Test case

$target_email = 'dummy-üñîçøðé.y.!#$%@üñîçøðé.gmail.com';
echo $target_email.'<br>';
echo sanitize_email($target_email).'<br>';
echo 'is_email : '.is_email($target_email);

Return

dummy-üñîçøðé.y.!#$%@üñîçøðé.gmail.com
dummy-email.y.!#$%@gmail.com
is_email :

Function is_email @ /wp-includes/formatting.php line 2177
Preg_replace @ line 2211 is not correct.

if ( !preg_match( '/^[a-zA-Z0-9!#$%&\'*+\/=?^_`{|}~\.-]+$/', $local ) ) {

Function sanitize_email() @ /wp-includes/formatting.php line 2430
Preg_replace @ line 2460 is not correct.

$local = preg_replace( '/[^a-zA-Z0-9!#$%&\'*+\/=?^_`{|}~\.-]/', '', $local );

Related

• #4616 Previous validation expansions.
• #9316 Introduces complex email validation with messages for rejected inputs.
• #24487 is_email() doesn’t recognize IDN domains.

• ​Extending Unicode Support in Email Addresses (Make)
• ​Call for testing

[boonebgorges]

Can you specify exactly what the bug is? As far as I can see, dummy-email.y.!#$%@gmail.com is a valid email address. See eg ​http://en.wikipedia.org/wiki/Email_address#Local_part.

[ysalame]

ugh... sorry. I actually pasted the email that was sanitized.

The test I made was

$target_email = 'dummy-üñîçøðé.y.!#$%@üñîçøðé.gmail.com';
echo $target_email.'<br>';
echo sanitize_email($target_email).'<br>';
echo 'is_email : '.is_email($target_email);

with return as

dummy-üñîçøðé.y.!#$%@üñîçøðé.gmail.com
dummy-email.y.!#$%@gmail.com
is_email : 

The unicode characters were all removed. For international emails this can be a real problem.

ps. I actually used the Wiki page you sent as a base for my ticket. I tried a mix of one of the last examples in the "Valid email Examples" list.

[jasonhendriks]

I had looked into email validation when I wrote my SMTP plugin, and eventually concluded it is nearly impossible to validate an email address at all. Most email validators are much more restrictive than any RFC requires.

I think currently my code doesn't even call sanitize_text_field because it failed too many of my test cases.

​https://wordpress.org/plugins/postman-smtp/

​http://girders.org/blog/2013/01/31/dont-rfc-validate-email-addresses/

[miqrogroove]

Replying to ysalame:

ugh... sorry. I actually pasted the email that was sanitized.

We need an admin to update the ticket description then.

[desrosj]

#51732 was marked as a duplicate.

[prfidneai]

Hi!

I opened #51732 ticket to propose a patch to change email validation rules to work with EAI, but it was recognized a duplicate of this one.

As I can see, nothing was changed with EAI for 6 years from the time this ticket was open and Unicode EAI cannot be used in WP. I would like to propose some things to fix it:

1) Let's modify sanitize_email() and is_email() to use special constants in default-constants.php for regex check username and domain name (WP_IDN_LOCAL_MAIL_REGEX and WP_IDN_DOMAIN_REGEX) which every admin can change to validate IDN domains rules for language he uses. For example I added Russian IDN rules to these regexes.

2) Let's also add decode_punycode() to sanitize_email() function which check if e-mail domain part in Punycode from and convert it to Unicode as required by uasg.tech

3) In addition to all these things we can modify sanitize_user() in the same manner to allow Unicode usernames to WP Universal Acceptance ready.

I attached the patch for all of these.

[liedekef]

This patch does not solve the problem with e.g. "测试5@普遍接受-测试.世界"
A more simpler approach would be to use php FILTER_VALIDATE_EMAIL (which doesn't work for unicode, but we add the second if for that) here. An example validation function could look like this:

   if (filter_var($email, FILTER_VALIDATE_EMAIL)) {
        return true;
   } elseif (preg_match("/^([\w\-\.\+]+@([\w\-]+\.)+[\w\-]{2,63})?$/u", $email)) {
        return true;
   } else {
        return false;
   }

I guess even more simple would be to use just the regex:

   if (preg_match("/^([\w\-\.\+]+@([\w\-]+\.)+[\w\-]{2,63})?$/u", $email)) {
        return true;
   } else {
        return false;
   }

In any case is_email currently blocks unicode email addresses and should get fixed.

[agulbra]

Patch to add support for unicode email addresses in is_email, but making no other changes and resolving no other limitations

[ironprogrammer]

Test Report

Patch tested: https://core.trac.wordpress.org/attachment/ticket/31992/31992-icann.patch

Steps to Test

1. In WP admin, navigate to Users > Add New.
2. Enter a Username in standard ASCII characters (no need to confuse with related #5918).
3. Enter a valid Email with extended ASCII or unicode characters, e.g.: grå@grå.org.
4. 🐞 Observe error returned: Error: The email address is not correct.

Expected Results

• ✅ An email with extended ASCII or unicode characters should be accepted.

Environment

• Hardware: MacBook Pro Apple M1 Pro
• OS: macOS 13.5
• Browser: Safari 16.6
• Server: nginx/1.25.2
• PHP: 8.2.9
• Database: SQLite
• WordPress: 6.4-alpha-56267-src

Actual Results

• ✅ The Add New User form accepts the email with special characters.
• ❌ The email stored in the database (and subsequently displayed on the page) has special characters stripped out. E.g. (also see Figure 1):
  • grå@grå.org => gr@gr.org
  • 测试5@普遍接受-测试.世界 => (empty!)

Supplemental Artifacts

Figure 1: Emails stored in database from test samples.

[agulbra]

Thanks for testing, and thanks for not confusing it with that other issue.

My patch fixes only email address validation, it doesn't touch database storage. I'll be happy to work on database storage too, but do you want it on this issue? I lean towards fixing only is_email in this issue, and fixing other related issues in separate patches.

(JFYI, my most important goal is to accept unicode email addresses in forms managed by plugins, CF7 for example.)

[ironprogrammer]

@agulbra, yes, it's fine to work on the DB portion with a separate patch file, but please do include it with this ticket. Based on testing 31992-icann.patch, addressing the DB storage issue is first required to prevent the issue noted in comment:10. Also, there are other workflows that will need testing, like during site creation, etc, so this could very well touch even more areas in Core.

I wanted to note that ​GitHub pull requests are the preferred mechanism for code review, as they facilitate easier reviews and collaboration. (For instance, see ​this branch with your first patch applied.) PRs are often faster to update, and additional requirements can be addressed through additional commits as they reveal themselves, such as the database storage update.

I've marked this ticket with changes-requested to signify that additional work is needed on top of the existing patch.

(In addition to the ​handbook, I invite you to join ​Make WordPress Slack if you have questions about the contribution workflow, particularly the ​#core channel.)

This adds support for unicode email addresses in is_email and sanitise_email. It is intended to be sufficient to accept unicode email addresses in contact forms etc.,

it does *not* aspire to unicode support everywhere, in particular it doesn't deal with unicode in logins or passwords.

It also does not add a tests/formatting/sanitizeEmail.php; I wouldn't mind adding one, but don't quite understand your testing policies. It looks like you want to keep the tests fast by avoiding unnecessary tests, I'd say, but I don't understand what you consider necessary and unnecessary. Anyway, if you want a file I'm happy to add one.

I don't understand the test failures. They appear to be unrelated to these code changes. Could you take a look, @ironprogrammer ? Thanks.

In response to @arnt:

It also does not add a tests/formatting/sanitizeEmail.php; I wouldn't mind adding one...

Absolutely, please do add unit tests (or updates to existing tests) for sanitize_email(). While Core test code coverage is pretty low today 😓, it is a project policy that new Core merges include appropriate tests.

I don't understand the test failures. They appear to be unrelated to these code changes. Could you take a look, @ironprogrammer ? Thanks.

It's not uncommon for E2E test timeouts to occur in this repo's automated jobs. These can be ignored.

Thanks for the updated PR! The updates to sanitize_email() appear to have resolved the issue with storing the correct email in the database 🎉.

This isn't a formal test report (there are lots of scenarios needed), but I ran through a few smoke tests so as to get back to you quickly on what could be addressed next. It would be great to get additional reviewers/testers involved to cover what I've missed:

• ✅ Add New User: email with unicode characters is accepted and saved to the database correctly.
• ✅ Profile (edit user): email can be changed with unicode characters, and when saved, persists correctly.
• ❓ Do new user creation, email change confirmation, or password change emails go to the correct address?
• ❓ Do links in these emails work correctly?
• ❓ Does the forgotten password workflow work correctly with a unicode address?
• ✅ Login: logging in with a unicode email works as expected.
• ✅ Add post comment: email with unicode is accepted for post comments.
• ❌ Comments (approval list): emails with unicode are not displayed correctly (Figure 1).
• ✅ In the Comments list, the Email field under Quick Edit does appear correctly.
• ❓ Site setup wizard: can the initial admin use a unicode address, and do the emails work correctly?
• ❓ Do various author dropdowns display unicode addresses as expected?
• ❓ Are unicode emails in feeds presented accurately?

_Figure 1: Display of commenter unicode email addresses._

Sigh. I see what you mean.

My own focus is/was quite different: I wanted Contact Form 7 and similar to accept unicode email addresses, which requires some utility functions in Wordpress to do the right thing. I didn't aspire to complete support in Wordpress. The thinking behind your response seems to be that either Wordpress supports it properly or not at all, there's no inbetween.

And, sadly, I think you're right and I was wrong.

I'll extend the PR. It'll take a few weeks. I'll also find some testers.

Hi,

you asked for wider testing. Around 20 sites have now run the patch in production, the longest-running one for more than six months already. Nothing has broken. The sites are real (University of Somewhere, not someone's private five-visitor site).

If you want, I can send you the email addresses of some testers. When I pinged them last week, around half said it was okay to give you their addresses, I'm sure most will say the same if I ask a few times ;)

I liked the accounts earlier today.

Thanks for the update, @arnt, and for seeking wider testing! There are still some ​outstanding questions and one previous failure to consider, but it's great to see this moving forward! 🙌🏻

A good next step would be to rebase this PR on the latest trunk, and to resolve the ​coding standards issues to tidy things up for an updated review and testing. When the CI checks are in a happy state, then it will help attract more attention.

Finding testers who were willing and able to run a patch on sites that mattered to them was hard work. That took a good long while. Do you want their names? If so, please send me mail.

Rebasing is not a problem, of course. The other stuff was a little difficult. I only recently found a good workaround for IntlChar's lack of support for ​UAX24.

Can you suggest a model end-to-end test that I could emulate and write automatic tests for the desired features? Manual testing is possible, but I'd much prefer to have complete automatic test coverage.

Hi,

the remaining problems, as far as I can tell:

1. The code now requires PHP 7.4, which I thought was OK but now I see that 7.4 is merely recommended, not required. Oops. Not clear to me how to do mb_str_split in 7.2.
2. I've clicked around, but don't feel sure that I have all of the "various dropdowns". The filters in default-filters.php are okay, but I don't feel confident that every third-party plugin will be okay, or that I've looked at everything in the UI. Wordpress is large, there are nooks and corners…

I added some code to prevent PHP 7.2.24-7.3.x from choking on mb_str_split().

A bit on testing:

• ✅ Add New User: email with unicode characters is accepted and saved to the database correctly.

• ✅ Profile (edit user): email can be changed with unicode characters, and when saved, persists correctly.
• ✅ New user creation, email change confirmation, and password change emails go to the correct address.
• ✅ Links in all messages work correctly AFAICT (plugins can add new kinds of course…).
• ✅ The forgotten password workflow works correctly with a unicode address.
• ✅ Login: logging in with a unicode email works as expected.
• ✅ Add post comment: email with unicode is accepted for post comments.
• ✅ Comments (approval list): emails with unicode are displayed correctly.
• ✅ In the Comments list, the Email field under Quick Edit does appear correctly.
• ✅ WordPress works on an IPv6-only server (oh, you weren't wondering?)

And… oh drat… Must retest the site setup wizard, too much code has changed.

Two questions remain.

1. What author dropdowns display email addresses? I can't find anything. Lots of user names displayed but not email addresses. Maybe I misunderstand. I checked for usage of email in the code that reads the database and use of that code in display code.
2. How can email addresses ever appear in feeds? The feed generators don't seem to reveal authors etc.

Just set up a new site from scratch using 'rød@grå.org' as admin address; it works.

Hi @ironprogrammer,

would you mind having a look?

Also, on an unrelated matter, I'm with you. I've worked as a full-time maintainer on a widely used open source project and got a lot of flak. It hurt. Some people can be really shitty. Don't let it grind you down.

Thanks a lot, @arnt, and apologies for not being involved with this the past several months 🙏🏻. I've got some other work priorities at this time, but hope to be able to dive back in soon!

In the meantime, for code review purposes, I would suggest raising this Trac ticket during ​#core ​Dev Chat, or dropping in a comment on the next ​agenda post to add it to open floor discussion. This will help spread some awareness of the initiative. If you're available on Slack and sync during that time to answer questions, that would be ideal.

Glad you could wrangle additional testing for the updates! If this info can be made public through ​test reports in Trac, that would be amazing -- and also prerequisite to merging as Core committers begin taking a look at the ticket.

You could get additional visibility by sharing the ticket in the ​#core-test channel. There may also be contributors willing to help put test instructions together based on what you've got so far. This update covers a lot of territory, so the more detailed the tests for each use case, the better.

[agulbra]

Hi @desrosj,

thanks for looking at this. I think reviewing/merging finally makes sense, since the required version of PHPMailer was released recently.

What can I do to help the review along? Would it help if I took a short trip to Basel for Wordcamp? Should ​https://github.com/WordPress/wordpress-develop/pull/5237 have any more documentation, or, or?

[agulbra]

This is the first of n test reports for this. All are basically the same — do something involving an å using unpatched WordPress 6.8, get a failure, do the same thing using this patch, it works. If you like Hindi, you can use अ instead of my å, if Chinese…

The domain grå.org belongs to me. Feel free to send mail there if you don't have an IDN yourself. Mail to rød@grå.org lands in my inbox, I won't complain, mail to blå@grå.org lands nowhere.

Environment: WordPress 6.8, in my case Brave as browser, both on linux, no plugins, WordPress unpacked but the installer has not yet run.

To pass this test, you need ​https://github.com/WordPress/wordpress-develop/pull/5237 and PHPMailer 6.10.

Steps to test: Open the installer web page. Enter rød@grå.org as the admin email address in the WordPress installer.

Test results: All is well if WordPress completes its installation and a message is sent to rød@grå.org. With 6.8, the email address is rejected and you are told to enter a valid email address.

[agulbra]

Test report 2/n.

Environment: WordPress 6.8, in my case Brave as browser, both on linux, no plugins, WordPress freshly installed, no additional plugins, no pages added.

To pass this test, you need ​https://github.com/WordPress/wordpress-develop/pull/5237 and PHPMailer 6.10.

Steps to test: Open the WordPress admin page. Select "users" on the left-hand bar, then "add new user" near the top of the screen. Enter "grå" as username, "grå@grå.org" as email address. Choose any name and role. Click "add new user" near the bottom. (Grå is Norwegian, it means grey, and I use that as a test since I'm Norwegian. It's pronounced as org backwards.)

Test results: All is well if WordPress adds the user and sends an email to the address you specified (and I suggest checking the log file of the outgoing MTA to see whether the message was actually sent). Later, WordPress will make an author page that looks like "/author/grå" and call the author "grå" on posts, which is a step forward from "gra" (one generally prefers not to call blog authors "gra").

With 6.8, you get an error message instead, no message is sent, no user is added.

(Feel free to use the grå.org domain for testing if you don't have one. It belongs to me (Arnt Gulbrandsen) and I don't mind. rød@ is forwarded to me, grå@ is an autoresponder.)

[SirLouen]

Related #17491

Hi @arnt, I was curious if this PR would also handle things related to rfc 5322.
As I have a port of this ready in JavaScript ( Initially for @wordpress/url's isEmail ).

If we are fine with this, I can help with that in form of reviews.
Regards.

I know 5322 very well, and also ​its replacement and am not aware of any related problems in Wordpress or PHPMailer. (My name is in both documents.) I'd be very happy to fix any related problems, if you'll point out anything specific.

I'm also more than happy to look at your javascript code.

I know 5322 very well, and also ​its replacement and am not aware of any related problems in Wordpress or PHPMailer. (My name is in both documents.) I'd be very happy to fix any related problems, if you'll point out anything specific.

(CORRECTION: No, my name isn't in 5322bis. The editor removed the names, the list grew too long.)

Ah my bad then, I was just doing tests using the current patch and saw some failing tests for comments, quotes and short domains ( I think that's what they called ? ), hence I wanted to add some improvements on them. 😄

I'm also more than happy to look at your javascript code.

Sure! here is the ​code Link

main focus points would be: ​#L161, ​#L175, ​#L189-L241

and then I guess supporting ip addresses.

This was what I was able to support from my JavaScript version
### Screenshot
#### Current Patch Converage

#### My isEmail Converage

cc: @arnt

That's admirably thorough testing!

By coinidence, Pete Resnick (the author of RFC 5322) and I were in the same room when I saw your table. We went over the cases together and agree about most of it.

First, a bit about the email address syntaxes. There are four:

• 5322 defines a very permissive syntax for interacting with legacy software (to be used e.g. when you parse old mail archives)
• 5322 defines a stricter syntax for generating mail today
• 5321 defines an even stricter syntax for addresses used with SMTP
• HTML defines ​an even stricter syntax) for using addresses in HTML forms

In general it's best to use addresses that meet the requirements of all four. Addresses that are valid according to only some rulesets tend to cause problems for users.

Now, to the addresses, with my opinions for how to handle each:

• to..to@couc.ou and to.@couc.ou are invalid according to RFC 5321, therefore Wordpress cannot send mail to them. Rejecting these is a good idea.

• (to)to@couc.ou and to@couc.ou(to) will be mangled during transmission by 5321-compliant software. Both addresses will arrive as to@couc.ou. Rejecting these is a good idea (while allowing to@couc.ou of course).

• cow@[dead::beef], 1@23456789 and a few others use address literals, which have been used for attacks. The same attacks apply to Wordpress (you could use this to probe for IP addresses that the Wordpress host can reach, but which are firewalled away from the general network). Rejecting these is a good idea.

• toto@co and admin@mailserver1 are local addresses, not world-reachable. (They could also be addresses within a TLD, but none of the TLDs allow that right now.) You could ssh to a Wordpress server and run 'ping co', because ping allows a local hostname. However, it's almost certainly bad to let users enter a local address in a Wordpress contact form (an attacker could use that to probe the local networkl). Rejecting these addresses makes sense.

• 1234567890123456789012345678901234567890123456789012345678901234+x@example.com is longer than one of the RFC limits. Most software supports longer localparts today. However, I surveyed a corpus of about 500,000 addresses, and *only* software used longer localparts, no users. issue-1234567890123456789012345678901234567890123456789012345678901234@support.example.com is a good example. If Wordpress allows longer localparts, bots will use that to spam the contact forms, and it will not help people. My opinion is that it's _probably_ best to reject these. Pete doesn't have an opinion on this address.

• user@[IPv6:2001:db8::1]:8080 passes, and I think it should not. It's not deliverable via SMTP on the public internet. However, it's also not a unicode matter. I'm uncertain whether _this PR_ should remove support for it.

Thanks for this testing! I'll add some of the addresses to the unit tests you used to the unit tests and push a new revision. I may change the handling of the :8080 address too.

Thanks for sharing such a detailed explanation!.

(to)to@… and to@…(to) will be mangled during transmission by 5321-compliant software. Both addresses will arrive as to@…. Rejecting these is a good idea (while allowing to@… of course).

O.O, I did think that () being treated as comment would cause issue like same address on arrival, but I thought as we are only doing looks like an email ( actual description for functionality of isEmail), accepting them would have been good Idea, but yeah rejecting them would make a lot of sense from a practical point of view.

cow@[dead::beef], 1@23456789 and a few others use address literals, which have been used for attacks. The same attacks apply to Wordpress (you could use this to probe for IP addresses that the Wordpress host can reach, but which are firewalled away from the general network). Rejecting these is a good idea.

toto@co and admin@mailserver1 are local addresses, not world-reachable. (They could also be addresses within a TLD, but none of the TLDs allow that right now.) You could ssh to a Wordpress server and run 'ping co', because ping allows a local hostname. However, it's almost certainly bad to let users enter a local address in a Wordpress contact form (an attacker could use that to probe the local networkl). Rejecting these addresses makes sense.

Oh, then that rejecting them make more sense, I was thinking of them as from local perspective like using tailscale vpn which allows shortdomains.

1234567890123456789012345678901234567890123456789012345678901234+x@… is longer than one of the RFC limits. Most software supports longer localparts today. However, I surveyed a corpus of about 500,000 addresses, and only software used longer localparts, no users. issue-1234567890123456789012345678901234567890123456789012345678901234@… is a good example. If Wordpress allows longer localparts, bots will use that to spam the contact forms, and it will not help people. My opinion is that it's probably best to reject these. Pete doesn't have an opinion on this address.

From what i have seen while searching and going through rfc 5321 & 2821, 64 seems to be the size that the local part of email address are allowed and I think that's still a bit far for from practically so ig sticking to 64 size looks better to me.

user@[IPv6:2001:db8::1]:8080 passes, and I think it should not. It's not deliverable via SMTP on the public internet. However, it's also not a unicode matter. I'm uncertain whether this PR should remove support for it.

Uh bit confused on this one, is this supposed to be valid email address as you said It's not deliverable via SMTP on the public internet. and right now both implementation is counting it as invalid.

---
Thanks for sharing the details again, I will update my implementation to reflect these. 😃

I was confused about user@[IPv6:2001:db8::1]:8080. Looked at the table, switched tabs, wrote several paragraphs and by then I had forgotten what the table line says. That address is handled correctly IMO.

I'm not sure whether I get to do anything more on this today. But those addresses definitely need to go into a unit test, with comments as above.

Just two quick comments:

• to..to@couc.ou and to.@couc.ou are invalid according to RFC 5321, therefore Wordpress cannot send mail to them. Rejecting these is a good idea.

Also invalid according to 5322 section 3 (the more restrictive) syntax.

• (to)to@couc.ou and to@couc.ou(to) will be mangled during transmission by 5321-compliant software. Both addresses will arrive as to@couc.ou. Rejecting these is a good idea (while allowing to@couc.ou of course).

While OK according to 5322, you'd have to strip the comments before sending to an SMTP (5321) server, and I suspect software that is getting these address from a WordPress form does not do that all the time, so I would avoid.

Hi so I was checking on how to support quoted local emails and IPs on php side and updated my tests to follow the suggestions that were made during the discussion.

Did some updates on the patch and was able to make it work.
If it is fine to support quoted local and IPs then I can add review comments.

PS: updated the table format so it is easier to read now 😅

[agulbra]

Test 3/n

Environment: WordPress on the tip, after August 20 or so and with ​https://github.com/WordPress/wordpress-develop/pull/5237 applied. You need no plugins. You may use Mailpit 1.25.0 or later to pick up outgoing mail. Mailpit needs to listen on localhost port 25. The installer has been run, but nothing added.

I used Brave as browser and ran both on linux.

Steps to test: Open the WordPress admin page. Select "users" on the left-hand bar, then "add new user" near the top of the screen. Enter "grå" as username, "grå@grå.org" as email address. Choose any name and role. Click "add new user" near the bottom. (Grå is Norwegian, it means grey, and I use that as a test since I'm Norwegian. It's pronounced as org backwards.)

Click Users, select the user and click edit. You're now on .../user-edit.php. Scroll down to contact info and change the email address to something else, perhaps rød@grå.org, then click "update user" at the bottom.

Test results: All is well if WordPress sends email to the user to notify about the email address change. WordPress 6.8 will not succeed in sending this email.

[agulbra]

Test 4/n

Environment: WordPress on the tip, after August 20 or so and with ​https://github.com/WordPress/wordpress-develop/pull/5237 applied. You need no plugins. You may use Mailpit 1.25.0 or later to pick up outgoing mail. Mailpit needs to listen on localhost port 25. The installer has been run, but nothing added.

I used Brave as browser and ran both on linux.

Steps to test: Open the WordPress admin page. Select "users" on the left-hand bar, then "add new user" near the top of the screen. Enter "grå" as username, "grå@grå.org" as email address. Choose any name and role. Click "add new user" near the bottom. (Grå is Norwegian, it means grey, and I use that as a test since I'm Norwegian. It's pronounced as org backwards.)

Click Users, select the user and click edit. You're now on .../user-edit.php. Scroll down to Account Management and click 'Password Reset'.

Test results: All is well if WordPress sends email to grå@grå.org with a new link, and clicking the link brings up the password reset page. WordPress 6.8 will not send the mail.

[agulbra]

Test 5/n

Environment: WordPress on the tip, after August 20 or so and with ​https://github.com/WordPress/wordpress-develop/pull/5237 applied. You need no plugins. You may use Mailpit 1.25.0 or later to pick up outgoing mail. Mailpit needs to listen on localhost port 25. The installer has been run, but nothing added.

I used Brave as browser and ran both on linux.

Steps to test: Open the WordPress admin page. Select "users" on the left-hand bar, then "add new user" near the top of the screen. Enter "grå" as username, "grå@grå.org" as email address. Choose any name and role. Click "add new user" near the bottom. (Grå is Norwegian, it means grey, and I use that as a test since I'm Norwegian. It's pronounced as org backwards.)

Open an incognito/guest browser window, open wp-admin and log in as grå with the correct password. Click Profile, then edit the email address to blå@grå.org and click 'Update Profile'

Test results: All is well if WordPress sends email to blå@grå.org with a link to confirm the email address, and clicking the link brings works. WordPress 6.8 will not send the mail.

[agulbra]

Test 6/n

Environment: WordPress on the tip, after August 20 or so and with ​https://github.com/WordPress/wordpress-develop/pull/5237 applied. You need no plugins. You may use Mailpit 1.25.0 or later to pick up outgoing mail. Mailpit needs to listen on localhost port 25.

I used Brave as browser and ran both on linux.

Steps to test: Open the WordPress admin page. Select "users" on the left-hand bar, then "add new user" near the top of the screen. Enter "grå" as username, "grå@grå.org" as email address. Choose any name and role. Click "add new user" near the bottom. (Grå is Norwegian, it means grey, and I use that as a test since I'm Norwegian. It's pronounced as org backwards.)

Open an incognito/guest browser window, open wp-admin and log in as grå with the correct password. Click Profile, then edit the email address to blå@grå.org and click 'Update Profile'.

Test results: All is well if WordPress sends email to blå@grå.org with a link to confirm the email address, and clicking the link leads to the expected page works, and the new addres can then be used to log in. WordPress 6.8 will not send the mail.

[agulbra]

Test 7/n

Environment: WordPress on the tip, after August 20 or so and with ​https://github.com/WordPress/wordpress-develop/pull/5237 applied. You need no plugins. You may use Mailpit 1.25.0 or later to pick up outgoing mail. Mailpit needs to listen on localhost port 25. The installer has been run, but nothing added.

I used Brave as browser and ran both on linux.

Steps to test: Open the WordPress admin page. Select "users" on the left-hand bar, then "add new user" near the top of the screen. Enter "grå" as username, "grå@grå.org" as email address. Choose any name and role. Click "add new user" near the bottom. (Grå is Norwegian, it means grey, and I use that as a test since I'm Norwegian. It's pronounced as org backwards.)

Open an incognito/guest browser window, open wp-admin and click 'lost your password', then enter grå@grå.org as email address and click 'get new password'.

Test results: All is well if WordPress sends email to grå@grå.org with a link to confirm the email address, and clicking the link brings you to the page where you can set a new password. WordPress 6.8 will not send the mail.

[agulbra]

Test 8/8 (I think these eight are enough)

Environment: WordPress on the tip, after August 20 or so and with ​https://github.com/WordPress/wordpress-develop/pull/5237 applied. You need no plugins. You may use Mailpit 1.25.0 or later to pick up outgoing mail. Mailpit needs to listen on localhost port 25. The installer has been run, but nothing added.

I used Brave as browser and ran both on linux.

Steps to test: Open the WordPress admin page.

Open an incognito/guest browser window, open the blog home page, click on the initial posting (the one with "welcome to workpress, this is your first post"). Scroll down, write a comment (try to make it rhyme), then enter address grønn@grå.org as address and click 'post comment'.

Return to the WordPress admin page. There should be a red circle next to 'Comments' in the left-hand-bar. Click 'Comments'. You should see your new comment (along with one supplied by the installer).

Test results: All is well if WordPress displays the author address correctly. WordPress 6.8 will display a remarkably garbled address (assuming you manage to post the comment).

Steps to test: Click Quick Edit.

Test results: All is well if the email address is displayed correctly under the comment body text.

[psykro]

@agulbra based on your 8 tests, would you say the PR is ready, or is further testing needed?

[agulbra]

IMO it's ready.

@arnt I sent you a PR with some improvements.
Also good to see that @dmsnell has self-assigned this. It will get through soon.

I'm grateful for the nitpicks. Seriously. I also wish this could be shipped; that would make life simpler for me.

I'd like to have a talk about growing/shrinking it. I'll keep an eye on Slack tomorrow. Have several meetings though, including one which I chair, so I might be pressed for time, depending on your timezone.

I decided not to rebase/push now. @SirLouen's comment change clearly should go in but it's minor, and I'd rather like to improve that cast but don't see how to write a unit test. I think we can continue talking without a new push.

I decided not to rebase/push now. @SirLouen's comment change clearly should go in but it's minor, and I'd rather like to improve that cast but don't see how to write a unit test. I think we can continue talking without a new push.

Btw, also while I was reviewing this patch, I sent you a little PR with some little extra formatting tweaks for the tests part. Nothing big, just improving CS a bit and setting a wider group to test this ticket at once.

@SirLouen I'll go and have some food now, and merge your PR either when I return or tomorrow morning. I really don't want to break anything due to jetlag and lack of focus.

jetlag

having just jumped ahead by nine hours, I feel something similar @arnt — hope you recover quickly.

---

here are some things I reviewed yesterday:

• WordPress appears to call sanitize_user() everywhere before detecting duplicate usernames, so changing that function should not accidentally allow duplicating user logins.
• the schema.php indicates a 60-character login and 100-character email. for utf8mb4 configurations that means people will still be able to enter emails with “100 characters”.

I was left with a few more questions in general though:

• for what reason are we adjusting sanitize_user() if this is about Unicode email addresses? I wouldn’t assume we have to do anything to the user objects.
• what should we be doing if the database table cannot store the email addresses? for instance, if a table is configured as latin1 we have a problem. while that allows us to store the raw bytes just fine, unless we ensure that PHP and MySQL agree on what’s transferring through the query we will end up with corruption.
• even if we ensure that we store non-ASCII characters properly, we end up with a length restriction for _some_ email addresses based on the fact that VARCHAR(100) means 100 bytes for a latin1 table but up to 400 bytes for a utf8mb4 table. this is something to consider otherwise people might end up with some confusing error messages when submitting a valid email that looks like it’s shorter than WordPress and the database count it.

CREATE TABLE test_l1 (t varchar(100) character set latin1 collate latin1_swedish_ci) DEFAULT CHARSET=latin1;

$wpdb->query( "INSERT INTO test_latin1 (t) VALUES ('a\u{1f170}b\u{1f171}')" );
$wpdb->query( "INSERT INTO test_latin1 (t) VALUES (_latin1'a\u{1f170}b\u{1f171}')" );

MariaDB [wordpress]> SELECT * FROM test_l1;
+----------------------+
|| t                    ||
+----------------------+
|| a?b?                 ||
|| a🅰b🅱           ||
+----------------------+
2 rows in set (0.000 sec)

In the first case, we sent UTF-8 data to the database, which knows that the table is storing latin1, so it transparently re-encoded from UTF-8 to latin1, replacing unrepresentable characters as ?. In the second case we told the database that the UTF-8 bytes should be explicitly interpreted as latin1 and so it did store the raw bytes, however…

php > var_dump( $wpdb->get_col( 'SELECT t FROM test_l1' ) );
array(2) {
  [0]=>
  string(4) "a?b?"
  [1]=>
  string(20) "a🅰b🅱"
}
php > var_dump( $wpdb->get_col( 'SELECT CAST(t as binary) FROM test_l1' ) );
array(2) {
  [0]=>
  string(4) "a?b?"
  [1]=>
  string(10) "a🅰b🅱"
}

…unless we also do something on the read query side then the database will perform the same transparent re-encoding on read turning those UTF-8 bytes _back into UTF-8_ and thus double-encoding them.

This seems problematic to me in a _new_ way because after this change we are allowing people to create accounts with Unicode login names and emails, but these database queries won’t fail; they will corrupt the data and someone might lose access to the account they just created, or to an account for which they just changed their email address. If we want to ensure this doesn’t happen we would need to trace the data through all places that access these and ensure we properly manage the character encoding in the SQL session, something that doesn’t tend to go well in WordPress due to the distributed nature of code contributions and plugins.

To me this highlights the need for deprecating non-UTF-8 support (Core-62172), though I guarantee we will be doing whatever we need to to support Unicode emails before we do that.

---

talk about growing/shrinking it

As named this PR seems very grand in purpose: what does it even mean to support RFC 6530? I feel like as a Core tracking ticket that is appropriate, but the work is likely to involve many independent PRs and changes, including some big semantic changes around usernames and emails, possibly involving some database schema changes.

For now, however, if we assume that this PR should move forward, here are some things I propose, pending further security review:

• Drop the detection for cross-script spoofing. I would love to see some evidence that the risk is substantial enough that it’s worth the complexity, but I suspect that having rules which change based on the content of the input is going to be confusing for users and developers.
• Missing from all of the changes are checks that the input is valid UTF-8. If we start by rejecting invalid UTF-8 then we can make a lot of assumptions and drop safety-checks without increasing risk. From my reading of the RFC I don’t see any allowance for addresses with invalid UTF-8.
• Maintain existing behavior in the antispambot function. That is, let’s not skip non-ASCII characters just because we don’t see them as being present as threats. I think we can probably deprecate that function entirely and skip this, but we will create another discussion for that.

Further, I think we need to talk about what email characters are allowable. Our current set of rejected ASCII character includes octets which could be allowable email addresses. If our intention is to be restrictive, say to reject non-printable characters, or ambiguous whitespace, then we probably ought to see if we can replicate that experience for the upper Unicode planes too and not just turn away from those goals because it’s not ASCII.

Please enjoy your conference. There’s no rush; as this work will probably take a lot of cooperation. We’ll bring it up next week in the developer chat.

[welcher]

This was reviewed in the 6.9 bug scrub today. We're 1 day from releasing RC 1 so we're going to move this to the 7.0 milestone.

@arnt

I added some code to prevent PHP 7.2.24-7.3.x from choking on mb_str_split().

At a glance, would a polyfill for mb_str_split() be helpful here, so that we could remove the function_exists() checks and make the behavior consistent across all supported PHP versions?

@SergeyBiryukov I would say that a polyfill makes sense if WP is to support PHP 7.2-3 for a while and a polyfill is simple. The ​support table makes it look as if support might be dropped in WP 7.0?

@dmsnell that's a long and very welcome message. Do you think I should attend to that now, or leave you in peace until 6.9 is released? I'd really like to merge this soon, for several reasons, but I imagine the 6.9 release needs developer focus.

Do you think I should attend to that now, or leave you in peace until 6.9 is released? I'd really like to merge this soon, for several reasons, but I imagine the 6.9 release needs developer focus.

@arnt you don’t need to worry about me or my schedule. just like ​anticipated objections, it will likely be more advantageous to just move at your own pace and let others work at theirs, be it fast or slow.

---

I am guessing the most important thing right now is figuring out the issue of database corruption since this creates a new avenue for that, and for potentially locking users out of their accounts because of it. we don’t necessarily have to solve it, but it’s a noteworthy impact from this change and creates a problem we didn’t previously have.

I'll write several separate comments here, on separate topics, and push a new revision at the end. First, cross-script usernames.

There's effectively no evidence of cross-script usernames being used, either for good or for bad. There's an exception and ha half for good: ① Q and other Latin letters appear to be used a little bit in Chinese, including the well-known name 阿Q, and ② Arabic digits (0-9) are used with many writing systems, even ones that have other digits. (Arabic itself is an example of that, in the areas east of Lebanon/Jordan.)

I know about web forums etc. that allow non-ASCII usernames. Unfortunately for this, the ones I know only allow one script.

There's domains, though. Domains in many TLDs are like self-service usernames, so that experience is instructive. ICANN receives abuse reports for domains and forwards them to registrars; I don't read that feed closely but I think I would hear about examples of cross-script similarity abuse. I haven't so far.

Visually similar letters have been used in real attacks, but I don't know about any cross-script use. Rather, i has been used for l (PaypaI a long time ago, phiIIips recently) and 0 for o, and one ​clever attack used a little-known diacritic and an all-Latin domain.

Impersonators mostly rely on non-letter similarity as far as I can tell. "Look, my email or web page has your bank's logo so I must be a legit representive of your bank." "My bluesky account has a photo of that person so you can trust that I really am that person." It's similarity but it's not letters.

However, I suspect that if it's possible to impersonate a wordpress user on the same site and use the impersonation for a swindle, someone will eventually do it. It's more likely to be done by copying a profile photo than by leveraging cross-script similarity.

So… I think it's reasonable to drop single_unicode_script, either nor or later. The only substantive reason to keep is that having it from the start is IMO easier than adding it later.

About UTF8 and other database encodings.

As I understand it, Wordpress 2.2 started using UTF8 for new databases and 4.2 converted old databases to UTF8, except when admins chose to override that. Overriding is possible to this day. There will be a few sites that do override.

I read ticket 62172 and it makes a great deal of sense to me. Agree about the brittleness. However, that change probably shouldn't be conflated with this.

It seems likely that a non-UTF8 database will suffer corruption in other tables, not just the usernames. That may be less serious. It won't lock anyone out. However, this problem could affect the core value of the site. When you publish a page, saving it to a non-UTF8 database can turn a dozen different currency symbols into question marks. Easy to overlook. Far from good. It seems… less urgent than usernames but worth solving.

Having written all this, I think I've formed an opion:

• This PR should check roundtrip conversion via DB_CHARSET and refuse to store user names, passwords and email addresses if that fails.
• If the check fails, the error message should say something like "This name/password/address cannot be stored in the database. Note: DB_CHARSET other than UTF8 is deprecated since 7.0 and will be removed in a future version of Wordpress" without naming a version number.
• The difference between utf8 and utf8mb4 is best ignored.

Does that sound good?

Next up: The scope of this PR.

As @dmsnell asks:

For what reason are we adjusting sanitize_user() if this is about Unicode email addresses? I wouldn’t assume we have to do anything to the user objects."

The first version of the PR did indeed touch only addresses. I extended it ​by request because as contributor, I try to conform to whatever is expected. At this point I'd prefer to keep the agglomerated PR because:

• The unit test and review issues are mostly similar.
• There are test scripts, props and commits that would either be difficult to split, or that would be bulky repetition.

Email address length:

Wordpress currently supports 100 bytes reliably, more unreliably. With the change I ​suggest above it'll store 100 characters reliably or reject the address at entry time.

Email address length limits are a bit of a mess in general. This is partly due to the specifications. The length limit for an email address differs from the length limit for an email address on the public internet. Too subtle for most people. Then there's ​widespread disobedience. and even worse, there's software and SaaS vendors that judge email addresses and have their own unpublished length limits. If you enter a 70-byte email address in a newsletter signup form, you may well be rejected because a SaaS vendor classifies the address as a bot.

Both 100 bytes and 100 characters are so high that any human who tries to use such an address will run into problems quickly. Much software has shorter limits (a well-known CRM has 80) and few people can spell reliably enough to type a really long address correctly from a business card.

It's sad, I wish the situation were cleaner, but 100 is a safe length in the world as it stands, and there isn't really any number that's obviously right.

FWIW, my name is in both RFC 5321 (length on net) and 5322 (length).

Does that sound good?

I’d like to hear the thoughts from some others who have more experience dealing with text encoding issues, particularly in places where we create new ones. Perhaps @mdawaffe has some or knows someone with thoughts.

A check for round-tripping seems reasonable and awkward at the same time, because making those checks everywhere would be excessive. For an email address though, maybe it’s worth the diversion.

It's sad, I wish the situation were cleaner, but 100 is a safe length in the world as it stands, and there isn't really any number that's obviously right.

We’re discussing a new reality which has no prior in WordPress and which can be dramatically different. The moment someone uses the flag of England in their email address, that’s 28 characters alone. Meaning that if someone uses the US flag in their email they are allowed 92 other ASCII characters but for the England flag it only leaves 72.

If we only allow storing Unicode emails on database tables with UTF-8 support then we could avoid this problem. Maybe that’s a reasonable feature-gate for this: don’t enable non-ASCII email address unless the {$wpdb->prefix}users table is stored as utf8mb4.

Were we to pursue that angle, it could be worth imagining a new global toggle or check like wp_supports_unicode_email_addresses() or something table-based. I know I have discussed the idea with @adamziel about checking database encodings at startup and caching them so we can reliably work with database tables regardless of the character set or collation. 🤔

I extended it ​https://github.com/WordPress/wordpress-develop/pull/5237#issuecomment-1736475880 because as contributor, I try to conform to whatever is expected.

Perhaps it’s my poor reading skills, but I re-read that comment multiple times and couldn’t find the request you’re referring to. Maybe @ironprogrammer left another comment on an older version of the code?

You are likely underselling your opinions here. This is a collaborative effort and we are all working together; I know from your previous comments that there are times you read more into things than people leave when they comment. It’s okay to ask questions and engage in the back and forth — if we all instantly did whatever anyone asked us then we’d have chaos, or LLM-level-slop.

---

I’d really love to consider this only in the context of sites with a utf8mb4 users table. I don’t know what to recommend as a way to do that, other than one obvious and necessary-but-insufficient task which is to verify that the emails are valid UTF-8.

---

There are a few places email addresses are read from query args: I think in the initial site installation and in some new user requests. This is not relatively important compared to the other feedback, but it will be something we want to make sure we check at some point, largely to ensure that when decoded they are valid UTF-8 and that they are properly percent-decoded. Perhaps it’s fine as-is and the email functions will verify that when passed the information (apart from the percent-encoding).

Either way, it’d be great to confirm what happens when some emails are sent with invalid query args.

---

If anyone would be interested in testing the same steps that others like @USERSATOSHI and @ironprogrammer have performed, but with a site setup with latin1 configured as the blog_charset and as the default database table charset, that would be helpful.

It would be valuable to test emails containing “normal” Unicode like non-latin scripts (which use only up to three bytes) as well as addresses containing characters from the supplemental planes, such as emoji (requiring four bytes).

This is a cool proposal. Note that I have not made the time to fully understand the conversation above. I don't think anything I've written below is particularly new: apologies if it is obviously repetitive :)

This PR should check roundtrip conversion via DB_CHARSET and refuse to store user names, passwords and email addresses if that fails.

wpdb->strip_invalid_text() (and friends) does something similar:

• allows anything in latin1 or binary columns/tables.
• allows 3-byte UTF-8 in utf8(mb3) columns/tables.
• allows all UTF-8 in utf8mb4 columns/tables.
• Does an actual DB query for other encodings.

As you'd imagine from the method name, it strips characters (and optionally truncates the string) rather than bailing. I think, though, you could do a clever SELECT statement that would check charsets against the correct columns/tables, check if the result is different than the input, then bail as necessary before doing the UPDATE/INSERT.

I don't think that's necessary, though, since I think we should instead only allow non-ASCII email addresses in utf8mb4 tables: no roundtrip necessary. Just reject non-valid UTF-8 byte sequences in PHP and check column(/connection?) charsets. Any other DB setups are stuck with ASCII email addresses.

The difference between utf8 and utf8mb4 is best ignored.

I don't think we can ignore the difference. We should require utf8mb4 (supposing we take the direction I mentioned in my previous paragraph).

I don't have a strong opinion about limiting strings to having characters from only one script (plus ASCII). On small sites, it likely doesn't matter at all. On large sites (like WordPress.com or sites where WordPress+bbPress is used for public forums) with user registration available to anyone, spoofing is likely a big concern. I agree that uses_single_unicode_script() is easy to add now, easy to stop using later, and hard to add later, so it might be worth just doing now and reevaluating later. Note that (as you all know) domains are a special case. Many TLDs do not allow IDNs, and some only allow certain scripts. Email addresses contain domains :) I'm not sure if we should care about individual TLD limits. We probably shouldn't care.

I do have one strong opinion: non-WordPress code should be able to make "raw" SELECTs to the database and retrieve the correct username/email address. That is, we should not write "corrupted" or otherwise transformed data to these columns even if WordPress correctly uncorrupts/untransforms the values when it read them. That means no ​charset shenanigans or hacky encodings (e.g., Base64).

For usernames, another (non-?)option would be to allow any characters, but to reject new usernames that are too "close" to existing ones. I don't know how to do that efficiently without adding a new (indexed) column to the users table, which I think is a bad idea.

Fly-by suggestions:

• start with email addresses only. I'm reasonably confident that non-ASCII usernames will break something, but I have no clue what :)
• consider making non-ASCII values opt-in (with a WP filter), so we can get real world experience with sites that want it without altering the behavior of sites that don't care. If it all works: make it opt-out in some future version. (I don't know if we ever do that sort of thing. Perhaps I'm being too conservative.)

I just pushed a new revision.

I discovered that wpdb already protects against not being able to store the data. The database may corrupt something, but wpdb already ​tries really hard to avoid getting that far. I added an extra unit test, since locking people out is worse than most failures.

Side note: That function uses latin1 in a way that looks incompatible with selecting using ilike at first glance. I didn't look closer into that, though.

FWIW I would be unhappy with a toggle for email address support. Those toggles are trouble magnets and I strongly advise against going that way. A toggle for username support is another thing.

I know a data set that can be used for username similarity detection… but that's an unpleasant slippery slope, with much scope for pedantry about details. Some of the decisions in the data set puzzled me. Best avoided, I'd say.

I've no opinion on restricting this to UTF8 databases. If WP were my baby, I'd announce that support for other databases will be sunset in the middlish future and thereby make the smaller question go away, along with the risk caused by brittle code.

I didn't have time to rebase, I'm afraid, but I wanted to get this pushed before I go on a trip tomorrow morning.

[senadir]

I'm looking into modifying WooCommerce checkout to accept such emails, for now, in the client, we use @wordpress/url isEmail which has this regex that won't pass the above /^(mailto:)?[a-z0-9._%+-]+@[a-z0-9][a-z0-9.-]*\.[a-z]{2,63}$/i;. Would it be possible that once a regex in this patch land, it also includes a change in the WordPress JS package?

[mukesh27]

@SergeyBiryukov Is this on your list for the 7.0 release?

@dmsnell Have you had a chance to review the latest changes in the PR since your previous review? Do you see any blockers for landing this in 7.0, or should we move it to a future release?

[agulbra]

I don't see any blockers, but that doesn't mean there are none. I'll list the issues that may be open, and my opinion:

• Email address length. The patch allows more than the strictest common validators. It's possible to make a valid address that's rejected by a lot of web signup forms. This PR permits everything that the common signup forms allow, and then some. I'd love to get more clarity on that (and am working towards that goal) but for the moment I think this PR is right.
• The PR's and issue's title vs. content. The content expanded as a result of comments and requests. At this point it would be difficult to split the PR while keeping the commit names etc. properly, so I lean towards merging as is, and perhaps renaming the github PR to reflect the eventual content rather than the initial idea.
• Latin1 database and what happens. If that's a blocker I'll test it. My gut feeling is that since UTF8 has been the default since 2006, sites with latin1 databases may be so idiosyncratic that my testing won't catch their possible issues. I can do it, I just don't feel that a positive outcome will promise very much.
• Allowed characters. I'll implement whatever. There's rough consensus outside WP on what a sensible repertoire is. Not perfect agreement.
• utf8 vs. utf8mb4. I don't have a strong opinion. Creating a domain/email address where the difference matters is nontrivial. I know how do it, it's far from impossible, but it's nontrivial enough that I don't consider the issue important.

I hope I haven't missed anything.

[dmsnell]

In the linked PR I left more extensive updates, but in short I do not think it’s feasible that this is ready for 7.0, as much as I want it to be. I’ve asked @agulbra to draft a post for Make to discuss the high-level status and impact of this change, independent from the code itself.

My guess is that there’s a lot more at stake here than it seems from the code itself, as third-party vendors will not necessarily know to expect non-ASCII emails, and some of that plus UX questions plus database questions make it a prime candidate for gathering more broad input.

This is an important and valid change for WordPress// but I want to make sure we don’t rush it too fast — it’s only been open for a decade 🙃

---

What I think is most likely is turning this into a series of smaller related changes. I could see it going one of two ways: make a bunch of tactical updates which permit non-ASCII email in the existing places in Core that would break; or go all out and try to cover all of the UX considerations.

Known needs

• We have to open up the email validator to allow non-ASCII bytes. Presumably this should require at a minimum that the bytes form valid UTF-8.
  • There is room for discussion beyond this as to how best to further restrict the allowable email addresses, but that seems secondary to me to this issue.
  • It’s worth discussing Unicode normalization and capitalization rules. A survey of how existing major email providers handle this would be great to see.
• Site owners may not be aware of the changes to allowable email addresses. It’s probably the wrong option, but these could be opt-in by default, giving admins an opportunity to read and understand the implications of the change and a health-check before turning them on with their potential consequences.
• Core should be audited for assumptions about ASCII email addresses; same goes for the database, particularly when tables are stored as latin1 for pseudo-UTF-80support. We don’t want errant collation settings to open up vulnerabilities because an email address was queried in a different form than it was stored.

[audrasjb]

As per today's 7.0 pre-RC1 bug scrub:
We're very close to RC1, and there's still work to do. Let's address this early in milestone 7.1. Ping @dmsnell.

Trac ticket: 31992

[dmsnell]

In 62225:

Email: Add unit tests covering email validation and sanitization.

In preparation for later work to allow non-US-ASCII email addresses, this change extends the unit test suite for is_email() and adds new tests covering antispambot() and sanitize_email().

This work was done collaboratively during WordCamp Vienna, 2026 as a Contributor Challenge in cooperation with and support from ICANN and also GeoTLDs Universal Acceptance Local Initiative.

Developed in: ​https://github.com/WordPress/wordpress-develop/pull/11552
Discussed in: https://core.trac.wordpress.org/ticket/31992

Props agulbra, akirk, benniledl, dmsnell.
See #31992.

[dmsnell]

In 62226:

Email: Add unit tests covering email validation and sanitization. (Take 2)

When the original patch from PR#11552 was merged, it did not include the latest version of the PR code, which had removed a failing test.

This patch removes the failing test to match what ran in the tests on the PR.

Developed in: ​https://github.com/WordPress/wordpress-develop/pull/11552
Discussed in: https://core.trac.wordpress.org/ticket/31992

Follow-up to: [62225].

Props agulbra, akirk, benniledl, dmsnell.
See #31992.

Merged in a34cf16ab66c3534154a6728a536fe7cc9e91af8
[62225]

[dmsnell]

Sharing for reference, because it was a nice blog post on email obfuscation, and this work touches antispambot()

​https://spencermortensen.com/articles/email-obfuscation/

Trac ticket: Core-31992

Updates antispambot() to prevent breaking with multibyte email addresses.

Need to split the mb_ord() polyfill into a separate ticket.

[amykamala]

Some hosting tests are failing in relation to this issue, errors with parsing xml: ​https://github.com/WordPress/phpunit-test-runner/issues/310

This refactors code into a new WP_Email_Address as we discussed in Vienna and leaves is_email and sanitize_email much shorter. Changes:

• is_email now accepts everything that passes the checks used for <input type=email> in the top browsers.
• It's now really easy to disable unicode address support, just change filters.
• sanitize_email() is well-aligned with mistakes people actually make. (New, only tangentially mentioned in Vienna.)
• sanitize_email() will not change an address if there's doubt about the intended result. For example, thorleif@… could be a different student from thorleif@… (ifi had a separate mail system, maybe still has) and the if there was a soft hyphen near the first dot, the previous sanitization could result in thorleif@… or thorleif@…. The current one performs only changes that are safe and correspond to known common problems such as pasting in an end-of-sentence dot at the end of an address, or copying from a ​source that inserted soft hyphens.
• some mostly unrelated tests expected unrealistic things, e.g. support for 190-character domains. This has now been changed to 60-character domains (still testing the same thing, in this case db column length).
• is_email and sanitize_email keep the filters for things a site admin can override, which is not everything (noone can make the DNS support 190-character labels, it's beyond our power).
• Pasting in info@… may be transformed to the human-readable format. (When WordPress sends email to that address, the To: header will say grå.org and the RCPT TO probably xn--gr-zia.org.)

It would be possible to split this into three:

1. Simplify is_email and add support for unicode addresses using the new WP_Email_Address.
2. Simplify sanitize_email, align it with observed mistakes, get rid of unsafe changes.
3. Enable unicode support.

I'm not sure, but perhaps a good idea is to split it like that and commit all three at the same time, for clarity in svn?

Trac ticket: Core-31992

Reported in WordPress/phpunit-test-runner#310

When serializing test output into XML, invalid UTF-8 bytes lead to a failure to load those test results when they are read. This patch adds code to remap those invalid bytes in an ASCII-readable form, whereas the invalid bytes are separated by parentheses and encoded in their hex form.

This ensures that a proper XML file is generated from the testing results.

## Use of AI Tools

AI assistance: No

[dmsnell]

In 62249:

Tests: Print invalid UTF-8 as ASCII to fix hosts test reporting failures.

When serializing test output into XML, invalid UTF-8 bytes lead to a failure to load those test results when they are read. This patch adds code to remap those invalid bytes in an ASCII-readable form, whereas the invalid bytes are separated by parentheses and encoded in their hex form.

This ensures that a proper XML file is generated from the testing results.

Developed in: ​https://github.com/WordPress/wordpress-develop/pull/11620
Discussed in: https://core.trac.wordpress.org/ticket/31992
Reported in: ​https://github.com/WordPress/phpunit-test-runner/pull/310

Follow-up to: [62225].

Props agulbra, amykamala, codexdemon, dmsnell, mywp459, rolle.
See #31992.

Closed by [62249]
​https://github.com/WordPress/wordpress-develop/commit/4440667d806019cbe9b9e67fad6836b0d74cce6d

@arnt I’ve been distracted by other work this past week but I’ll try and follow-up here over the weekend. I only gave it a coarse read but I like the direction here. cheers!

Merged in [62425]
cc2779291c7c34e4630b428fe3c22edb45d0b2a1

[dmsnell]

Merged [62425]
Accidentally marked the wrong ticket with that commit.

[dmsnell]

Noting some follow-up work:

• Normalize all Unicode domains to their punycode form in get_ascii_domain().
• Audit calls to sanitize_email() since some are slashed and some are unslashed.
• Add alerts in JavaScript to warn against unrecognized email addresses in the front-end before submitting to the server.

[dmsnell]

In 62482:

General: Add support for unicode email addresses in is_email and sanitize_email

This adds support for the unicode address extensions in RFC 6530-3 and refactors the code so there are fewer long regexes and less duplication between sanitize_email and is_email. A new class, WP_Email_Address, provides the shared parts.

Opting out of unicode support is easy, default-filters.php adds unicode support by adding filters, which can be removed.

sanitize_email no longer does major changes like removing an entire subdomain from someone's address, it only cleans up things like soft hyphens and whitespace — changes that happen when coping an email address from text.

Developed in: ​https://github.com/WordPress/wordpress-develop/pull/5237
Discussed in: https://core.trac.wordpress.org/ticket/31992

Props agulbra, akirk, benniledl, dmsnell, ironprogrammer, justlevine, mdawaffe, mukeshpanchal27, SirLouen, tusharbharti.
Fixes #31992.

[dmsnell]

Copying ​a report from the Make post, from @krupajnanda

---

Hello @dmsnell👋🏻

I did the testing on a local WordPress trunk environment (Docker, utf8mb4 database, MailHog configured).

Note: Testing was performed on a vanilla WordPress setup only. I did not test any third-party plugins, themes, or external integrations.

### Findings

✅ User creation with Unicode email addresses worked.

✅ Unicode email addresses were stored correctly in the database.

✅ Password reset worked for ASCII email addresses and for Unicode domains.

❌ Password Reset: Password reset failed for email addresses containing Unicode characters in the local part (before the @), e.g. joséexample.com. The reset key was generated, but no email was sent and WordPress displayed:

The email could not be sent. Your site may not be correctly configured to send emails.

❌ Comment Submission: joséexample.com failed validation, while josé@éxample.com passed.

❌ Duplicate Email Validation: joséjosé@grå.org and josejose@grå.org were treated as the same email address and the second user could not be created.

❌ PHP Warnings: During duplicate email validation, the following warnings were displayed:

Attempt to read property user_login on bool
Attempt to read property user_email on bool
Cannot modify header information – headers already sent
⚠️ Observation: User search appears to be accent-insensitive. Searching for josé and jose returned the same results.

Thank you for the testing opportunity. I have reproduction steps and screenshots available if they would be helpful.

[dmsnell]

Based on an analysis from Codex, verified in mysql, database collation is getting us. @agulbra — this is a tricky one. I expected that we would want to store the Unicode address in the database so that basic searches would work; _i.e._ we want WHERE user_email = "josé@éxample.com" to work and not have to translate into Punycode and…??? for the local part?

Looks like we can either force collation in the SQL query or store the Punycode variant, which may not actually be an option because we don’t have a way to translate the localpart. Going back to our chats in Vienna, it seemed like the big email providers use byte-for-byte equivalence — am I remembering that properly? This would mean that NFC and NFD variants of the same address would be considered distinct, which again, I think is desired in this case? (just the same way that plus-address variants of the same mailbox are actually the same address, but generally treated as distinct for purposes of logins).

SELECT * FROM $wpdb->users WHERE BINARY user_email = %s

[dmsnell]

Email form field rejection

[dmsnell]

The comment form is rejecting email addresses with Unicode localparts. This is not a problem _per se_ with the added server support, but technically an interaction bug between Safari + Chrome + Firefox (which all seem to still validate only ASCII email addresses) and the <input type=email> fields. A theme could replace the comment form with a standard <input type=text> field to allow Unicode signups.

That means that WordPress will be capable of accepting them even if the browsers are not permitting sending them.

[agulbra]

Hi,

1. The first error here is likely that the mail system doesn't advertise SMTPUTF8.

2. The second is a tricky collation issue. In some countries this is a feature, e.g. in Italy Nicolò is considered to be the same name as Nicolo, in others it's more of a bug, in France Amélie is correct Amelie a misspelling. Overall, I feel that blocking distinct users from having extremely similar addresses isn't bad. I think we should do it the way the code is now until/unless there's a concrete bug report for pair of bona fide users. Let's call it an impersonation guard. (Larry Wall once answered a bug report by "let's call it an accidental feature".)

3. The browsers have an issue open for this. Progress has been extremely slow, for, uh, reasons. Including the lack of server support. It needs to be solved in the same way by Firefox, Chrome and Safari, or at least the teams think so. IIRC two of the teams have now committed to implement the address spec they'll decide on.

I believe that it would be bad mistake to use punycode in the database. Punycode breaks ilike '%foo%', for a start.

[agulbra]

Come to think of it, this collation accident works nicely with another feature.

I've seen two mail systems with unicode address support that will add ASCII versions by default (and choose sender address intelligently). I haven't seen what they'll do with an é, but I would guess that if you create user amélie@… they'll suggest adding amelie@… as an alias.

So if that Amélie doesn't remember exactly what she used to sign up for WordPress, WordPress will work. DWIM.

[dmsnell]

likely that the mail system doesn't advertise SMTPUTF8

That is what Codex surmised as well, particularly at the mention of MailHog. It would be problematic, I would think, to allow these emails and then not be able to send to them.

That’s not something easily and statically assertible either; it would simply appear after WordPress creates a user account and leaves them hanging.

I’m not sure what to do about this. It looks like an existing problem, which is that it’s possible to create an account for an undeliverable mailbox, but in this case we know that the problem is on the server. Maybe we need a follow-up enhancement to link to the Site Health screen and show mail configuration limitations? Perhaps we could create a throwaway address on wordpress.org so sites could attempt to send an email with Unicode localparts and report on failure.

The second is a tricky collation issue

Here I’m worried about confusion and inconsistencies. For example, I tried creating three account email variations:

• bücher@domain (with U+FC)
• bücher@domain (with u followed by U+0308)
• buecher@domain

WordPress accepted the first and last, but rejected the email with the combining diaeresis as a duplicate. If I query with the default collation on bücher@domain, WordPress returns the exact match; if I query with utf8mb4_de_pb_0900_ai_ci then it returns both addresses as matches.

So this seems like a case where system-dependent configuration is going to produce some bizarre bug reports, particularly since someone might change the locale after emails are in the database (for example, in a site import/transfer).

With the perspective of existing behaviors, WordPress allows creating multiple accounts for variations of the same mailbox (e.g. plus-addressing). However, it never treats those as equivalent, because no database collation elides the pluses, or GMail-style dots.

This is why I was wondering about BINARY and treating emails byte-for-byte.

The browsers have an issue open for this.

On this point I do not actually think we have to solve it. It’s something that themes need to take into account, but I think that a restriction in the browser is acceptable since the backend technically supports it. We can update official themes to allow this, or wait for browser support to update.

---

Would love to hear some thoughts from hosts or those close to email issues too.

[agulbra]

I've thought more about these things.

Let me try some rambling on the first of the three issues. There are several root causes that lead to undeliverable email messages:

• user typos
• spam filters
• incompatible TLS settings
• and newly: SMTPUTF8

Examples I've seen (I run a couple of web sites): user entered gamil.com as domain instead of gmail, user's host's spam filter blocked all OVH customers by default, user's host requires TLS and the ​outgoing MTA didn't do that by default. I haven't seen a lack of SMTPUTF8, because every server I touch supports that, but I've heard reports of it happening ;)

To my mind, this raises two questions. ① Is adding a fourth cause a real problem, when there already are three? ② Is fixing just the SMTPUTF8-related breakage practical? Can WordPress assess the outgoing mail channel?

I know someone who checks at install time. That sounds transferable to WordPress, I'm sure there's a hook that's run when core is upgraded, it could just set the value of the validation hooks if… it can't check exactly whether delivery will work. But it can approximate.

Or it's possible to live with it. The browsers mostly block this in <input type=email> for now (that will pass, though).

[agulbra]

The collation issue isn't any less tricky.

Except for a site import/transfer: MySQL's guard moves those errors to the import. If you try to import something and the collation is different from the source and that would lead to bizarre errors later, then you'll get user_email not unique at some point at import time.

I'm uncertain about whether to override the collation to BINARY. On one hand an override prevents people from following advice they don't understand and getting into trouble. On the other it removes the database's ability to handle strings handle data input on different keyboards equally when they look equal.

Trac ticket: Core-31992
See WordPress/wordpress-develop#5237

Replaces the legacy is_email() and sanitize_email() functions, augmenting them through instead by selecting email address support.

The new wp_email_address_support filter provides an opportunity for plugins to upgrade or downgrade email support depending on an individual site’s needs, while paving a path towards future support for Unicode email addresses by default. This patch downgrades default support to the updated ascii mode, which rejects Unicode addresses but applies updated parsing rules for better ecosystem integration.

Follow-up to: [32482].

cc: @arnt

[dmsnell]

There is some discussion about what to include in 7.1. What little testing has been performed revealed some inconsistencies when the configured SMTP server doesn’t support Unicode addresses. That’s not qualitatively different than when it’s unable to deliver specific email addresses, but applies at a class-level for all non-ASCII email addresses.

Still it is best to continue expanding testing, and I would like to see this remain in 7.1, even if we opt out of Unicode support unless a site specifically opts-in. I will try to summarize some of the broad considerations as this patch stands now, because the work which has been done so far addresses multiple issues.

A new email address class centralizes understanding about email addresses.

There are multiple places in the existing codebase where emails are validated or parsed ad-hoc. The new class provides a helpful place to call on to answer questions about email addresses comprehensibly. It’s also providing an opportunity for us to start distinguishing human-readable forms of email addresses vs. machine-readable forms, which is relevant to this ticket.

The class is a very helpful construct and I see no drawbacks to its inclusion. It provides a single place to keep updates to ensure that WordPress and web browsers speak of the same thing when discussing an email, with the benchmark being “what validates inside an <input type=email> field.

Updated email parsing for ASCII and for Unicode addresses.

WP_Email_Address::from_string() performs double-duty for is_email() and also for sanitize_email(). sanitize_email() should probably be deprecated entirely, as it’s current operation is purely to attempt ad-hoc email validation, returning an empty string for all emails it doesn’t accept, otherwise mangling subdomain labels it doesn’t accept.

Parsing an email indicates whether it’s valid and also provides a normalized form to be used throughout. Normalizing generally shouldn’t happen, but internationalized domain names (IDN) provide a single point of normalization that’s valuable: a domain should consist entirely of punycode ASCII-compatible encoded labels; or it should be encoded fully as UTF-8.

The new class adopts ​the email specification from the WHATWG living HTML standard. This is a change from legacy email parsing and will allow some addresses which were formerly rejected, rejecting some that were formerly allowed.

So this aspect, for the ASCII character set, is a backwards-compatibility change. However, it may not represent a backwards-compatible break. It could be that WordPress has been rejecting valid email addresses and allowing some which may only be submitted through the API, not through a browser.

The new class also parses Unicode addresses, but does so only when requested through its $character_set parameter. Most of the validation is shared between the two character sets, the vocabulary here aids comprehension of the differences between the two.

Adopting the new parsing would-likely bring WordPress‘ behavior closer to browsers and is probably worth doing anyway, as the story before 7.1 has been a bit rough. For instance, is_email() and sanitize_email() themselves apply divergent rules, meaning that code which trusts is_email() will end up surprised by sanitize_email(), and code trusting sanitize_email() will be surprised by is_email().

Here are some examples where the divergence is evident:

Username	is_email()	sanitize_email()
john smith@…	false	johnsmith@example.com
"john"@example.com	false	john@example.com
user@foo..bar.com	false	user@foobar.com
user@.example.com	false	user@example.com
user@…	false	user@example.com
user@…	false	user@example.com
user@foo.@@.com	false	user@foo.com
user@@example.com	false	user@example.com
josé@españa.es	false	jos@espaa.es
a @b.c	false	a@b.c

The point here is that the current situation is already pretty messy and already exhibits some of the challenges which are discussed with Unicode email addresses: there are existing pathways which conflate and impersonate email.

What adopting the merged and updated ASCII parsing would do is at least bring harmony inside of WordPress on answering the question: “is this an email address, and if so, which one is it?”

WordPress already partially supports Unicode email addresses

There’s broad confusion on what “Unicode email address” means and I think this can add friction to discussions about it. Email addresses can include non-ASCII characters in two distinct ways:

• a domain can be an Internationalized Domain Name (IDN) and represented through an ASCII-compatible encoding (ACE) — punycode. these are the domains we know that are like java@xn--53h.coffee. this is an ACE representation of java@☕️.coffee. these kinds of domains have been supported for a very long time, largely because they are simply ASCII domains and it’s up to the application to reinterpret the ASCII bytes into the Unicode form.
• a mailbox or local part may contain non-ASCII UTF-8 bytes as per RFC6530. this hasn’t been widely supported for a long time, but is supported today by most major email services. there is no ASCII-equivalent or ASCII-compatible form of conveying Unicode mailboxes.

WordPress has largely supported punycode-encoded IDNs through being agnostic about them. Unicode mailboxes are not the same because they are rejected by existing functions like is_email() and sanitize_email() and non-UTF-8 databases don’t support them.

but even still there are complications in pre-WordPress-7.1 code with these emails. WordPress will accept java@xn--53h.coffee and reject java@☕️.coffee even though they are the same email address, and so it will be unaware of equivalences that exist.

One way that WP_Email_Address helps this situation is that it can assert those equivalences and normalize addresses to one form or another.

The issues with database storage and retrieval are stuck for mailboxes because of the lack of a non-UTF-8 form, though that does help in other ways. For example, someone searching for me@naïve.com won’t find me@xn--nave-6pa.com in the database as-is. Storing the Unicode/UTF-8 version would find that.

Database collation plays an unfortunate role here.

Because the database performs its own Unicode processing based on the collation, it may declare two addresses equivalent in one situation and not in another. Usually this should be relatively static for a site, but technically an individual database query can change the collation.

WordPress would need to decide whether to enforce a byte-for-byte comparison or lean on the database rules to provide implicit conversion for addresses with sequences of bytes that collate differently on different locales. Whatever the choice is, we just need to follow-up and make sure that all email-interacting code uses the same means.

Deliverability is complicated

All I really have to say on this one is that I think we could potentially resolve the missing SMTPUTF8 support on the Site Health screen, and optionally allow people to send a test message to a special address, like emailing 📧@wordpress.org and provide feedback on whether the current server support this.

It’s not great to allow Unicode addresses that can’t receive invitation or verification emails, so that’s an important distinction from the other undeliverable that @agulbra mentioned; in this case, someone may have entered everything properly, and other email servers are properly configured, but the WordPress site is using one that isn’t. I’d like to know in practice the representation of production email servers that don’t support these addresses. It leads me to wonder if the expected sources of failures are test setups.

---

When this work started I was more anxious about what kinds of breakdown might occur, but I have seen a number of things clean up with more thought and assessment.

I think that I would be personally willing to open up full Unicode email support in 7.1, though I think that the new filter I’m working on in ​PR#12567 is essential before the full release, just to make it easier for plugins and site administrators to disable it.

I think that the new ASCII parsing is even more important to include because the legacy code is in such a peculiar state. In the PR I brought back the legacy parsing, but I think that’s a mistake I will undo before I actually propose merging the PR.

We have some testing on this but less than I hoped; or possibly we’ve had fewer failures, but my guess is just less testing. I’d appreciate hearing from you if you are aware of risks I’ve overlooked.

It was decided not to add Unicode email support to Core.
See ​https://make.wordpress.org/core/2026/05/22/extending-unicode-support-in-email-addresses-usernames-and-slugs/#comment-49045:

Because of the vast surface area Unicode introduces and the security implications we should keep core as it is, and put this work into a community plugin. This introduces a lot of complexity, and there are much more pressing issues our users face that we need to devote our time to.

This reverts r62482 / 2001ef14e6bada1575d2e8b144efff4ce9c01dab.

Trac ticket: https://core.trac.wordpress.org/ticket/31992
Follow-up to r62482.

[jonsurrell]

In 62829:

Revert [62482]: Add support for unicode email addresses.

This reverts [62482] based on this decision:

Because of the vast surface area Unicode introduces and the security implications we should keep core as it is, and put this work into a community plugin…

• ​https://make.wordpress.org/core/2026/05/22/extending-unicode-support-in-email-addresses-usernames-and-slugs/#comment-49045

Developed in ​https://github.com/WordPress/wordpress-develop/pull/12637.

Follow-up to [62482].

See #31992.

Merged in r62829.

This was reverted in r62829 ​based on this decision.

[jonsurrell]

​Based on this decision I think this ticket can be closed.

Because of the vast surface area Unicode introduces and the security implications we should keep core as it is, and put this work into a community plugin. This introduces a lot of complexity, and there are much more pressing issues our users face that we need to devote our time to.

[wildworks]

I will close this ticket as wontfix.