Add to the privacy tools UX a means to erase personal data by username or email address

[allendav]

Similar to #43546 but for GDPR Right to Erasure

Provides a means for an admin to erase personal data for a registered or no-priv user based on their username or email address.

Adds it to the appropriate place in the overall UX defined by #43481

Leverages the work done in #43438

Allows plugins to optionally surface controls to allow the admin to opt-out of erasing certain data (e.g. for legal or other reasons)

[allendav]

The ideal admin UX will also allow the admin to delete a registered user's account after erasing their personal data

[allendav]

Similar to #43546

Here’s the flow i’m imaginging:

• user contacts admin to request erasure of their data (can be by email, phone call, postal mail, etc)
• admin enters the user’s email address in the box near the erasure request table and hits request-verification button
• mike jolley’s magic code sends the verification email to the user
• user clicks the link in the email thus verifying the request
• admin sees a badge/dot/number on the wp-admin sidebar menu and goes to the erasure requests wp list table and sees the user verified the request
• admin clicks on the "erase personal data" action for the verified request
• a progress indicator is displayed for that row of the wp list table while erasure is in progress
• after erasure is complete, if the user was a registered user, admin is prompted whether they'd like to also delete the user's account
• if the admin consents, the user account is then deleted

[azaozz]

Replying to allendav:

We discussed this a bit more on Slack.

• user contacts admin to request erasure of their data (can be by email, phone call, postal mail, etc)

For registered users it makes sense to have a button on the Profile screen that would trigger the verification email.

For commenters (no-priv users) we may need to add some minimal UI to be able to trigger the verification email themselves from the front-end. Alternatively can leave these requests to be made through the site's contact form or other method.

• admin enters the user’s email address in the box near the erasure request table and hits request-verification button
• mike jolley’s magic code sends the verification email to the user
• user clicks the link in the email thus verifying the request
• admin sees a badge/dot/number on the wp-admin sidebar menu and goes to the erasure requests wp list table and sees the user verified the request
• admin clicks on the "erase personal data" action for the verified request
• a progress indicator is displayed for that row of the wp list table while erasure is in progress
• after erasure is complete, if the user was a registered user, admin is prompted whether they'd like to also delete the user's account

IMHO the erasure or anonymization should be done in one step here. It's simpler and easier to understand. If plugins store information that is optionally anonymized, they should have settings where the user can select what is overwritten/deleted. Thinking that we shouldn't be asking the admin to decide what to keep and what to delete every time.

As far as I see we are not going to delete comments or posts belonging to the user. The users can do that themselves before leaving the site. We will anonymize the comments, create new user and reassign the posts, then delete the old user account.

• if the admin consents, the user account is then deleted

This step should be automatic, see above.

[allendav]

Update: @mikejolley and I are working on this in a temporary repo here: ​https://github.com/allendav/wp-privacy-requests while we await @melchoyce 's designs

See also patch added today in #43442

[allendav]

Replying to azaozz:

Replying to allendav:

We discussed this a bit more on Slack.

• user contacts admin to request erasure of their data (can be by email, phone call, postal mail, etc)

For registered users it makes sense to have a button on the Profile screen that would trigger the verification email.

For commenters (no-priv users) we may need to add some minimal UI to be able to trigger the verification email themselves from the front-end. Alternatively can leave these requests to be made through the site's contact form or other method.

Could we make a non-admin UX a ticket separate from the admin UX please? It will be a fair bit of work itself.

[allendav]

Demo of personal data erasure

[allendav]

Work in progress

[allendav]

Work in progress. Actually erases personal data when tested with #43637 (erasure ajax support) and #43442 (comment erasure handler). Install all three patches to test.

Needs to surface additional messages from erasers and needs styling. Also needs to kick off registered user deletion after deleting personal data of a registered user.

cc @azaozz @mikejolley @melchoyce

[allendav]

Oops. Forgot to include the javascript in the patch

[azaozz]

43602.2.diff looks good. In 43602.3.diff only moved the js string localization to script-loader.

[desrosj]

Should there be a confirmation when the Remove Personal Data button is clicked?

[maxfein]

@desrosj seems like there should be a confirmation generally

...and I had been wondering how this works in a (multinetwork) multisite - does clicking 'Remove Personal Data' at one subsite effect only that subsite?

If doing this at any one network subsite will effect whole WP then, well, then imo there should be a very clear confirmation mssg if user exists at more than one subsite|network... maybe list the various sites?

[allendav]

Replying to maxfein:

@desrosj seems like there should be a confirmation generally

...and I had been wondering how this works in a (multinetwork) multisite - does clicking 'Remove Personal Data' at one subsite effect only that subsite?

If doing this at any one network subsite will effect whole WP then, well, then imo there should be a very clear confirmation mssg if user exists at more than one subsite|network... maybe list the various sites?

Yep - this is for a single site only. Perhaps a separate ticket for multisite?

[allendav]

Removed ajax-actions which was inadvertently included from 43637

[allendav]

Replying to azaozz:

43602.2.diff looks good. In 43602.3.diff only moved the js string localization to script-loader.

Thank you @azaozz - in 43603.4.diff I kept your change but removed ajax-actions from the patch (I had inadvertently included it earlier) - ajax-actions is already part of 43637's changes.

[allendav]

Replying to desrosj:

Should there be a confirmation when the Remove Personal Data button is clicked?

@melchoyce - what do you think?

[melchoyce]

Yeah, probably better safe than sorry here.

re: 43602-demo.gif​, the notice should appear inside the container — so, inside the blue, not underneath it. I know it's a bit notice-ception, but that's how it currently works in other areas of the admin (like plugins).

[ericdaams]

I am testing the latest patch alongside the latest ones from #43637 and #43442, and wanted to bring up a couple user experience/interface issues:

It feels odd to me that admin can go ahead and remove personal data without getting a user's confirmation. This seems likely to lead to mistakes on the part of admin accidentally removing the personal data.

On the flipside, once a user has confirmed their request, the data removal link is gone, replaced by the button on the right. There are visual issues with the button on smallish screens.

(Nevermind that stray arrow. Too lazy to fix the screenshot. :))

[ericdaams]

Registered exporters are required to return an array containing the number of items removed, the number retained and an array of messages. However, the number of items removed/retained are never actually used client-side in any helpful way. I would suggest instead allowing exporters to return simple boolean values for removed_items, retained_items and done.

i.e. :

return array(
    'removed_items' => true,
    'retained_items' => true,
    'done' => true,
);

This simplifies plugin developers' jobs while satisfying all the needs I currently see in xfn.js for how the returned values are handled. It also avoids doubt about what is supposed to count as an "item".

Example: If I scrub a donation its personal data, including a set of 10 donation meta items that are part of that donation, have I just removed one item or eleven? Does it matter?

#43442 is relevant to this discussion too.

[azaozz]

It feels odd to me that admin can go ahead and remove personal data without getting a user's confirmation.

This is the case now too. Admin is "the boss" and can do... everything on the site, even delete the whole site :)

[desrosj]

Moving to the 4.9.6 milestone after consensus was reached in the most recent GDPR chat (​https://wordpress.slack.com/archives/C9695RJBW/p1524063200000304).

[coreymckrill]

Change required capability to delete_users

[coreymckrill]

In 43602.5.diff​ I changed the required capability for accessing the data removal page to delete_users. This is a capability that Admins have on single sites, but only Super Admins have on multisites. Given that the procedure here affects the user tables, which are shared across all sites in a network, it seems like this should only be available to Super Admins in a multisite context.

[azaozz]

In 42986:

Privacy: add means to erase personal data by username or email address. First run.

Props allendav, coreymckrill, ericdaams, azaozz.
See #43637, #43602.

[swissspidy]

Stuff like <a href="#" > and <a class="button" href="#" > really goes against our accessibility standards.

See ​https://make.wordpress.org/core/handbook/best-practices/coding-standards/accessibility-coding-standards/.

[azaozz]

Replying to swissspidy:

This is the "first run" that is committed primarily to make the functionality easier to test. Enhancements are not only desirable but also highly appreciated :)

[azaozz]

Replying to coreymckrill:

I changed the required capability for accessing the data removal page to delete_users. This is a capability that Admins have on single sites, but only Super Admins have on multisites.

Right. This should work quite differently on multisite. Seems the user should only be "removed" from the current website. If the user is not a member of any other website, then the data can be deleted completely, perhaps even without SA cap?

Super admins will also need some more UI to be able to choose if the user should only be removed from one site or deleted completely.

[mikejolley]

@allendav This will obviously break what we have now, but I notice between this and the exporter we have eraser_friendly_name and exporter_friendly_name.

Why don't we unify the format here and just use nicename for both?

[xkon]

After doing some back and forth tests after #43442 got commited I was confused on this and I've got a question / proposal.

What I did:

• I went on my dev site and spamed some comments as a visitor with mail visitor@dev.oo.

• I went from the admin and made a custom request to remove data to the visitors email visitor@dev.oo and confirmed that request.

• Went back on the admin and I removed visitor@dev.oo data.

• I went again and left yet another comment as visitor@dev.oo.

• I still had access from the admin since that mail had already made a confirmation to remove his data without the need of a NEW confirmation.

Proposal ( if we haven't already talked about it ):

After removing a users data, replace the button with something like 'Data removed on dd-mm-yy' this acts as:

• a reminder for the Admin when that action took place
• you can't use the same action without confirmation ( if you don't want to create a newer entry in the future )
• a page being less confusing
• needing a confirmation from scratch to remove a persons data with the same e-mail for safety reasons

[mikejolley]

@allendav

I'm not sure if this is working or if it's my code.

No personal data was found for this user.

This is the message I'm seeing in admin, but I can see my erasure code is returning results:

​https://dl.dropboxusercontent.com/s%2Fe4zbt1qvm85r9fh%2FSlack%2520-%2520A8C%25202018-04-20%252016-46-09.png

Code ​https://github.com/woocommerce/woocommerce/pull/19808/files#diff-3d2ec5172ea5ae5cd9b4cecaa433f507R163

[birgire]

Just an idea into the cosmos:

After working with the Export Personal Data table for some time, I could feel the urge to have the "Add new request" form hidden from view, when working with the requests. So export-personal-data-with-add-request-link.jpg​ is what came first to mind, as it is very familiar layout for such a table ;-)

---

It's also interesting idea from @azaozz (don't remember the ticket) to consider storing some data in the post table fields. That might give better performance for very large tables, when sorting or searching, instead of meta queries. But maybe very large request tables will be an edge case?

[xkon]

Oh nice thinking @birgire ! I really like how it goes along the lines of the already usual Add New button we normally find there. Will there be a proposed patch as well to see how it looks like in action maybe :) ?

[birgire]

@xkon Thanks, this kind of "Add New" button usually takes us to a new page, like we see on the plugin/theme/post screens. Would that be an option here?

[xkon]

I don't think there's a point of having a new page just for a single input really, we could just toggle() a div containing the Request form imho but that's for @melchoyce better.

Still the button feels like home there even if you don't get redirected as normally :) which is really nice.

[allendav]

Doc for how plugins can hook into the personal data eraser

[allendav]

Replying to mikejolley:

@allendav

I'm not sure if this is working or if it's my code.

No personal data was found for this user.

This is the message I'm seeing in admin, but I can see my erasure code is returning results:

@mikejolley - sorry about that - i've tracked it down to xfn.js:

retainedCount += responseData.num_items_removed;

should be

retainedCount += responseData.num_items_retained;

I also need to update the patch to actually add any messages returned to the DOM

[allendav]

Fix retention count bug that could cause the UX to say no personal data was found when in fact data was retained

[melchoyce]

If we think folks won't be using this manual request very often, we could hide it behind an "Add New" toggle like on the Media screen, like @xkon suggested. Is that code reusable?

[xkon]

We could make it code reusable but the 'manual request' was something that I forgot.

Good point @melchoyce , afaic all the requests are for the time being manually made after a Contact form e-mail type of thing to the admin so he can then go start the confirmation process (except if I missed something). So an extra 'step' to that would be not good at this moment I suppose :D.

We can keep the idea for the future though if we go forth automating it fully.

[ericdaams]

Replying to ericdaams:

Registered exporters are required to return an array containing the number of items removed, the number retained and an array of messages. However, the number of items removed/retained are never actually used client-side in any helpful way. I would suggest instead allowing exporters to return simple boolean values for removed_items, retained_items and done.

i.e. :

return array(
    'removed_items' => true,
    'retained_items' => true,
    'done' => true,
);

This simplifies plugin developers' jobs while satisfying all the needs I currently see in xfn.js for how the returned values are handled. It also avoids doubt about what is supposed to count as an "item".

Example: If I scrub a donation its personal data, including a set of 10 donation meta items that are part of that donation, have I just removed one item or eleven? Does it matter?

#43442 is relevant to this discussion too.

@allendav Is there merit to simplifying the return values for plugins to just say whether any items were removed, or retained? I am happy to put together a patch.

[ericdaams]

Simplify return values for data erasers

[ericdaams]

Updated docs for what plugin erasers should return.

[birgire]

@ericdaams

I wonder if we should use erased instead of removed in general?

Example:

'items_removed'     -> 'items_erased'
'num_items_removed' -> 'num_items_erased'

because e.g. the summary of the wp_comments_personal_data_eraser() function is:

 * Erases personal data associated with an email address from the comments table.

ps: I noticed you changed from ++ to += 1. If I remember correctly I used ++ here because the general (not core) WordPress Code Standard seemed to prefer the ++ case:

Increment operators should be used where possible; found "$num_items_removed += 1;" but expected "$num_items_removed++"

[allendav]

@ericdaams - instead of having $num_items_removed in src/wp-includes/comment.php at all, I'd prefer if we just had a flag that was set to true on if ( $updated ) - what do you think?

[ericdaams]

Replying to allendav:

@ericdaams - instead of having $num_items_removed in src/wp-includes/comment.php at all, I'd prefer if we just had a flag that was set to true on if ( $updated ) - what do you think?

That's what I was going to do originally, but that marginally complicates how we track whether any items were retained. But fine to do it this way, as that is more helpful to plugin developers looking to this function as an example of how to create their own eraser. Patch coming.

[ericdaams]

Update comment erasing tests. Avoid counting number of comments erased.

[ericdaams]

Change "remove" to "erase"

[ericdaams]

Replying to birgire:

@ericdaams

I wonder if we should use erased instead of removed in general?

Example:

'items_removed'     -> 'items_erased'
'num_items_removed' -> 'num_items_erased'

because e.g. the summary of the wp_comments_personal_data_eraser() function is:

 * Erases personal data associated with an email address from the comments table.

I agree that erased makes more sense. removed is used all over the place though, both in the code and in the UI. See above screenshot showing the updated tools page using "erase" instead of "remove".

[xkon]

Just commenting here in case it's missed on slack.

Since GDPR is calling it Right to erasure let's go with erasure everywhere imho. That's why the function names etc are using erasure as well (I remember the original chat with @allendav we had about naming things for ease of use).

[birgire]

I think it would be good for consistency.

Thanks for the screenshot @ericdaams

ps:

I can hear the voice of Arnold Schwarzenegger: You've just been erased. ;-)

... maybe a plugin idea for the more adventurous ;-)

[mnelson4]

I tested this and am not sure what to expect. Here's what happened:

-using the develop branch, I pulled the latest

-I applied the patch https://core.trac.wordpress.org/attachment/ticket/43602/43602.7.diff (only that one- I'm not sure if I should apply other patches too... sorry, but I'm not sure if I should apply ALL patches or just the latest)

-I initiated a data erasure request from the admin,

-received the email to confirm data erasure

-clicked the link to confirm data erasure

-as site admin, confirmed "Remove Personal Data"

-some of the user's comments were removed, and some had the commenter renamed to "Anonymous"

-the WP user was not removed or anonymized (this ticket's description gave me the impression it should have)

[mnelson4]

https://core.trac.wordpress.org/attachment/ticket/43602/ERASURE.2.md#L8

has this

Among the tools added is a Personal Data Removal tool which supports erasing/anonymizing personal data for a given user in a ZIP file.

The words "in a ZIP file" seem wrong and was copy-and-pasted from the data exporter documentation.

Also, I'm confused by this text in the original description:

Allows plugins to optionally surface controls to allow the admin to opt-out of erasing certain data (e.g. for legal or other reasons)

This doesn't actually facilitate plugins adding controls that allow the admin to control what data gets erased, right? (If it does, I didn't see it mentioned in https://core.trac.wordpress.org/attachment/ticket/43602/ERASURE.2.md#L8). It just allows plugins' to hook in their own data erasers. If plugins want to ask admins what data to erase, it's up to plugins to present those options to the user. Is there a recommended way for plugins to interrupt the normal data erasure AJAX request to ask users about their preferences?

[ericdaams]

@mnelson4 I'm not sure if this is the recommended way, but in Charitable (​https://github.com/Charitable/Charitable) the approach I am taking is to have additional plugin settings that allow users to define what happens when data is erased. This gives them plenty of control and saves them having to check the same boxes every time they need to erase a user.

Screenshot of our User Privacy settings: ​https://www.dropbox.com/s/l8uzbneawcdcbmt/Screenshot%202018-05-01%2013.17.22.png?dl=0

[azaozz]

Replying to ericdaams:

@mnelson4 I'm not sure if this is the recommended way, but the approach I am taking is to have additional plugin settings that allow users to define what happens when data is erased.

Yep, that's the right way to do things as we discussed in Slack few weeks ago. The plugins "know" what data they store and what needs erasing, and if the admin should be able to control what gets erased.

[allendav]

@mnelson4 - we weren't going to automatically delete the user too - that is still a separate action for an admin to take after erasing their personal data. The rationale is that a plugin may say "no" to any particular erasing and then the admin can make the judgement call on whether or not that means they can erase the registered user account or not.

I'll fix the ZIP mention in the markdown doc - that was a copy and paste error.

[allendav]

Replying to mnelson4:

This doesn't actually facilitate plugins adding controls that allow the admin to control what data gets erased, right? (If it does, I didn't see it mentioned in https://core.trac.wordpress.org/attachment/ticket/43602/ERASURE.2.md#L8). It just allows plugins' to hook in their own data erasers. If plugins want to ask admins what data to erase, it's up to plugins to present those options to the user. Is there a recommended way for plugins to interrupt the normal data erasure AJAX request to ask users about their preferences?

That is correct - this code doesn't facilitate any plugin controls. The intention for now is that plugins may choose to expose erasure controls in their own settings user interfaces where it makes the most contextual sense. Plugins don't need to interrupt the erasure AJAX request to get those preferences - they should be set well in advance of any erasures. All a plugin needs to do, when an erasure happens, is respond to the erasure page calls for their eraser(s) and use the settings (if any) that they have to decide how much / how little should be erased for a given item.

A good example is a WooCommerce order. WooCommerce's settings could say that personal data should be retained for orders < XXX days old. The WooCommerce order eraser could look at that setting against each order and decide whether or not to remove it. If it decides not to remove it, it can emit a message in the AJAX to say as much to the admin (i.e. Order 1234 was not erased because it is less than XXX days old.)

[allendav]

@azaozz @mikejolley @ericdaams - we still need to either commit the bugfix in 43602.retention.count.bug.diff or move forward with @ericdaams 's 43602.7.diff that boolean-izes the retained and removed counts in the erasure response.

I'd like us to do 43602.7.diff - but we'll need to make sure to also communicate this change since it changes plugin's eraser response shape.

[allendav]

Removes ZIP reference (copy and paste error), gives guidance on how plugins can control erasure, fixes 0 in example response to boolean false to match new response shape

[azaozz]

In 43061:

Privacy: improve wp_privacy_erase_personal_data(), return boolean values.

Props ericdaams.
See #43602.

[allendav]

I added ERASURE.3.md just now:

@ericdaams - I fixed a 0 in the example response to make it false since we're going to do flags now instead of counts

@mnelson4 - I removed that ZIP reference, clarified that user account deletion is separate, and copied those comments above into the doc as guidance to plugin devs that want to control how little or much an eraser takes.

[SergeyBiryukov]

In 43075:

Privacy: add means to erase personal data by username or email address. First run.

Props allendav, coreymckrill, ericdaams, azaozz.
Merges [42986] to the 4.9 branch.
See #43637, #43602.

[mnelson4]

Replying to allendav:

I'd like us to do 43602.7.diff - but we'll need to make sure to also communicate this change since it changes plugin's eraser response shape.

+1 as a plugin dev currently working on integrating with this. I assume core code is in flux until release

[SergeyBiryukov]

In 43110:

Privacy: improve wp_privacy_erase_personal_data(), return boolean values.

Props ericdaams.
Merges [43061] to the 4.9 branch.
See #43602.

[desrosj]

Related: #43923.

[allendav]

Updated to show registration using a key instead of a simple numeric array; removed copy paste error (unused export variable)

[azaozz]

Set priority to low for tickets that only need unit tests.

[desrosj]

Moving to the new Privacy component.

[casiepa]

Moving enhancement to 4.9.8

[desrosj]

Creating a new ticket helps keep the enhancement that was added with the 4.9.6 milestone and makes it clearer when tests were committed, and avoids continuously punting this ticket.

I opened #44234 and moved the uncommitted patches over to that ticket to wrap up and add other missing tests.