Make WordPress Core

Opened 6 years ago

Closed 5 years ago

Last modified 4 years ago

#43797 closed enhancement (maybelater)

Logging for GDPR privacy/security

Reported by: xkon's profile xkon Owned by: xkon's profile xkon
Milestone: Priority: normal
Severity: normal Version: 4.9.6
Component: Privacy Keywords: privacy-roadmap feature-plugin
Focuses: Cc:

Description

After a little chat in #gdpr-compliance as some questions where raised https://ico.org.uk/for-organisations/guide-to-the-general-data-protection-regulation-gdpr/lawful-basis-for-processing/consent/ was presented.

It seems that we have to create a mechanism so plugins/themes can tap into and record any consents so the admins can keep a proper log of them for later use if needed.

I think it would be prefered if it's somewhere centralized again instead of each plugin keeping it's own log.

Attachments (1)

43797.diff (11.5 KB) - added by xkon 6 years ago.
introducing consent log tool

Download all attachments as: .zip

Change History (38)

This ticket was mentioned in Slack in #gdpr-compliance by xkon. View the logs.


6 years ago

#2 @Shelob9
6 years ago

I'm the author of Caldera Forms, a fairly popular plugin on WordPress.org for making forms. [We are working on adding a field to consent to personally identifying](https://github.com/CalderaWP/Caldera-Forms/issues/2428). It would be very helpful to me, if I had an API like this:

<?php
//$consent_api contains instance of class that serves as API for this info

//Check if we have consent, by email address
$has_consent = $consent_api->has_given_consent( 'hi@hiroy.club' );

//Record consent by email address
$consent_api->add_consent( 'hi@hiroy.club' );

//Remove consent by email address
$consent_api->remove_consent( 'hi@hiroy.club' );

This ticket was mentioned in Slack in #gdpr-compliance by shelob9. View the logs.


6 years ago

#4 @mnelson4
6 years ago

The “WP GDPR framework” already has an implementation of this that sounds pretty good: https://codelight.eu/wordpress-gdpr-framework/developer-docs/ (scroll to “consent”).
They mention functions

<?php
gdpr('consent')->register(...)

(so you can register meta information about the consent, like a pretty name, so that data doesn’t need to be repeated),

<?php
$dataSubject = gdpr('data-subject')->getByEmail($_POST['email']);
  $dataSubject->giveConsent('my_custom_consent_slug'); 

(because not all data subjects will be users, and grants specific consent), and a filter for handling what to do when consent is revoked.
I haven’t used it myself, but thought I’d mention the prior art.

#5 @xkon
6 years ago

I've been doing something similar with a website that has custom forms / cookies etc, so I thought of converting the code a little bit to see if it would help here at all. I went along the lines of how the requests CPTs where created for the privacy tools but I'm not sure if there's a better/clever way to do it for core. I have a custom admin page that lists all of the consents so the admin can easily who has done what also.

Ofc this might be pretty basic but it has served the websites needs pretty well until now.

I can add this on the user.php that we currently have all of our extra classes if this is ok for a start so we can check / enhance it better.

<?php
class WP_Privacy_Consent_Logs {

        public function consent_exists( $attr ) {

                $args = array(
                        'post_type'  => 'consent_log',
                        'meta_query' => array(
                                'relation'     => 'AND',
                                '_user_email' => array(
                                        'key'   => '_user_email',
                                        'value' => $attr['email_address'],
                                ),
                                '_consent_identifier' => array(
                                        'key'   => '_consent_identifier',
                                        'value' => $attr['identifier'],
                                ),
                        ),
                );

                $query = new WP_Query( $args );

                if ( $query->have_posts() ) {
                        return $query->post->ID;
                } else {
                        return false;
                }

        }

        public function has_consent( $attr ) {

                $exists = $this->consent_exists( array(
                        'email_address' => $attr['email_address'],
                        'identifier'    => $attr['identifier'],
                ));

                if ( $exists ) {
                        if ( 'yes' === get_post_meta( $exists ,'_consent_status', true ) ) {
                                return true;
                        }
                } else {
                        return false;
                }
        }

        public function add_consent( $attr ) {

                $exists = $this->consent_exists( array(
                        'email_address' => $attr['email_address'],
                        'identifier'    => $attr['identifier'],
                ));

                if ( ! $exists ) {

                        $user_id = 0;

                        $consent = wp_insert_post( array(
                                'post_author'   => $user_id,
                                'post_status'   => 'publish',
                                'post_type'     => 'consent_log',
                                'post_date'     => current_time( 'mysql', false ),
                                'post_date_gmt' => current_time( 'mysql', true ),
                        ), true );

                        update_post_meta( $consent, '_user_email', $attr['email_address'] );
                        update_post_meta( $consent, '_consent_identifier', $attr['identifier'] );
                        update_post_meta( $consent, '_consent_status', $attr['accepted'] );

                        return true;

                } else {
                        return false;
                }

        }

        public function remove_consent( $attr ) {

                $exists = $this->consent_exists( array(
                        'email_address' => $attr['email_address'],
                        'identifier'    => $attr['identifier'],
                ));

                if ( $exists ) {
                        wp_delete_post( $exists );
                        return true;
                } else {
                        return false;
                }

        }

        public function update_consent( $attr ) {

                if ( ! empty( $attr['accepted'] ) ) {
                        $exists = $this->consent_exists( array(
                                'email_address' => $attr['email_address'],
                                'identifier'    => $attr['identifier'],
                        ));

                        if ( $exists ) {
                                update_post_meta( $exists, '_consent_status', $attr['accepted'] );
                                return true;
                        } else {
                                return false;
                        }
                }

        }

}

I'm using it this way pretty much:

<?php
// consent_exists 
$consent = new WP_Privacy_Consent_Logs();
$args = array(
        'email_address' => 'test@test.test',
        'identifier'    => 'cookie_form_1',
);

$check = $consent->consent_exists( $args );

if ( $check ) {
        error_log( 'exists' );
} else {
        error_log( 'exists not' );
}

// has_consent 
$consent = new WP_Privacy_Consent_Logs();
$args = array(
        'email_address' => 'test@test.test',
        'identifier'    => 'cookie_form_1',
);

$check = $consent->has_consent( $args );

if ( $check ) {
        error_log( 'has' );
} else {
        error_log( 'has not' );
}


// add_consent
$consent = new WP_Privacy_Consent_Logs();
$args = array(
        'email_address' => 'test@test.test',
        'identifier'    => 'cookie_form_1',
        'accepted'      => 'no',
);

$check = $consent->add_consent( $args );

if ( $check ) {
        error_log( 'added' );
} else {
        error_log( 'not added' );
}


// remove_consent
$consent = new WP_Privacy_Consent_Logs();
$args = array(
        'email_address' => 'test@test.test',
        'identifier'    => 'cookie_form_1',
);

$check = $consent->remove_consent( $args );

if ( $check ) {
        error_log( 'removed' );
} else {
        error_log( 'not removed' );
}

// update_consent
$consent = new WP_Privacy_Consent_Logs();
$args = array(
        'email_address' => 'test@test.test',
        'identifier'    => 'cookie_form_1',
        'accepted'      => 'yes',
);

$check = $consent->update_consent( $args );

if ( $check ) {
        error_log( 'updated' );
} else {
        error_log( 'not updated' );
}
Last edited 6 years ago by xkon (previous) (diff)

This ticket was mentioned in Slack in #gdpr-compliance by xkon. View the logs.


6 years ago

#7 @mnelson4
6 years ago

Yeah @xkon I think that satisfies the requirement. Personally I'd be inclined to have two arguments for the functions, the first being an email and the second being the consent's identifier. That would be clearer than providing an array IMO.

Using the posts table seems a bit heavy-handed because this data isn't very post-y (most columns are unused, and all the important data needs to be stored separately), but I don't have any better suggestions than using a custom table. Usermeta won't work because we need to track consent for non-users (eg commenters). Storing this info as a option seems barbaric.

TBH I'd be most happy with a custom table but I realize that has a lot of migration work etc.

#8 @xkon
6 years ago

Thanks for your input @mnelson4 I've actually added a small repo and converted this into a plugin so it's easier for anyone that wanted to have a look to see how this feels or expand it for the ease of use ( https://github.com/mrxkon/consent-log ).

As for the custom table, I left it as is with a CPT since we used CPTs for the exports other tools also for now.

An extra note after a little bit of discussion the 'feeling' about this is that it might get abused by plugins overchecking consents ( although each plugin should add a unique ID per consent so there shouldn't be a problem there as any other plugin wouldn't know where to look ) + the idea of keeping 'extra' data such as e-mails etc.

From one hand you allow the user to get anonymized and from the other you have to keep his consent log to prove that at some past point he had accepted a consent, if the consent is anonymized as well how are you going to prove it in the first place? It seems that some things here are going against each other if I have it down correctly.

If we find a way to rule out the possibility of doing something actually 'against' the regulation so we can be safe and this is something that more people actually need we can surely check it out for the future.

---

Since my position is also conflicting ( by giving code and saying to wait a bit longer ), I'll say again that I had made something similar to this recently for a client as it was requested by his lawyers. It doesn't mean that even they are 100% correct though :) .

#9 @desrosj
6 years ago

  • Owner set to xkon
  • Status changed from new to assigned

#10 @mnelson4
6 years ago

As for the custom table, I left it as is with a CPT since we used CPTs for the exports other tools also for now.

Ya I see this follows that precedent, and that's probably fine.

And I'll try to take the plugin for a spin.

@xkon
6 years ago

introducing consent log tool

#11 follow-up: @xkon
6 years ago

  • Keywords 2nd-opinion has-patch added

After back & forth discussions about this, the opinions are somewhat in the middle of if this should be in core or not.

I'm in favor of trying to provide as much as possible built-in so we know that the Admins have a single page to view for their proof of consents, even if core itself doesn't actually need it on it's own.

In 43797.diff I've made a first pass of moving the plugin into core the same way as the other privacy tools are handled at the moment.

  • This creates a new consent_log CPT that can be used pretty much to log anything ( not only consents to be honest ).
  • It adds 2 new post statuses consent_accepted - consent_declined
  • It adds a Consent Log page under Tools with the option to remove all consents for a certain User by the admin.

There are 4 functions requires 3 things:

User ID: you can place here whatever, either email, id, or any other identifier that fits your needs.

Consent ID: This should be unique as a prevention of plugins overriding each other and taping into / reading different consents to avoid making extra ones. This would be considered misuse. The unique consent ID will be provided on any given form as a random maybe and it will be the plugins/themes/authors responsibility to keep track of so it can re-check it.

Consent Status: either consent_accepted, or consent_declined

<?php
$the_consent = new Consent_Log();

$consent = $the_consent->has_consent( 'test@test.oo', 'unique_form_id_1' );

$consent = $the_consent->create_consent( 'test@test.oo', 'unique_form_id_1', 'consent_accepted' );

$consent = $the_consent->update_consent( 'test@test.oo', 'unique_form_id_1', 'consent_declined' );

$consent = $the_consent->remove_consent( 'test@test.oo', 'unique_form_id_1' );

I'm sure there are more stuff to add / fix / alter but I think this is a decent start for this kind of Tool.

This ticket was mentioned in Slack in #core by allendav. View the logs.


6 years ago

#13 in reply to: ↑ 11 ; follow-up: @ericdaams
6 years ago

Storing consents in the posts table seems fraught with peril to me, if I understand the purpose behind consents correctly.

For example, if I have a cookie notice that is displayed to all visitors, and this is agreed to by the visitor, would this be something that I could/should store a consent for?

If that's the case, then every single visitor to my website who accepts this will end up having a consent entry created for them; that's a whole lot of bulk added to the posts table.

I understand that there is precedent for doing things as CPTs, say with requests for data erasure or export, but those requests are likely to be far fewer; consents is something that can very likely be hit again and again.

On the topic of developing this in core vs. as a plugin, would this be an appropriate candidate for a feature plugin? That might allow us to develop it in a less rushed fashion, with a custom table designed just for it.

#14 in reply to: ↑ 13 @xkon
6 years ago

Replying to ericdaams (also adding general things discussed in slack so we can have them here as well):

Storing consents in the posts table seems fraught with peril to me, if I understand the purpose behind consents correctly.
I understand that there is precedent for doing things as CPTs, say with requests for data erasure or export, but those requests are likely to be far fewer; consents is something that can very likely be hit again and again.

This first patch was made from a plugin that I already made / using for my clients as mentioned above as it was using CPTs already. One of the websites already has over 3k consents within a matter of days logged and they're moving up so I can easily see the concerns on this.

I pretty much took some time to match it with our current way of providing tools so you could just hit apply patch and check how things work ( at least for me :) ) the last weeks after I saw that @Shelob9 and @mnelson4 mentioned pretty much the same stuff that I was already using.

For example, if I have a cookie notice that is displayed to all visitors, and this is agreed to by the visitor, would this be something that I could/should store a consent for?

Depends on if you want proof for it or yes you might as well log everything.

If that's the case, then every single visitor to my website who accepts this will end up having a consent entry created for them; that's a whole lot of bulk added to the posts table.

It is but you can avoid logging if you want. It's not 'automated' and won't be (by core) in any way for the consent part at least. We provide the means to do it, but it's up to any Admin to use them or not.

On the topic of developing this in core vs. as a plugin, would this be an appropriate candidate for a feature plugin? That might allow us to develop it in a less rushed fashion, with a custom table designed just for it.

So there have been some chats in slack already and the thinking around this is to make it even more global to log more things and not only Consents. That being said the records will be even more and the logging will be 'endless' but that's how logs work + that's up to any admin to decide for how long the records will be kept for.

This is in general something that I will continue to try and have it in Core as it's apparently needed from the regulation + a privacy/security measure that the Core itself can easily provide with an api that everybody can use and follow.

Examples that have been mentioned:

  • Consents
  • User Logs (Registration/Profile Updates/Deletion)
  • Privacy Tools Logs ( to port them here for centralization )
  • Page/Post Editing ( User/Date -> pretty much like the revision table is viewed inside an Edit page at the moment, but again this will just be a mention + centralized for a quick review )

As you can imagine the list might even get bigger/smaller etc it's still on early discussions.

#15 @xkon
6 years ago

  • Summary changed from Consent Logging to Logging for GDPR privacy/security

This ticket was mentioned in Slack in #gdpr-compliance by eric.daams. View the logs.


6 years ago

This ticket was mentioned in Slack in #gdpr-compliance by allendav. View the logs.


6 years ago

This ticket was mentioned in Slack in #gdpr-compliance by brento. View the logs.


6 years ago

This ticket was mentioned in Slack in #gdpr-compliance by allendav. View the logs.


6 years ago

This ticket was mentioned in Slack in #gdpr-compliance by allendav. View the logs.


6 years ago

This ticket was mentioned in Slack in #gdpr-compliance by desrosj. View the logs.


6 years ago

#22 @azaozz
6 years ago

IMHO the first step here is to not let admins (or anybody else) remove/delete any of the privacy "requests", just like post revisions are not removable. They are the current log, both completed and failed.

If somebody don't want to use them in their logging/audit trail capacity, they can add a plugin that will let them delete these requests / CPT posts. Same case like post revisions :)

Last edited 6 years ago by azaozz (previous) (diff)

#23 @desrosj
6 years ago

  • Component changed from General to Privacy

Moving to the new Privacy component.

#24 @stiofansisland
6 years ago

I think this really needs to be in core.

I think it needs it's own table so not to clutter the posts table.

Whats the chances of this being in core anytime soon? If not can we start working on a featured plugin for this? (i would like to help)

#25 @xkon
6 years ago

@azaozz yeah, none of the log entries should be deletable (or at least not 'easily' deletable). Also another thinking of a more concrete 'logging' would be to not update a record but just add a new one.

So each 'action' ( either consent, export, erasure etc ) is kept into it's own record that would give you a better timeline of what happened as well.

Now that we have the release live I'll be adding some more things as well as they're needed around here (in the office) and also change some minor things on the previous patch just for the sake of previewing.

I'll try to get an update for a more complete 'log-type' of list asap to see what we're missing and how to get everything going.

This ticket was mentioned in Slack in #gdpr-compliance by stiofansisland. View the logs.


6 years ago

This ticket was mentioned in Slack in #gdpr-compliance by desrosj. View the logs.


6 years ago

#28 @desrosj
6 years ago

Related: #44043.

#29 @desrosj
6 years ago

  • Keywords privacy-roadmap added

#30 @dejliglama
6 years ago

An update on where we're headed in a test setup internally.


Step 1

Pseudo coding a bit here :

Log_function (email, action, status);
3 strings are required, e-mail being a unique identifyer, action is the hook that has been run, and status is a string detailing the intend and status of the action (remove, delete, revoke, failed, deleted... and so on).

The email is hashed into a key - this makes the log entry anonymized
A date for the entry is also stored.

-----

Step 2

Fetch_log_Function (hash, date-range);
The function takes a hashed value (so an e-mail, that is hashed, so that we can match it in the log).
Date start and Date end.

The function returns an array of entries within that daterange where there is a match on the hash.
Sort by date


Step 3

  • Save it somewhere outside of WP database - since it cant be a part of the database it's trying to govern.

#31 @xkon
5 years ago

@dejliglama do you have any 'internal' updates maybe on this? I'm not sure what point you have reached -or- if you want to start sharing code thoughts on this so I and others can jump in again.

I'm still on the side of keeping the log entries in the database as it would be easier for backups to get handled by backups etc and since the entries would be hashed already there shouldn't be any issues whatsoever. Maybe an 'extra' action to send them over to the admin by an option would be better if they want an extra backup maybe or record keeping functionality maybe?

In any case inform us when possible where you're at so we could move this forward a bit and re-start the chats in slack as well about how it should be done eventually.

Mentioning this here since we had some DMs about this on Slack some time ago so everybody can get up to speed as well.

#32 @dejliglama
5 years ago

Sorry for the delay @xkon

I've been debating the issue of GDPR activity logging with my colleagues, and although we do see the need as we previously described in this ticket. The idea of having the log IN WordPress, and inside the same database/ filesystem as the data it's to keep a log over, seems to be a bad idea.

We're currently working on an external solution based on Logstach which will probably be baked into a Plugin that will hook into any identifiable GDPR related actions within WordPress. To be able to store anonymous data outside of the WP install.

In WP terms, I believe that leaves this ticket as a stub, and perhaps it's more important for WP core to be able to help clearly identify which activities within WP that can be deemed GDPR activity...

#33 @garrett-eclipse
5 years ago

  • Keywords needs-refresh added; gdpr removed
  • Version set to 4.9.6

#34 @xkon
5 years ago

  • Milestone Awaiting Review deleted
  • Resolution set to maybelater
  • Status changed from assigned to closed

All future discussions and maintenance until this has a solid proposal for core will be made on it's ( really soon to come :) ) repo at https://github.com/wordpress-privacy .

I'm marking this as a maybelater to keep our lists here clean as well and we'll either re-open or start fresh when it's time.

Thank you all for the comments and suggestions here!

#35 @garrett-eclipse
5 years ago

  • Keywords feature-plugin added; 2nd-opinion has-patch needs-refresh removed

This ticket was mentioned in Slack in #core-privacy by xkon. View the logs.


4 years ago

Note: See TracTickets for help on using tickets.