WordPress.org

Make WordPress Core

Opened 3 years ago

Closed 7 months ago

Last modified 3 months ago

#37820 closed defect (bug) (reported-upstream)

wp_remote_get referrer not being sent correctly

Reported by: schrapel Owned by:
Milestone: Priority: normal
Severity: normal Version: 4.6
Component: HTTP API Keywords: has-patch
Focuses: Cc:
PR Number:

Description

We use an API that would check the referrer url. In the process of upgrading to 4.6 their API now responds telling us an invalid referrer was provided. If I roll back to 4.5.3 and run wp_remote_get it works without problems.

Attachments (4)

patch_37820.diff (662 bytes) - added by tristangemus 3 years ago.
Screen Shot 2016-10-03 at 3.32.05 PM.png (41.0 KB) - added by tristangemus 3 years ago.
Screen Shot 2016-10-03 at 3.30.41 PM.png (47.4 KB) - added by tristangemus 3 years ago.
referer.png (14.0 KB) - added by differentthemes 3 years ago.
Referer header

Download all attachments as: .zip

Change History (20)

#1 @swissspidy
3 years ago

Hey there, thanks for your report!

What kind of arguments are you passing to wp_remote_get()? Some sample code to look at would help greatly.

Looking at the code from 4.5 and 4.6, no referer was ever set by WordPress in 4.5. Now with 4.6, Requests seems to set the referer to the URL being requested:

curl_setopt($this->handle, CURLOPT_URL, $url);
curl_setopt($this->handle, CURLOPT_REFERER, $url);

Citing RFC 2616:

The Referer[sic] request-header field allows the client to specify, for the server's benefit, the address (URI) of the resource from which the Request-URI was obtained

Seems legit in that case.

#2 @schrapel
3 years ago

Why would the CURLOPT_URL be the same as the CURLOPT_REFERER? If I manually set CURLOPT_REFERER to the URL of the domain I am sending from then I get the response I desire.

The code is below but I obviously removed our API key. I'm going to try get another API key for you that I can share here

<?php
wp_remote_get('http://api.embed.ly/1/feature?feature=oembed&key=***')

#3 @swissspidy
3 years ago

Why would the CURLOPT_URL be the same as the CURLOPT_REFERER?

I don't know why it was set that way. It has been like this in Requests ever since, see https://github.com/rmccue/Requests/commit/78d4f3c117642445cf729a7eaaaa87d8d3897fd6.

There's an open issue on GitHub, so maybe @rmccue can chime in.

#4 @schrapel
3 years ago

I had a look at Guzzle and they don't seem to do this. Hopefully @rmccue can offer his opinion on this

#5 @jeremyfelt
3 years ago

#37888 was marked as a duplicate.

#6 @differentthemes
3 years ago

Any news here? Will it be fixed in next updates as a additional argument?

#7 @swissspidy
3 years ago

@differentthemes As Requests is an external library, this requires some upstream changes first. You can follow https://github.com/rmccue/Requests/issues/232 for updates on that.

#8 @differentthemes
3 years ago

A month has gone and still don't see any changes here :(

#9 follow-up: @tristangemus
3 years ago

  • Keywords has-patch added

Wordpress 4.5.4 did not include the referrer at all by default. The referrer should be the root URL of the referring website, site_url should be an adequate solution to this. See patch.

#10 @dd32
3 years ago

  • Keywords upstream added

Small note, You can send a Referer header along with your wp_remote_get() which will cause cURL not to send the Referer.
Unfortunately, the Streams handler will send a Referer: header anyway.

This should be fixed upstream at https://github.com/rmccue/Requests/issues/232 - inputs and patches would be welcome there.

#11 in reply to: ↑ 9 ; follow-up: @differentthemes
3 years ago

Replying to tristangemus:

Wordpress 4.5.4 did not include the referrer at all by default. The referrer should be the root URL of the referring website, site_url should be an adequate solution to this. See patch.

Thanks! That helps, but only we can't now force all our customers to edit they wp files. This really should be fixed ASAP at least as a new parameter, if not even by default.

#12 in reply to: ↑ 11 @tristangemus
3 years ago

Replying to differentthemes:

Replying to tristangemus:

Wordpress 4.5.4 did not include the referrer at all by default. The referrer should be the root URL of the referring website, site_url should be an adequate solution to this. See patch.

Thanks! That helps, but only we can't now force all our customers to edit they wp files. This really should be fixed ASAP at least as a new parameter, if not even by default.

Considering this was selected as a solution, it could be part of an upcoming Wordpress release. Hoping to get some support on this or another solution as the referrer being set as the destination URL does not make any sense at all.

Another solution is to not include the referrer and make it an option at all as it is optional in HTTP.

I'd also look into this solution - https://core.trac.wordpress.org/ticket/37820#comment:10

Last edited 3 years ago by tristangemus (previous) (diff)

@differentthemes
3 years ago

Referer header

#13 @differentthemes
3 years ago

Not quite sure if I get it right, but like this it doesn't work.

Edited: Sorry missed this part

$referer = site_url();
Last edited 3 years ago by differentthemes (previous) (diff)

#14 @differentthemes
3 years ago

I have checked WordPress 4.7 BETA 2, but seems like the changes wan't been made there..

#15 @desrosj
7 months ago

  • Keywords upstream removed
  • Milestone Awaiting Review deleted
  • Resolution set to reported-upstream
  • Status changed from new to closed

I am going to close this one out as reported upstream, since action on the GitHub repository is required for this to be merged into WordPress Core.

Please center discussion on Requests issue 232.

#16 @nico23
3 months ago

I am confused about this. I have not digged though the source but looking at the patch it seems the URL is hard-coded in. However, if I got this right, people say we can set the referer with in the $argsheaders? the screenshot above shows code like this and the codex says the arrays should contains a array with header lines. However I have seen a lot of code that uses key-value pairs in the array so my questions are.

  1. Can I set the referer with the $args on wp_remote_get/post.
  2. If yes, do both of this ways work?
  3. Any easy way to test this?
<?php
        $referer = site_url();

        $wp_remote_get_args = array(
                'headers' => array( "Referer: $referer\r\n" );
        );
<?php
        $wp_remote_get_args = array(
                'headers' => array( 'Referer' => site_url() )
        );

        $response = wp_remote_get( $api_url, $wp_remote_get_args );

Edit

Ok I just tested it on https://webhook.site awesome site for testing requests. So turns out the code in the Screenshot from @differentthemes above is actually wrong. It end with with header 0 so not sure where this keyless array with newline is coming from. So only my 2nd example will work.

After thinking about this I think its fine if it stays this way or of WP sends no referer at all for privacy reasons as long as you can manually set it if needed. And having it to the same URL as requested seems kind if the same as setting none all all.

Last edited 3 months ago by nico23 (previous) (diff)
Note: See TracTickets for help on using tickets.