WordPress.org

Make WordPress Core

Opened 2 years ago

Last modified 3 months ago

#28507 accepted task (blessed)

Secure oEmbeds

Reported by: johnbillion Owned by: johnbillion
Milestone: Future Release Priority: normal
Severity: normal Version:
Component: Embeds Keywords: ongoing https
Focuses: Cc:

Description (last modified by johnbillion)

We need to audit our oEmbed providers and determine:

  • Which ones don't support embedding an https URL
  • Which ones don't support embedding content over SSL

If we have providers in core which do not support embedding content over SSL then we (or the WP.com team) should make contact and see if they're open to implementing it. This is pretty much a prerequisite for #28249 as it stands.


Problem providers:

ProviderCore supports HTTPS URLEndpoint recognises HTTPS URLEmbed supports HTTPSNotes
hulu.comYesYesNoInvalid SSL certificate (points to Akamai)
photobucket.comNoYesNoSite doesn't resolve over HTTPS
poll.fmYesYesYesInvalid SSL certificate (points to polldaddy.com)

Recently fixed providers:

  • flic.kr - HTTPS everywhere. Regex corrected in r28834.
  • slideshare.net - HTTPS embeds since r28834.
  • wordpress.tv - HTTPS embeds for HTTPS URLs.
  • meetup.com and meetu.ps- HTTPS embeds for HTTPS URLs.
  • instagram.com - HTTPS everywhere since r31710.
  • instagr.am - HTTPS URLs are now supported.
  • dailymotion.com - Uses the HTTPS oEmbed endpoint since r34587.
  • dai.ly - Cert is now valid for the dai.ly domain.
  • smugmug.com - Embeds now use HTTPS by default.
  • funnyordie.com - Embeds are now protocol-relative and cert is now valid.
  • imgur.com - Embeds are now protocol-relative.
  • collegehumor.com - HTTPS embeds for HTTPS URLs.
  • animoto.com and video214.com - Embeds now use HTTPS by default. See also r34588.
  • kck.st - Cert is now valid for the kck.st domain.

Ok providers:

  • youtube.com and youtu.be - HTTPS everywhere.
  • vimeo.com - Embeds are protocol-relative.
  • flickr.com - HTTPS everywhere (same for flic.kr).
  • polldaddy.com - Embeds are served over HTTPS if the parent container uses HTTPS. Effectively protocol-relative via JavaScript.
  • twitter.com - HTTPS everywhere.
  • soundcloud.com - HTTPS everywhere. (Minor note: their oEmbed response includes an http URL for the thumbnail on their CDN, but it resolves over https if you change it.)
  • rdio.com and rd.io - HTTPS embeds by default.
  • spotify.com - HTTPS everywhere.
  • issuu.com - Embeds are served over HTTPS if the parent container uses HTTPS. Effectively protocol-relative via JavaScript.
  • mixcloud.com - Embeds are protocol-relative.
  • tumblr.com - Embeds are partly HTTPS and partly protocol-relative.
  • vine.co - HTTPS everywhere.
  • scribd.com - HTTPS embeds by default.
  • ted.com - HTTPS embeds for HTTPS URLs.
  • videopress.com - HTTPS embeds for HTTPS URLs.
  • reverbnation.com - HTTPS embeds by default.
  • speakerdeck.com - Embeds are protocol-relative.
  • facebook.com - HTTPS everywhere.

Attachments (3)

28507.diff (2.5 KB) - added by johnbillion 2 years ago.
28507.2.diff (30.3 KB) - added by swissspidy 6 months ago.
28507.3.diff (2.6 KB) - added by swissspidy 4 months ago.

Download all attachments as: .zip

Change History (77)

#1 @johnbillion
2 years ago

Immediate task list:

  • Add support for https URLs for flic.kr.
  • Switch oEmbed endpoint to https for flickr.com, flic.kr, and slideshare.net.
Last edited 2 years ago by johnbillion (previous) (diff)

This ticket was mentioned in IRC in #wordpress-dev by johnbillion. View the logs.


2 years ago

@johnbillion
2 years ago

#3 @johnbillion
2 years ago

  • Keywords has-patch added

28507.diff is a patch for the above immediate task list.

This ticket was mentioned in IRC in #wordpress-dev by johnbillion. View the logs.


2 years ago

#5 @jkudish
2 years ago

As part of some partnership work that I'm doing for WordPress.com, I have been in contact with engineers at Instagram. I contacted them about this issue today. I'll reply back on this ticket when I hear back.

My team at Automattic (we work on partnerships) will see if we can contact most of the other providers as well.

#6 @johnbillion
2 years ago

That would be fantastic jkudish, thank you. Do also keep us posted on what wp.com are planning with relation to their SSL switchover.

#7 @johnbillion
2 years ago

In 28834:

Switch to SSL for the Flickr and Slideshare oEmbed endpoints. Add support for SSL embeds on flic.kr. See #28507.

This ticket was mentioned in IRC in #wordpress-dev by wonderboymusic. View the logs.


2 years ago

#9 @azaozz
2 years ago

Related: #28195 and the first-run patch for handling secure oEmbed in the admin (wpView) https://core.trac.wordpress.org/attachment/ticket/28195/28195.16.patch.

We may need to change some of the regexp/strings for the providers in order to easily detect which of them support https.

Last edited 2 years ago by azaozz (previous) (diff)

#10 @azaozz
2 years ago

In 28919:

Secure embeds in the editor (first run):

  • When the user pastes an embeddable http URL, try to get the https embed.
  • If an embed provider doesn't support ssl embeds, show a placeholder/error message.
  • Revise the way we return error messages.

See #28195, #28507.

#11 @DrewAPicture
2 years ago

In 28949:

Introduce an annotated list of oEmbed providers, their flavors, whether they support SSL, and when they were added to the oembed_providers filter docs.

See #28507.
Fixes #28372.

#12 @DrewAPicture
2 years ago

In 28950:

Remove duplicate of the 'oembed_providers' filter accidentally introduced in [28949].

Move annoted table of oEmbed providers into the existing filter docs.

See #28507.
Fixes #28372.

This ticket was mentioned in IRC in #wordpress-dev by azaozz. View the logs.


2 years ago

#14 @stephdau
2 years ago

This is to let you know that WordPress.tv now has a valid SSL certificate, and could be whitelisted as an SSL-Ok providers.

#15 @stephdau
2 years ago

WordPress.tv: Actually, not fully true [yet], sorry... It does have a valid SSL cert, but work needs to be done for embeds to be protocol-relative ready, etc.

#16 @johnbillion
2 years ago

Thanks Stephane. It also looks like the oEmbed endpoint fails if the scheme of the video URL doesn't match the scheme of the endpoint URL. For example this fails:

http://wordpress.tv/oembed?url=https://wordpress.tv/2014/06/07/andrew-nacin-wordcamp-connecticut-keynote/

#17 @stephdau
2 years ago

Going to look into either making it work, or find the right people to get it to, this morning.
I'll get back to you. :)

#18 @stephdau
2 years ago

WordPress.tv update: I have the protocol-relative and mismatching schemes figured out, getting a patch reviewed before applying it (wp.tv runs on wp.com).

On the other hand, I'm hitting a temporary wall with wp.tv itself playing its videos (though VideoPress) under SSL. Digging further into that.

#19 @stephdau
2 years ago

Happy to report that https://wordpress.tv/ is now SSL-ready, for playback on the site, or embedded.

oEmbed responses now use protocol-relative URLs for the video location, which work under both HTTP and HTTPS.

JSON:

{"type":"video","version":"1.0","title":null,"width":400,"height":225,"html":"<embed src=\"\/\/v.wordpress.com\/3JiLCPst\" type=\"application\/x-shockwave-flash\" width=\"400\" height=\"225\" allowscriptaccess=\"always\" allowfullscreen=\"true\" wmode=\"transparent\"><\/embed>"}

XML

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<oembed>
	<type>video</type>
	<version>1.0</version>
	<title></title>
	<width>400</width>
	<height>225</height>
	<html>&lt;embed src=&quot;//v.wordpress.com/3JiLCPst&quot; type=&quot;application/x-shockwave-flash&quot; width=&quot;400&quot; height=&quot;225&quot; allowscriptaccess=&quot;always&quot; allowfullscreen=&quot;true&quot; wmode=&quot;transparent&quot;&gt;&lt;/embed&gt;</html>
</oembed>

The protocol mismatch issue was also handled.

The most network-inquisitives in the crowd might notice some 404s in the load process, upon playback, but those will soon be handled as well, and should not affect playback on the site, or embedded. :)

We're also hoping to look into iframe support in the future, but no timeframe on that.

#20 @johnbillion
2 years ago

In 29110:

Add support for secure wordpress.tv embeds (thanks stephdau). See #28507.

This ticket was mentioned in IRC in #wordpress-dev by DrewAPicture. View the logs.


2 years ago

#22 follow-up: @johnbillion
2 years ago

  • Keywords has-patch removed
  • Milestone changed from 4.0 to Future Release

The issues with Flickr and Slideshare were fixed in r28834. Moving to future milestone for ongoing SSL issues with the providers.

jkudish: Were you able to get anywhere with Instagram or any of the other providers?

#23 in reply to: ↑ 22 @jkudish
2 years ago

Replying to johnbillion:

jkudish: Were you able to get anywhere with Instagram or any of the other providers?

Instagram is aware of the situation and working on it.

I believe we've contacted some of the others but will make sure we get them all this week.

#24 @jkudish
2 years ago

My team and I have now contacted the remaining providers. We'll keep this ticket updated if/when folks reply.

#25 @stephdau
2 years ago

Talked to devs at scribd.com, and pointed out the details of the redirect issue with their oembed endpoint, and an issue when passing https urls to the same endpoint (invalid url). They're on it. Will post updates here.

#26 @stephdau
2 years ago

Talked to the devs at TED, and they're hoping to have SSL support within the next month, which might make it possible to whitelist them for 4.0. I'll post updates on that here too.

#27 @stephdau
2 years ago

I've reported the SSL and HTTPS related issues to Meetup.com on GitHub: https://github.com/meetup/api/issues/38

#28 @jkudish
2 years ago

Imgur is aware and working on it as well

#29 @jkudish
2 years ago

revision3.com sent me a predef reply, not sure I actually reached a real dev, if someone else has a contact there, please pursue it

#30 @stephdau
2 years ago

funnyordie.com update: in talk with them, and will be in a "tech call" with them next Monday or Tuesday (Aug 11/12) to help them with supporting SSL across the board.

#31 @stephdau
2 years ago

Meetup.com update, from their dev on https://github.com/meetup/api/issues/38#issuecomment-50941609

so I should have queued up next week a fix that should return secure photo urls if
 * the X-Meta-Photo-Host header is set to secure
 * the request to the oembed endpoint was made over https
 * the url starts with https
evaluated in that order

Rules 2 and 3 should be the one solving our issues.
Not seeing it deployed yet, we'll see later in the week.

#32 @stephdau
2 years ago

FunnyOrDie.com update: just had a phone meeting with them today: they are proceeding forward with SSL, and are in discussion with Akamai to do so. They currently do not have a precise timeline for SSL support, but it's in the works.

I have passed along this ticket number, and they'll update it when they are ready (or let me know to).

Last edited 2 years ago by stephdau (previous) (diff)

#33 @stephdau
2 years ago

Meetup.com update: their fix is now live, meaning that:

  • assets (images) in oembed response will now be returned under SSL if we query the oembed endpoint under SSL
  • same is the case if we query with a URL starting with HTTPS, as per cited example in comment:1 (although not techincally a valid one, since Meetup.com does not support public content under SSL, nor has plans to).

I do note that any query returns image URLs under SSL though (which works).
Made a note of that to them on GitHub.

Last edited 2 years ago by stephdau (previous) (diff)

#34 @johnbillion
2 years ago

Thanks for the updates Stephane. Great stuff.

Looks like DailyMotion has recently added support for HTTPS embeds. I'm going to update this ticket description with a table layout so we can more easily see where we're at.

#35 @johnbillion
2 years ago

  • Description modified (diff)
  • Owner set to johnbillion
  • Status changed from new to accepted

I've added a tabular view to the ticket description to aid our sanity.

My task list:

  • Audit the providers that we've added in 4.0
  • Find a meetu.ps URL to audit
  • Switch polldaddy.com oEmbed endpoint to HTTPS as it now redirects there

#36 @johnbillion
2 years ago

  • Description modified (diff)

#37 @johnbillion
2 years ago

In 29476:

Switch the Polldaddy oEmbed endpoint to HTTPS as it now redirects there. See #28507.

#38 @johnbillion
2 years ago

  • Description modified (diff)

Animoto was the only provider missing from the list. Animoto embeds are HTTPS by default, but there's some mixed content in there when playing a video. Found and tested a meetu.ps URL (works fine). Updated the list.

#39 @stephdau
2 years ago

Awesome. We'll get them all, eventually. :)

This ticket was mentioned in IRC in #wordpress-dev by stephdau. View the logs.


2 years ago

#41 @pento
2 years ago

instagram.com now loads over SSL, but it does give a content warning (the profile image is loaded over HTTP).

instagr.am still tries to provide the certificate for instagram.com.

#42 @GunGeekATX
21 months ago

I believe Instagram is forcing everything to HTTPS now. I tested it out by adding a provider:

add_filter( 'oembed_providers', 'pn_test_instagram_oembed' );
function pn_test_instagram_oembed( $providers ) {
	$providers ['#https://instagr(\.am|am\.com)/p/.*#i'] = array( 'https://api.instagram.com/oembed', true );
	return $providers;
}

Appears to be working: https://petenelson.com/instagram-https-embed-test/

https://api.instagram.com/oembed?url=https://instagram.com/p/0DU6jJIvyw/

#43 @johnbillion
21 months ago

Confirmed. Everything's redirecting to HTTPS.

instagr.am still isn't available over HTTPS, but embeds for them continue to work as expected.

#44 @johnbillion
21 months ago

In 31710:

Allow https URLs for Instagram embeds, and switch to https for its oEmbed API endpoint.

See #28507.

#45 @johnbillion
21 months ago

  • Description modified (diff)

#46 @johnbillion
21 months ago

  • Description modified (diff)

#47 @johnbillion
21 months ago

  • Description modified (diff)

#48 @johnbillion
21 months ago

  • Description modified (diff)

#49 @johnbillion
21 months ago

  • Description modified (diff)

#50 @johnbillion
21 months ago

In 31711:

Some updates to the oEmbed provider table.

See #28507

#51 @johnbillion
21 months ago

  • Description modified (diff)

#52 @johnbillion
16 months ago

  • Description modified (diff)

#53 @johnbillion
15 months ago

In 34587:

Switch to the https oEmbed endpoint for Dailymotion so secure embeds are supported.

See #28507

#54 @johnbillion
15 months ago

  • Description modified (diff)

#55 @johnbillion
15 months ago

  • Description modified (diff)

#56 @johnbillion
15 months ago

  • Description modified (diff)

#57 @johnbillion
15 months ago

  • Description modified (diff)

#58 @johnbillion
15 months ago

  • Description modified (diff)

#59 @johnbillion
15 months ago

In 34588:

Switch to the https oEmbed endpoint for Animoto because it redirects there anyway.

See #28507

#60 @johnbillion
15 months ago

  • Description modified (diff)

#61 @DrewAPicture
15 months ago

In 34589:

Docs: Adjust the table of providers in the hook doc for oembed_providers to use "No" instead of "!" to signify lack of SSL support.

When displayed in the Code Reference, the "!" doesn't convey enough information.

See #32246. See #28507.

#62 @johnbillion
15 months ago

  • Keywords ongoing added

#63 @johnbillion
13 months ago

  • Description modified (diff)

#64 @johnbillion
12 months ago

  • Keywords https added

#65 @johnbillion
8 months ago

  • Description modified (diff)
  • Keywords changed from ongoing, https to ongoing https

#66 @johnbillion
8 months ago

  • Description modified (diff)

#67 @swissspidy
7 months ago

There's a patch on #36274 addressing YouTube and Vimeo, whose API endpoints are currently configured with http://, but redirect to https:// versions anyway. We should add this for 4.6.

#68 @swissspidy
7 months ago

#36274 was marked as a duplicate.

@swissspidy
6 months ago

#69 @swissspidy
6 months ago

28507.2.diff is the patch from #36274 (props zsusag) to change the YouTube endpoint URLs to https as they redirect there.

#70 @paulschreiber
4 months ago

I just found #36274, which was marked as a duplicate of this ticket. It looks like the patch didn't make it for 4.6. Can we get that in to 4.7?

@swissspidy
4 months ago

#71 @swissspidy
4 months ago

Yeah, I think we can add 28507.3.diff in 4.7

#72 @johnbillion
3 months ago

In 38365:

Embeds: Always use the HTTPS endpoint for YouTube embeds. The scheme parameter is no longer required as all YouTube assets now use HTTPS.

See #36274, #28507
Props zsusag, tollmanz

#73 @johnbillion
3 months ago

In 38366:

Embeds: Many of our oEmbed providers now default to HTTPS embeds, redirect to the HTTPS oEmbed endpoint, or have complete support for HTTPS even if they don't default to HTTPS.

This change defaults to using HTTPS endpoints for oEmbeds for those providers that have full HTTPS support and don't redirect back to HTTP when clicking through from the embed. It covers:

  • Vimeo
  • SmugMug
  • Scribd
  • WordPress.tv
  • SoundCloud
  • Meetup
  • issuu
  • Mixcloud
  • TED

See #28507

#74 @johnbillion
3 months ago

  • Description modified (diff)
Note: See TracTickets for help on using tickets.