WordPress.org

Make WordPress Core

Opened 11 months ago

Last modified 2 months ago

#28507 accepted task (blessed)

Secure oEmbeds

Reported by: johnbillion Owned by: johnbillion
Milestone: Future Release Priority: normal
Severity: normal Version:
Component: Embeds Keywords:
Focuses: Cc:

Description (last modified by johnbillion)

We need to audit our oEmbed providers and determine:

  • Which ones don't support embedding an https URL
  • Which ones don't support embedding content over SSL

If we have providers in core which do not support embedding content over SSL then we (or the WP.com team) should make contact and see if they're open to implementing it. This is pretty much a prerequisite for #28249 as it stands.


Problem providers:

ProviderCore supports HTTPS URLEndpoint recognises HTTPS URLEmbed supports HTTPSNotes
blip.tvNoNo-Their website resolves over SSL but is broken
dailymotion.comYesYesNearlyEmbeds are served over HTTPS if the oEmbed endpoint uses HTTPS (example)
dai.lyNoNo-Invalid SSL certificate (points to dailymotion.com)
smugmug.comYesYesNo-
hulu.comYesYesNoInvalid SSL certificate (points to Akamai)
photobucket.comNoYesNoSite doesn't resolve over HTTPS
poll.fmYesYesYesInvalid SSL certificate (points to polldaddy.com)
funnyordie.comYesYesNoInvalid SSL certificate (points to Akamai)
instagr.amYesYes-Invalid SSL certificate (points to instagram.com)
imgur.comYesYesNo-
collegehumor.comYesYesNo-
ted.comYesYesYesAlmost there, just some mixed content in embeds
animoto.comYesYesYesHTTPS embeds by default, but mixed content when playing an embed
video214.comYesYesYesHTTPS embeds by default, but mixed content when playing an embed
kck.stYesYes-Domain isn't available over HTTPS

Recently fixed providers:

  • flic.kr - HTTPS everywhere. Regex corrected in r28834.
  • slideshare.net - HTTPS embeds since r28834.
  • wordpress.tv - HTTPS embeds for HTTPS URLs.
  • meetup.com and meetu.ps- HTTPS embeds for HTTPS URLs.
  • instagram.com - HTTPS everywhere since r31710.

Ok providers:

  • youtube.com and youtu.be - HTTPS embeds via the scheme=https parameter.
  • vimeo.com - Embeds are protocol-relative.
  • flickr.com - HTTPS everywhere (same for flic.kr).
  • polldaddy.com - Embeds are served over HTTPS if the parent container uses HTTPS. Effectively protocol-relative via JavaScript.
  • twitter.com - HTTPS everywhere.
  • soundcloud.com - HTTPS everywhere. (Minor note: their oEmbed response includes an http URL for the thumbnail on their CDN, but it resolves over https if you change it.)
  • rdio.com and rd.io - HTTPS embeds by default.
  • spotify.com - HTTPS everywhere.
  • issuu.com - Embeds are served over HTTPS if the parent container uses HTTPS. Effectively protocol-relative via JavaScript.
  • mixcloud.com - Embeds are protocol-relative.
  • tumblr.com - Embeds are partly HTTPS and partly protocol-relative.
  • vine.co - HTTPS everywhere.
  • scribd.com - HTTPS embeds by default.

Attachments (1)

28507.diff (2.5 KB) - added by johnbillion 11 months ago.

Download all attachments as: .zip

Change History (52)

comment:1 @johnbillion11 months ago

Immediate task list:

  • Add support for https URLs for flic.kr.
  • Switch oEmbed endpoint to https for flickr.com, flic.kr, and slideshare.net.
Last edited 9 months ago by johnbillion (previous) (diff)

comment:2 @ircbot11 months ago

This ticket was mentioned in IRC in #wordpress-dev by johnbillion. View the logs.

@johnbillion11 months ago

comment:3 @johnbillion11 months ago

  • Keywords has-patch added

28507.diff is a patch for the above immediate task list.

comment:4 @ircbot11 months ago

This ticket was mentioned in IRC in #wordpress-dev by johnbillion. View the logs.

comment:5 @jkudish11 months ago

As part of some partnership work that I'm doing for WordPress.com, I have been in contact with engineers at Instagram. I contacted them about this issue today. I'll reply back on this ticket when I hear back.

My team at Automattic (we work on partnerships) will see if we can contact most of the other providers as well.

comment:6 @johnbillion11 months ago

That would be fantastic jkudish, thank you. Do also keep us posted on what wp.com are planning with relation to their SSL switchover.

comment:7 @johnbillion11 months ago

In 28834:

Switch to SSL for the Flickr and Slideshare oEmbed endpoints. Add support for SSL embeds on flic.kr. See #28507.

comment:8 @ircbot11 months ago

This ticket was mentioned in IRC in #wordpress-dev by wonderboymusic. View the logs.

comment:9 @azaozz11 months ago

Related: #28195 and the first-run patch for handling secure oEmbed in the admin (wpView) https://core.trac.wordpress.org/attachment/ticket/28195/28195.16.patch.

We may need to change some of the regexp/strings for the providers in order to easily detect which of them support https.

Last edited 11 months ago by azaozz (previous) (diff)

comment:10 @azaozz11 months ago

In 28919:

Secure embeds in the editor (first run):

  • When the user pastes an embeddable http URL, try to get the https embed.
  • If an embed provider doesn't support ssl embeds, show a placeholder/error message.
  • Revise the way we return error messages.

See #28195, #28507.

comment:11 @DrewAPicture11 months ago

In 28949:

Introduce an annotated list of oEmbed providers, their flavors, whether they support SSL, and when they were added to the oembed_providers filter docs.

See #28507.
Fixes #28372.

comment:12 @DrewAPicture11 months ago

In 28950:

Remove duplicate of the 'oembed_providers' filter accidentally introduced in [28949].

Move annoted table of oEmbed providers into the existing filter docs.

See #28507.
Fixes #28372.

comment:13 @ircbot11 months ago

This ticket was mentioned in IRC in #wordpress-dev by azaozz. View the logs.

comment:14 @stephdau11 months ago

This is to let you know that WordPress.tv now has a valid SSL certificate, and could be whitelisted as an SSL-Ok providers.

comment:15 @stephdau11 months ago

WordPress.tv: Actually, not fully true [yet], sorry... It does have a valid SSL cert, but work needs to be done for embeds to be protocol-relative ready, etc.

comment:16 @johnbillion11 months ago

Thanks Stephane. It also looks like the oEmbed endpoint fails if the scheme of the video URL doesn't match the scheme of the endpoint URL. For example this fails:

http://wordpress.tv/oembed?url=https://wordpress.tv/2014/06/07/andrew-nacin-wordcamp-connecticut-keynote/

comment:17 @stephdau11 months ago

Going to look into either making it work, or find the right people to get it to, this morning.
I'll get back to you. :)

comment:18 @stephdau11 months ago

WordPress.tv update: I have the protocol-relative and mismatching schemes figured out, getting a patch reviewed before applying it (wp.tv runs on wp.com).

On the other hand, I'm hitting a temporary wall with wp.tv itself playing its videos (though VideoPress) under SSL. Digging further into that.

comment:19 @stephdau10 months ago

Happy to report that https://wordpress.tv/ is now SSL-ready, for playback on the site, or embedded.

oEmbed responses now use protocol-relative URLs for the video location, which work under both HTTP and HTTPS.

JSON:

{"type":"video","version":"1.0","title":null,"width":400,"height":225,"html":"<embed src=\"\/\/v.wordpress.com\/3JiLCPst\" type=\"application\/x-shockwave-flash\" width=\"400\" height=\"225\" allowscriptaccess=\"always\" allowfullscreen=\"true\" wmode=\"transparent\"><\/embed>"}

XML

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<oembed>
	<type>video</type>
	<version>1.0</version>
	<title></title>
	<width>400</width>
	<height>225</height>
	<html>&lt;embed src=&quot;//v.wordpress.com/3JiLCPst&quot; type=&quot;application/x-shockwave-flash&quot; width=&quot;400&quot; height=&quot;225&quot; allowscriptaccess=&quot;always&quot; allowfullscreen=&quot;true&quot; wmode=&quot;transparent&quot;&gt;&lt;/embed&gt;</html>
</oembed>

The protocol mismatch issue was also handled.

The most network-inquisitives in the crowd might notice some 404s in the load process, upon playback, but those will soon be handled as well, and should not affect playback on the site, or embedded. :)

We're also hoping to look into iframe support in the future, but no timeframe on that.

comment:20 @johnbillion10 months ago

In 29110:

Add support for secure wordpress.tv embeds (thanks stephdau). See #28507.

comment:21 @ircbot10 months ago

This ticket was mentioned in IRC in #wordpress-dev by DrewAPicture. View the logs.

comment:22 follow-up: @johnbillion10 months ago

  • Keywords has-patch removed
  • Milestone changed from 4.0 to Future Release

The issues with Flickr and Slideshare were fixed in r28834. Moving to future milestone for ongoing SSL issues with the providers.

jkudish: Were you able to get anywhere with Instagram or any of the other providers?

comment:23 in reply to: ↑ 22 @jkudish10 months ago

Replying to johnbillion:

jkudish: Were you able to get anywhere with Instagram or any of the other providers?

Instagram is aware of the situation and working on it.

I believe we've contacted some of the others but will make sure we get them all this week.

comment:24 @jkudish10 months ago

My team and I have now contacted the remaining providers. We'll keep this ticket updated if/when folks reply.

comment:25 @stephdau10 months ago

Talked to devs at scribd.com, and pointed out the details of the redirect issue with their oembed endpoint, and an issue when passing https urls to the same endpoint (invalid url). They're on it. Will post updates here.

comment:26 @stephdau10 months ago

Talked to the devs at TED, and they're hoping to have SSL support within the next month, which might make it possible to whitelist them for 4.0. I'll post updates on that here too.

comment:27 @stephdau10 months ago

I've reported the SSL and HTTPS related issues to Meetup.com on GitHub: https://github.com/meetup/api/issues/38

comment:28 @jkudish10 months ago

Imgur is aware and working on it as well

comment:29 @jkudish10 months ago

revision3.com sent me a predef reply, not sure I actually reached a real dev, if someone else has a contact there, please pursue it

comment:30 @stephdau10 months ago

funnyordie.com update: in talk with them, and will be in a "tech call" with them next Monday or Tuesday (Aug 11/12) to help them with supporting SSL across the board.

comment:31 @stephdau10 months ago

Meetup.com update, from their dev on https://github.com/meetup/api/issues/38#issuecomment-50941609

so I should have queued up next week a fix that should return secure photo urls if
 * the X-Meta-Photo-Host header is set to secure
 * the request to the oembed endpoint was made over https
 * the url starts with https
evaluated in that order

Rules 2 and 3 should be the one solving our issues.
Not seeing it deployed yet, we'll see later in the week.

comment:32 @stephdau9 months ago

FunnyOrDie.com update: just had a phone meeting with them today: they are proceeding forward with SSL, and are in discussion with Akamai to do so. They currently do not have a precise timeline for SSL support, but it's in the works.

I have passed along this ticket number, and they'll update it when they are ready (or let me know to).

Last edited 9 months ago by stephdau (previous) (diff)

comment:33 @stephdau9 months ago

Meetup.com update: their fix is now live, meaning that:

  • assets (images) in oembed response will now be returned under SSL if we query the oembed endpoint under SSL
  • same is the case if we query with a URL starting with HTTPS, as per cited example in comment:1 (although not techincally a valid one, since Meetup.com does not support public content under SSL, nor has plans to).

I do note that any query returns image URLs under SSL though (which works).
Made a note of that to them on GitHub.

Last edited 9 months ago by stephdau (previous) (diff)

comment:34 @johnbillion9 months ago

Thanks for the updates Stephane. Great stuff.

Looks like DailyMotion has recently added support for HTTPS embeds. I'm going to update this ticket description with a table layout so we can more easily see where we're at.

comment:35 @johnbillion9 months ago

  • Description modified (diff)
  • Owner set to johnbillion
  • Status changed from new to accepted

I've added a tabular view to the ticket description to aid our sanity.

My task list:

  • Audit the providers that we've added in 4.0
  • Find a meetu.ps URL to audit
  • Switch polldaddy.com oEmbed endpoint to HTTPS as it now redirects there

comment:36 @johnbillion9 months ago

  • Description modified (diff)

comment:37 @johnbillion9 months ago

In 29476:

Switch the Polldaddy oEmbed endpoint to HTTPS as it now redirects there. See #28507.

comment:38 @johnbillion9 months ago

  • Description modified (diff)

Animoto was the only provider missing from the list. Animoto embeds are HTTPS by default, but there's some mixed content in there when playing a video. Found and tested a meetu.ps URL (works fine). Updated the list.

comment:39 @stephdau9 months ago

Awesome. We'll get them all, eventually. :)

comment:40 @ircbot9 months ago

This ticket was mentioned in IRC in #wordpress-dev by stephdau. View the logs.

comment:41 @pento7 months ago

instagram.com now loads over SSL, but it does give a content warning (the profile image is loaded over HTTP).

instagr.am still tries to provide the certificate for instagram.com.

comment:42 @GunGeekATX2 months ago

I believe Instagram is forcing everything to HTTPS now. I tested it out by adding a provider:

add_filter( 'oembed_providers', 'pn_test_instagram_oembed' );
function pn_test_instagram_oembed( $providers ) {
	$providers ['#https://instagr(\.am|am\.com)/p/.*#i'] = array( 'https://api.instagram.com/oembed', true );
	return $providers;
}

Appears to be working: https://petenelson.com/instagram-https-embed-test/

https://api.instagram.com/oembed?url=https://instagram.com/p/0DU6jJIvyw/

comment:43 @johnbillion2 months ago

Confirmed. Everything's redirecting to HTTPS.

instagr.am still isn't available over HTTPS, but embeds for them continue to work as expected.

comment:44 @johnbillion2 months ago

In 31710:

Allow https URLs for Instagram embeds, and switch to https for its oEmbed API endpoint.

See #28507.

comment:45 @johnbillion2 months ago

  • Description modified (diff)

comment:46 @johnbillion2 months ago

  • Description modified (diff)

comment:47 @johnbillion2 months ago

  • Description modified (diff)

comment:48 @johnbillion2 months ago

  • Description modified (diff)

comment:49 @johnbillion2 months ago

  • Description modified (diff)

comment:50 @johnbillion2 months ago

In 31711:

Some updates to the oEmbed provider table.

See #28507

comment:51 @johnbillion2 months ago

  • Description modified (diff)
Note: See TracTickets for help on using tickets.