Support uploading files in HEIF and HEVC formats

[mattheweppelsheimer]

Mac OS X 10.13 High Sierra introduces support for new native image format HEIF and video format HEVF.

These aren't widely used by anyone yet, and absolutely no browsers support displaying HEIFs at present (per CanIUse.com as of 2017-12-02 ​https://caniuse.com/#feat=heif) — not even Safari. So this clearly isn't a big priority, but it seems likely these will become ubiquitous in time, and it would be good to at least support uploading these file formats, since they are valid media files.

This was brought to my attention by this line in Ars Technica's review of OSX 10.13:

The Ars WordPress CMS won't even let me upload HEIF files [...]

​https://arstechnica.com/gadgets/2017/09/macos-10-13-high-sierra-the-ars-technica-review/

We should probably fix that. :)

[kirasong]

I think it makes sense to whitelist the type while support grows for it to be displayable/workable by readily available tools, while putting off full image support until those tools are common.

WordPress whitelists uploads of various types including TIFF that are not natively displayable. As far as I'm aware, this list usually contains most common file formats that are not considered to be a security risk.

[jeffr0]

I first tried to upload a HEIC file through the browser to a wordpress.com site and it didn't work because of file permissions. I then logged into the WordPress for iOS app, adding a new post and uploaded a media file directly from the phone which I assume is a HEIC file and it worked.

It turns out the file is not HEIC but is JPEG. I asked the WordPress for iOS developers if the app converts the image format and it does not.

"Hi Jeff, when we pull the image from the OS, it gives it to us in jpg. So we don’t do any converting ourselves within the app" - Will Kwon

I then tried to upload a media file from within the app to a WordPress.org powered site and it didn't work.

I've attached a screenshot of the error that's seen when trying to upload a HEIC file to WordPress.

[jeffr0]

Error When Trying to Upload a HEIC File to WordPress

[mattheweppelsheimer]

@mikeschroder's +1 above satisfies the need for a "2nd-opinion"; removing that tag.

I plan to contribute a patch to add these file formats to the upload whitelist. Not a big priority, so this might linger for a while.

[harveyfloyd]

I would very much like this feature. I offer products created from the new HEIC format and although a preview would be nice I DON'T need or want the file converted to a JPG. I NEED that extra information contained within the file.

[mattheweppelsheimer]

@harveyfloyd Good news: It is already possible to modify the default list of file types WordPress will allow you to upload. Check out the filters upload_mimes and wp_check_filetype_and_ext. Here's an example of a plugin that uses those and might meet your needs out of the box: ​WP Extra File Types.

Given those workarounds, and given that this change is not critical for the vast majority of WordPress users at this moment in time, I'm walking back the severity property.

[mattheweppelsheimer]

This discussion in Slack is important: ​https://wordpress.slack.com/messages/C02SX62S6/

Summary of takeaways:

• The scope of this ticket is only to allow *uploads*, not full image support, since underlying libraries WordPress uses to process and display images (GD and Imagick) don't support HEIF. (Here is an ​Imagick feature request discussion on Github.)
• So then, why bother? In media asset management use cases it is still valuable to be able to upload media to store and organize it in your WordPress, even if you aren't displaying it.
• The mime type whitelist is a bit stale; if we're going to change it, it's worth considering more than just heif. webp is further along than heif. flac is a widely-used lossless audio format that apparently we're also missing. Perhaps there are other formats we should also add.
• Besides a patch, this will need testing to ensure there aren't generic image/ MIME matching conditions that could inadvertently try to process heif files unsupported by GD/Imagick. (Nothing's ever simple, is it?) ​@bobfolio in Slack raised this point:

only open question would be if there are any generic image/ MIME matching conditions hidden around that might need to be made more specific, if nothing else to avoid unnecessary overhead of running files through libraries that won't end up supporting them anyway. I know there used to be, but am not sure what remains in 4.9.1

[mattheweppelsheimer]

Hmm... Are there any default white listed MIME types that can't be displayed on the front end?

If not, introducing some might make for bad UX. I think it's likely some users will be confused if we allow uploading a file that can't be displayed or used in site content. It might appear "broken".

Perhaps this deserves some kind of UI heads up somewhere. Something like, "Hey, this media file can't be displayed in web browsers. You can upload and download HEIF files, but you can't display them on your site." — Or, perhaps we're already supporting un-displayable file type uploads, and already doing exactly that?

[harveyfloyd]

Too bad it doesn't work for heic, heif, heics or heifs files. Not even if you add them at the bottom and get all the mime types right.

Replying to mattheweppelsheimer:

@harveyfloyd Good news: It is already possible to modify the default list of file types WordPress will allow you to upload. Check out the filters upload_mimes and wp_check_filetype_and_ext. Here's an example of a plugin that uses those and might meet your needs out of the box: ​WP Extra File Types.

Given those workarounds, and given that this change is not critical for the vast majority of WordPress users at this moment in time, I'm walking back the severity property.

[mattheweppelsheimer]

@harveyfloyd that's odd, not what I expected, and disappointing. It might be worth opening a support ticket for the plugin author to address that. Meanwhile, using the upload_mimes and wp_check_filetype_and_ext hooks sound like the way to go.

[desrosj]

Based on the plan of 5.0.1 being a quick, 2-week release, and the fact that there is not a workable patch yet, I am going to punt this to 5.1.

[Wilhelm Ahrendt]

HEIC Uploads from iPhone or iPad is so important

Trac ticket: https://core.trac.wordpress.org/ticket/42775

[azaozz]

Been looking at this for a bit. Thinking that if the users can upload HEIF images they will expect them to "just work" (which is the right thing), but they won’t work as there's no browser support.

Seems that currently the best way to handle HEIF images would be to check if they can be converted to JPEGs on the server.

• If yes, allow uploading, do the conversion on the server.
• If no, show a warning/notice to the user to convert the image to JPEG before uploading.

As far as I see ImageMagick can convert HEIF to JPEG using external library. GD cannot. That will introduce inconsistencies depending on the hosting setup, but can be considered progressive enhancement.

Moving to the 5.5 milestone to consider adding the above message and do the conversion when supported by the server.

[harveyfloyd]

NO. Automatically converting to JPG is NOT a good option. HEIC and HEIF files contain extra information that myself and others would like to utilize. Converting to JPG destroys that information. Creating a JPG preview for display in a gallery or an a page would be good but uploading the original file without modification is needed.

Replying to azaozz:

Been looking at this for a bit. Thinking that if the users can upload HEIF images they will expect them to "just work" (which is the right thing), but they won’t work as there's no browser support.

Seems that currently the best way to handle HEIF images would be to check if they can be converted to JPEGs on the server.

• If yes, allow uploading, do the conversion on the server.
• If no, show a warning/notice to the user to convert the image to JPEG before uploading.

As far as I see ImageMagick can convert HEIF to JPEG using external library. GD cannot. That will introduce inconsistencies depending on the hosting setup, but can be considered progressive enhancement.

Moving to the 5.5 milestone to consider adding the above message and do the conversion when supported by the server.

[johnbillion]

I question why core needs to support these file types at all.

If you share or export a HEIC from iOS it converts it to JPEG, so you lose any special data contained within the HEIC.

If you need that special data from HEIC files then I think you'll be doing custom development anyway and can whitelist the mime types.

What advantage is gained by core supporting these file types that are not supported in any browser?

[johnbillion]

@harveyfloyd What special information is contained in a HEIC that you need to access?

[azaozz]

Replying to johnbillion:

I question why core needs to support these file types at all.

The only reason is to support uploading and using of photos taken on iPhone. This seems to be a somewhat common use case.

Agree it needs a second opinion/more consideration. Perhaps a message about converting them before uploading would be sufficient.

Just thinking WP shouldn't leave the users wondering why they cannot upload the photos they've taken.

[azaozz]

Replying to harveyfloyd:

NO. Automatically converting to JPG is NOT a good option.

If this is implemented it will keep the originally uploaded file as well. This is the "general policy" when storing files in the uploads directory (see the handling of big images). Of course, a plugin may delete these files, but that will be the user's choice.

[andraganescu]

I tend to agree with @johnbillion in questioning why upload these files. Even if we convert a copy to a format that can be displayed, I don't think it is neither the moment nor worth the effort to implement a special flow in the media library that ties the JPG and the HEIC.

Therefore the user will be left with weirdly named files that they have to manually link together to have something that makes sense on the front end.

Also, who distributes these files online anyway? Either way, it will be a clunky experience, simply because the format is not supported by the browsers. If the upload is because of some sort of backup or photo library management desire then plugins definitely will do this better.

[johnbillion]

Are there any standard workflows on an iPhone or iPad that result in a HEIF or HEVC file being uploaded to WordPress? My understanding is that any time a photo or video gets shared from iOS, it gets exported as a JPEG or MPEG.

In what situation(s) does a HEIF or HEVC get exported without being converted?

[andraganescu]

When you share media using other methods, such as AirDrop, Messages, or email, it might be shared in a more compatible format, such as JPEG or H.264, depending on whether the receiving device supports the newer media format.

This ​Apple support page explains that when sharing HEIC/HEIF the device will somehow assess that the receiving device is able to open these formats and send originals if so or convert to JPG otherwise.

This means that by simply uploading from the library to the WP Admin you will always get JPG files. It therefore appears that the only situation we cover is when someone has a compatible Apple device, receives a photo in this format (say on iMessage) and then tries to upload that on WordPress.

Is this a flow we want to support in core, the poor support for this format?

[desrosj]

I did some testing today to see how different scenarios are handled. I did experience a few hiccups with HEIC files that I'd like to detail here.

There is a setting in Settings > Photos called "Transfer to Mac or PC". This setting dictates whether to always keep the original file format or to automatically check for compatibility and export as a compatible format. I _believe_ this is set to automatic (always convert to a compatible format).

In the past, I somehow had this set to "Keep Originals". When this is selected, if photos are transferred to a device, unless the application sharing converts the file (Gmail seems to convert to JPG), the file is shared/transferred as HEIC (using Airdrop this happens, for example). If a user tries to upload that file to their site, it will return the "Sorry, this file type is not permitted for security reasons" warning.

Even with "Keep Originals" selected, uploading an image to the media library through Safari and Firefox appears to convert the image to JPG format on iOS. This is opposite behavior as desktop.

Another situation would be if a user has an older version of MacOS (Sierra or earlier) that does not support HEIC. The previously mentioned setting would also need to be set to "Keep Originals". It would be impossible for the user to convert the file to JPG or similar format locally without the help of an additional application.

I am in the camp that core should not support this format yet.

I could be wrong, but I don't think that there is currently a way core can convert HEVC files. So even if support is added for HEIC files by generating a JPG file for maximum support (while maintaining the HEIC file as the original), it doesn't solve the HEVC problem.

In my opinion, a plugin is the best place for this until (if) browsers start adopting support for this. I propose closing as maybelater.

A few other thoughts. @mattheweppelsheimer had mentioned .flac files above. This was addressed already in r42451. There is also a separate ticket for .webp format (#35725). However, support for that format is also still lacking.

[mattheweppelsheimer]

Based on recent input and research contributed to this ticket, I'm closing it out.

To summarize:

1. When I first opened the ticket, I imagined the HEIF and HEVC formats would gain support across browsers and mobile platforms. That has not happened.
2. Apple devices don't support a consistent workflow for exporting or manipulating these file types but do export them to JPEG in a graceful degradation scenario, so it isn't as if the images are stuck on those devices. This isn't really WordPress' problem to fix.
3. There's no way to display these file types visually due to lack of browser support, with no sign of that changing. It doesn't make sense for a web content management system to support media formats that don't work on the web.
4. The majority of WordPress users do not need this — it clearly doesn't pass the 80/20 test.
5. For the minority of users who do want this, there are filter hooks available to exploit.

[matt]

Reopening this one, as it's a very real issue for anyone on an iOS device with the default settings... which is a lot of people.

[azaozz]

Yes, may not be very common, but is really off putting when it happens.

Seems the way to go would be to show a warning that the user is trying to upload a non-supported image format, and ask them to convert it/export it if they intend to use it on their site.

[andraganescu]

So the way to stumble upon WordPress not supporting this format is this:

• shoot a _live_ photo on your phone with at least iOS 11
• have it synced to your iCloud Photos
• using a browser (Safari) go to your iCloud and open the cloud Photos
• open your photo and download using the top bar download button

This will result in an "iCloud Photos" in the Downloads folder (at least on OSX it will) and in that folder you'll get a .heic file and a .mov file.

Attempting to upload this file will not work on stock WordPress.

Uploading directly from an iPad or iPhone to stock WordPress' Media Library will work as Safari (tested with Firefox mobile as well) will convert the file to JPEG directly on upload.

Since browsers don't support these formats at all, but users expect them to work as normal images I suspect there should be a flow like:

• user uploads these formats
• server side we convert to something browsers can display
• we show the converted (?) file in media library and default attachment pages to that
• we keep (or not?) the original and if we do keep we should link the converted file to the original

That would be smooth and expected. The other option is to simply let these files be uploaded.

[desrosj]

I don't think that the files can be reliably converted with the current expected setup (Imagick or the less desirable GD).

While ​HEIC formats are supported in Imagick, it appears to require an ​additional delegate library which does not appear to be configured by default on hosts (so users would not receive converted images).

I also have been unable to find any PHP based libraries that we could consider for server side converting. There are some JS based ones, but they are not widely adopted.

I lean towards the suggestion that @azaozz had above of displaying a warning to a user that the format they are uploading is not supported and maybe linking to some how-to documentation.

[azaozz]

In 42775.diff:

• Add .heic to the supported uploads (as far as I see no security implications).
• Add an "unsupported image" message to the uploads errors.
• Add a param to the Plupload.js settings to show the error by using a filter. Plugins can remove the filter if needed.

TODO/TBD: Currently if more than one .heic image is being uploaded at the same time, the error is shown for each. Perhaps better to only show it once.

Also, the "unsupported image" user-error message text could perhaps be improved? :)

[azaozz]

User error when uploading from the media library.

[azaozz]

When uploading from the media modal.

[kirasong]

Just a quick note that I noticed that the library in question can be compiled into JS:
​https://github.com/strukturag/libheif#compiling-to-javascript

I don't know if that's something we'd want to do, but figured the information might be helpful in the ticket.

[azaozz]

Replying to mikeschroder:

I don't know if that's something we'd want to do, but figured the information might be helpful in the ticket.

Ah, great, thanks! Was mostly looking at ways to convert on the server (ImageMagick) but that doesn't seem commonly supported.

Converting in the browser (canvas + js) is a great option (although it comes with the normal limitations for large images on low memory devices). Tested the demo and it seems to be more or less usable. Quality is lower and the file size is still quite large, 3.8MB .jpeg from a 9MB .heic, but suppose that's about right? There are also ​https://www.npmjs.com/package/libheif-js and ​https://github.com/catdad-experiments/heic-convert. Perhaps can look at doing that in 5.6?

[azaozz]

In 48288:

Media: Show an error message when a .heic file is uploaded that this type of files cannot be displayed in a web browser and suggesting to convert to JPEG. The message is shown by using filters, plugins that want to handle uploading of .heic files can remove it.

Props mattheweppelsheimer, mikeschroder, jeffr0, andraganescu, desrosj, azaozz.
Fixes #42775.

[SergeyBiryukov]

In 48296:

Media: Add heic extension to wp_get_ext_types(), for consistency with wp_get_mime_types().

Follow-up to [48288].

Props imath.
Fixes #50557. See #42775.

[desrosj]

In 48343:

Docs: Correct typo introduced in [48288].

See #42775.

[adamsilverstein]

A note that in https://core.trac.wordpress.org/ticket/53645 (which will ship in WordPress 6.7) we added support for uploading HEIC images (like those from an iOS device) as long as your server configuration supports processing these HEIC files to create a more web-friendly format like JPEG.