Make WordPress Core

Opened 3 years ago

Last modified 5 months ago

#55443 reopened enhancement

Create WebP sub-sizes and use for output

Reported by: adamsilverstein's profile adamsilverstein Owned by: adamsilverstein's profile adamsilverstein
Milestone: Future Release Priority: normal
Severity: normal Version: 6.0
Component: Media Keywords: has-unit-tests needs-dev-note needs-docs needs-user-docs has-patch
Focuses: performance Cc:

Description (last modified by adamsilverstein)

Summary

This ticket introduces core support for generating and using additional mime types for sub-sized image uploads, with a focus on the WebP format. In addition, it filters frontend content to use the modern format when available for an image. After this change, WordPress will generate and use WebP sub sizes by default when available.

Currently, when users upload images in a supported mime type like JPEG or WebP, WordPress creates sub-sized images of the same type for use on the front end. In WordPress 5.8 we introduced the image_editor_output_format filter to enable filtering the mime type used to save images.

This ticket introduces the capability to generate more than a single mime type. For example, when users upload JPEG images, WordPress can automatically generate both WebP and JPEG sub-sized images. Then WordPress can use the WebP images when serving up the front end. JPEG images are still being generated by default, to support other use-cases outside of common web browsers (e.g. email newsletters) where WebP may not be supported yet.

The above is only the default though - the behavior of which image formats are generated and used can be customized via filters.

New Filters

This patch introduces two new filters, one to control the mime types WordPress generates when you upload an image and one to control which image mime type is used for displaying images on the front end of the website.

Mime type image generation filter

This patch adds a new filter named wp_upload_image_mime_transforms (and similarly named function) that maps the upload image mime type to an array of one or more output mime types:

By default, the mapping is:

	array(
		'image/jpeg'        => array( 'image/jpeg', 'image/webp' ),
		'image/webp'      => array( 'image/webp', 'image/jpeg' ),
	);

The attachment id is also passed to the filter for context.

Mime type front end output filter

This patch adds a new filter wp_content_image_mimes for the front end that specifies the preferred format(s) to use for content output (as long as the image file is available). When outputting images in the content, the first mime type available will be used. The attachment id and context are also passed to the filter.

WebP as a default output format for images

On the front end, sites will benefit from using WebP in place of JPEG for their images. Research on https://github.com/WordPress/performance/issues/7 indicates images will average around a 30% overall file size reduction. This work also includes verifying the visual image quality of the output using dssim.

Developers Note

The function make_subsize in both of core’s bundled WP_Image_Editor implementations (WP_Image_Editor_Imagick and WP_Image_Editor_GD) has been updated to accept a $mime_type parameter which in turn passed the mime type to the _save function (which already supports a $mime_type parameter). If your code implements ​​WP_Image_Editor::make_subsize() make sure you update your function to support the $mime_type parameter.
Plugins that modify image use (CDN, S3) or already create alternate mime types should test integrating with the code on this ticket and consider supporting the new sources meta data.

How to Test

Test with this patch or the Performance Lab plugin
Create a post and upload a jpeg image, placing in the post
Publish the post and view it’s source code, the image URLs should be “.webp” images

Technical Details

  • In src/wp-admin/includes/image.php::wp_create_image_subsizes:

loop through all of the mime types returned from the ​​wp_upload_image_mime_transforms filter, generating sub-sized images for every mime format supported by the system. Default when a type isn’t mapped in the array is outputting in the mime type of the uploaded image.

  • _wp_get_primary_and_additional_mime_types extracts the primary and additional mime types
  • “Full” sized images are created for each mime type supported by the system with a -{mime extension} appended to the file name
  • When images over big_image_size_threshold are uploaded, the primary image gets a -scaled suffix (unchanged), alternate types get a -{mime extension}-scaled suffix
  • Similarly, alternate mime rotated images get a -{mime extension}-rotated suffix
  • New image meta sources array stores each mime type created for each full sized image and for each sub-sized image - both the file name (file) and size (filesize) are stored.
  • In src/wp-admin/includes/image.php::_wp_make_subsizes:
  • Using a mapping (from the input file mime type) provided by wp_upload_image_mime_transforms, output sub sized images in one or more output mime types
  • In src/wp-admin/includes/image.php
    • Add a new function wp_upload_image_mime_transforms as well as a matching filter named wp_upload_image_mime_transforms . Returns an array with the list of valid mime types that a specific mime type should be converted into.
    • In src/wp-includes/class-wp-image-editor-gd.php and src/wp-includes/class-wp-image-editor-imagick.php:
    • Update make_subsize to accept a new mime_type parameter that gets passed to the _save function when set (which already has a mime_type parameter). Add the new parameter to internal function calls as well.
  • In src/wp-includes/media.php: Adds new function wp_image_use_alternate_mime_types that is called for every image in wp_filter_content_tags. This function:
    • Adds a new filter wp_content_image_mimes that returns the mime type(s) to try to use for output, in the order they should be tried. The default order to try is array( 'image/webp', 'image/jpeg' ), meaning if a WebP version of the image exists, WordPress will use that image for the image tag output
    • Replaces all image components (including urls in the src and srcset attributes) when the same image is available in the preferred mime type
    • Returning an empty array will skip any image mime type substitution ( ie. add_filter( ‘wp_content_image_mimes’, ‘__return_empty_array’ );
  • In src/wp-includes/post.php::wp_delete_attachment_files:
    • Delete additional mime type images referenced in the image meta root ‘sources’ array as well as each sizes->sources array when present. This ensures that additional mime images are cleaned up if the original image is deleted.

PHPUnit test changes

New sources data added to expected results in test_wp_generate_attachment_metadata_pdf, test_crop_setting_for_pdf
Remove new wp_content_image_mimes filter for tests in tests/phpunit/tests/media.php

Todo items remain for this patch:

Attachments (17)

55443.diff (48.0 KB) - added by adamsilverstein 3 years ago.
55443.2.diff (53.7 KB) - added by adamsilverstein 3 years ago.
latest from PR
55443.3.diff (56.4 KB) - added by adamsilverstein 3 years ago.
55443.4.diff (56.9 KB) - added by adamsilverstein 3 years ago.
55443.5.diff (89.7 KB) - added by adamsilverstein 2 years ago.
latest from git
55443.6.diff (91.8 KB) - added by adamsilverstein 2 years ago.
55443.7.diff (91.5 KB) - added by adamsilverstein 2 years ago.
55443.8.diff (91.3 KB) - added by adamsilverstein 2 years ago.
55443-fix-secondary-scaled.diff (1.8 KB) - added by adamsilverstein 2 years ago.
55443-multi-mime-with-edit-and-restore-flows.diff (17.6 KB) - added by adamsilverstein 2 years ago.
55443-misnamed-variable.diff (762 bytes) - added by adamsilverstein 2 years ago.
55443-tests.diff (27.7 KB) - added by adamsilverstein 2 years ago.
WebP usage by WordPress version.jpg (333.5 KB) - added by adamsilverstein 2 years ago.
revert-53786-53848-r53847-r53845-r53751.diff (94.5 KB) - added by adamsilverstein 2 years ago.
reintroduce-webp-default-output.diff (9.3 KB) - added by adamsilverstein 2 years ago.
reintroduce-webp-default-output.2.diff (12.9 KB) - added by adamsilverstein 2 years ago.
reintroduce-webp-default-output.3.diff (14.3 KB) - added by adamsilverstein 2 years ago.

Download all attachments as: .zip

Change History (224)

This ticket was mentioned in Slack in #core-media by antpb. View the logs.


3 years ago

felixarntz commented on PR #2393:


3 years ago
#3

@adamsilverstein I just noticed something quite interesting and valuable for us in this implementation based on reviewing @kirtangajjar's code in https://github.com/WordPress/performance/pull/235#discussion_r833601640: We may be able to simplify the code here and get around adding the $mime_type parameter to make_subsize(). Furthermore, even with multi_resize() the feature would work correctly based on this.

Let me explain:

  • When we create the additional format for the original image, we call the save() method. That method updates the file class variable (and mime_type fwiw), which means subsequent images generated via the same editor instance (which is what we should do, following core's implementation) will use the same MIME type.
  • When you look at the get_output_format() method implementation, you'll see how that works. Any time that no MIME type or file name is specified (which is the case in make_subsize()), it'll fall back to whatever the original file is. And since we can generate the original image's additional format first, the editor instance than references that, so it'll be set to WebP already.

Let me know if this makes sense. We can use this to our advantage, this will bring our implementation more in line with existing core behavior and we don't need to adjust the image editor classes at all.

adamsilverstein commented on PR #2393:


3 years ago
#4

@adamsilverstein[](https://teams.googleplex.com/adamjs@google.com) As discovered and fixed by @mitogh in https://github.com/WordPress/performance/pull/259 for the Performance Lab plugin, we need to make sure to choose an editor that actually supports the format we're trying to create. That also needs to be fixed in this PR:

Good point, we could otherwise doing runs for unsupported mimes.

For every new format we're iterating through, get a new editor instance with passing the MIME type to it.

maybe we can initialize these all at the start, one editor for each mime type so we get an array of editors, one for the primary type and one for each additional mime type.

also realizing I should add a wp_image_editor_supports check in _wp_get_primary_and_additional_mime_types - excluding unsupported types there (while ensuring primary_mime_type is returned).

We should make sure we generate the sources primarily by format (i.e. outer loop is the MIME types, inner loop the formats), so that we can minimize the number of wp_get_image_editor() calls necessary.

This might be tricky to restructure that way in some places, however if we construct the editors we need ahead of time (one for each mime type), the loops can stay as is.

adamsilverstein commented on PR #2393:


3 years ago
#5

note: when calling wp_get_image_editor( $file, array( 'mime_type' => $output_mime_type ) ); I almost expect this to set the actual mime type for the returned editor, what do you think? currently this comes only from the file passed, but we could potentially override it if mime_type was also passed which would remove the need for our calls to set_mime_type.

adamsilverstein commented on PR #2393:


3 years ago
#6

We should make sure we generate the sources primarily by format (i.e. outer loop is the MIME types, inner loop the formats), so that we can minimize the number of wp_get_image_editor() calls necessary.

After looking further I agree this will be the best approach overall. similar to my original PR, we can have the first element in the array of types be the primary type and treat it differently only in that the data is stored in the traditional spot as well as the new spot.

Conveniently, this change also solves the file name issue I ran up against trying to reuse the same editor for all mime types.

felixarntz commented on PR #2393:


3 years ago
#7

@adamsilverstein

After looking further I agree this will be the best approach overall. similar to my original PR, we can have the first element in the array of types be the primary type and treat it differently only in that the data is stored in the traditional spot as well as the new spot.

Are you suggesting to bring them back all in one array? I think the approach of having the primary MIME type separate from additional MIME types should still work here and not conflict with the changed loop order. Not sure I understand what you mean.

adamsilverstein commented on PR #2393:


3 years ago
#8

Are you suggesting to bring them back all in one array? I think the approach of having the primary MIME type separate from additional MIME types should still work here and not conflict with the changed loop order. Not sure I understand what you mean.

Yes.

A single loop simplifies the code a good bit, since every image always gets the same treatment, we don't need to repeat the code that resizes or rotates the images during processing. The only difference with the primary type is how we store its data in the root and 'sizes' (and creating sizes if missing), otherwise all images are processes in the same way.

if we separate the primary and secondary types, we need to repeat the same code blocks for scale and rotation.

I tried this in `c5d6078` (#2393) (compare that to the previous https://github.com/WordPress/wordpress-develop/pull/2393#:~:text=this%20working%20in-,98ab980%20(%232393),-Reply...) The code is much simpler now, and we get a single editor initiated per mime type, so we'll get the right one (and file naming works correctly).

one issue i noticed: right now we are checking that the editor supports the target mime type, ideally we should check that it supports both the source and target mime. I think that is missing. For example, what happens if a user uploads a WebP and their system doesn't support WebP?

This ticket was mentioned in Slack in #core-media by adamsilverstein. View the logs.


3 years ago

@adamsilverstein
3 years ago

latest from PR

#10 @adamsilverstein
3 years ago

  • Description modified (diff)

#11 @adamsilverstein
3 years ago

55443.3.diff brings latest from PR.

Note for reviewers:

#12 @milana_cap
3 years ago

  • Keywords needs-docs needs-codex added

#13 @pbiron
3 years ago

In a comment on the Enabling WebP by default make/core post I asked "if a site already has a bunch of JPEGs in the media library, will regenerating the thumbnails (intermediate sizes) via either wp media regenerate or a plugin like Regenerate Thumbnails automatically produce the WebP versions?"

Just reporting here that I have now tested this with WP 5.9.2, Performance Lab 1.0.0-beta.3, WP-CLI 2.6.0, and Regenerate Thumbnails 3.1.5 and it all works as I expected it to.

Nice work!

#14 follow-up: @markhowellsmead
3 years ago

When converting medium-resolution photographs (approx 1600px – 2500px on the long edge), WebP files are often larger than the JPEG equivalent. (In my tests using my own photography, in around 60% of cases.) This change might make the “modern image format” test of Page Speed Insights happy, but enforcing WebP by default on sites which use a lot of photography will often cause longer image loading times.

#15 in reply to: ↑ 14 @eatingrules
3 years ago

Replying to markhowellsmead:

enforcing WebP by default on sites which use a lot of photography will often cause longer image loading times.

This is just one of many reasons why this feature should not be in core.

There is strong, legitimate opposition to this proposal being shared in the announcement blog post comments - I won't repeat them here, but I hope everyone will click over to see that discussion.

This ticket was mentioned in Slack in #core-media by seedsca. View the logs.


3 years ago

#17 @adamsilverstein
3 years ago

When converting medium-resolution photographs (approx 1600px – 2500px on the long edge), WebP files are often larger than the JPEG equivalent. (In my tests using my own photography, in around 60% of cases.) This change might make the “modern image format” test of Page Speed Insights happy, but enforcing WebP by default on sites which use a lot of photography will often cause longer image loading times.

Hey @markhowellsmead - thanks for the feedback. Are you comparing compression you achieve on your own desktop machine or on your WordPress site? WordPress may not show the same results for the sub-sizes it generates, since the compression relies on GD/Imagick which may not have the latest compression algorithms. If you are compressing, uploading and using the "original" uploaded image you may indeed be able to get better sizes with JPEG, however that isn't the way most images are used (instead, users upload large originals and WordPress compresses various sub sizes for display). Could you describe your workflow (and also what compression approach you use, is it MozJPEG?)?

Part of this proposal includes storing the generated file size, one thing we are considering is always choosing the smaller image (at display time), especially since compression effectiveness can vary by image and size.

Could you provide a sample original image we could test with where the WebP version is larger?

#18 follow-up: @markhowellsmead
3 years ago

https://www.dropbox.com/s/enndvyze8rzhgsy/20220328-DSCF1424.jpg?dl=0 at 757kb versus https://www.dropbox.com/s/enndvyze8rzhgsy/20220328-DSCF1424.webp?dl=0 at 1.2mb

Converted on the local command line and via SSH on the server using ImageMagick:

mogrify -format webp -quality 82 "$DIR/*.jpg"

Addendum: resizing down to 1024x768 on that image creates a file at 300kb if the result is a JPG, and 306kb if it's a WebP file.

Last edited 3 years ago by markhowellsmead (previous) (diff)

#19 @adamsilverstein
3 years ago

Thanks @markhowellsmead, I'll give your image a test using GD and Imagick conversion on the server to see how that works.

#20 @adamsilverstein
3 years ago

  • Milestone changed from 6.0 to Future Release

Changing Milestone to "future release" for now, follow up Make blog post in the works.

#21 @milana_cap
3 years ago

  • Keywords needs-user-docs added; needs-codex removed

felixarntz commented on PR #2393:


2 years ago
#22

@adamsilverstein Since you're OOO until next week, I went ahead and continued to work on this PR, revamping the logic to generate the images on upload quite a bit. This is a follow-up to my review from 2 days ago. Here's an explanation of the changes:

  • Introduce _wp_maybe_scale_and_rotate_image() function, to get a WP_Image_Editor instance and scale and rotate the input file as needed right from the beginning.
    • The function furthermore is mime type-aware, configuring the editor instance as needed.
    • _wp_get_image_suffix() is another tiny helper function that was introduced and is closely related to this.
  • Clearly separate the primary and additional mime types in _wp_get_primary_and_additional_mime_types() and everywhere that function is used.
    • The logic on how to treat primary vs additional mime types is different, and the new approach avoids all the "is index equal to zero" checks, making it easier to follow.
    • There are of course a lot of similarities in parts of the behavior, but those have now been outsourced to reusable functions to avoid duplicate code.
  • Add an optional $mime_type parameter to the wp_get_missing_image_subsizes() and _wp_make_subsizes() functions and tweak the functions to operate on a specific mime type.
    • This keeps the code quite a bit simpler, and the consuming logic can simply call these functions for every mime type.
  • Introduce _wp_make_additional_mime_types(), which is somewhat compareable to the existing _wp_make_subsizes() in that it will create the "full" versions of an image for all mime types provided.
    • This function is critical, as it fixes a problem in the previous implementation where all "full" images had to be successfully created in one go for the upload to work, effectively breaking the retry logic.
  • Add support for the retry mechanism, which was missing before, by integrating the different mime types into wp_update_image_subsizes().
    • This function now generates all missing sub-sizes for each mime type, and it also generates any missing "full" images, using the _wp_make_additional_mime_types() function explained above.
  • Update the WP_Image_Editor::get_output_format() implementation to treat a manually set $mime_type class property as a fallback to the $mime_type parameter.
    • This effectively means, as long as we call WP_Image_Editor::set_mime_type(), any files saved will by default use the correct mime type and extension.
  • Update all relevant @since version numbers to 6.1.0.

Let me know your feedback, or if you have any questions.

I have yet to test this code, there are probably a few quirks to address when testing. Once we have addressed those, we should really dive into adding unit test coverage.

felixarntz commented on PR #2393:


2 years ago
#23

Note to self and @adamsilverstein: We need to incorporate the fix from https://github.com/WordPress/performance/issues/358 into this PR as well.

adamsilverstein commented on PR #2393:


2 years ago
#24

Overall looks good, I like the code restructuring.

I brought the branch up to date with trunk resolving some merge conflicts. I will do some additional testing, particularly around the retry mechanism to validate those fixes.

This effectively means, as long as we call WP_Image_Editor::set_mime_type(), any files saved will by default use the correct mime type and extension.

makes sense - we should make sure this is documented in the dev note.

#25 in reply to: ↑ 18 @studiolxv
2 years ago

@adamsilverstein, First of all thank you so much to everyone here for helping bring webp (and hopefully .jxl upon global adoption) to WordPress.

That said, after installing Performance-Lab and running a few upload tests I've run into a dilemma where the 'full' size .webp is larger (201 KB) than my original pre-upload imagick converted .jpg (1920x1280 126KB using -quality 25). Even the new .jpg '1536x1536' size came out larger at (262 KB).

In the mogrify or magick tests I've run locally with -quality 80 or higher the results do not seem to offer the amount of compression to make the automatic conversion to .webp warranted upon upload. When I use imagick locally pre-upload I tend to compress to -quality <25-45> which seems to get the file size and quality in the range I'm pleased with.

Maybe I've missed something, but I have searched through the Performance-Lab plugin, I'm not finding where you set the quality... Will there be a filter for developers to adjust this conversion -quality <int> value for the 'Full' size and the other core or theme-set $subsize .webp output images? Is this a "one -quality size fits all $subsizes" scenario? Something to think about and I would love to hear your take on this.

It would be great if developers could adjust the quality per $subsize. As larger dimensions require lower -quality <int> and smaller dimensions require higher -quality <int> to safeguard against visible quality degradation.

#26 @adamsilverstein
2 years ago

@studiolxv to alter the WebP output quality you can use the wp_editor_set_quality filter.

It would be great if developers could adjust the quality per $subsize.

This is being proposed in https://core.trac.wordpress.org/ticket/54648 - the current filter doesn't really work per-size the way it is currently called, we ether need to call it again or add an additional filter.

adamsilverstein commented on PR #2393:


2 years ago
#27

@felixarntz I reviewed all of your comments above and added/verified the changes are in the current patch. I also fixed up some minor failing tests, all tests now pass.

I will do some additional testing with the updated code.

adamsilverstein commented on PR #2393:


2 years ago
#28

@felixarntz I created this PR to add filtering by size name which would enable more fine grained control over sub-sized image generation - https://github.com/adamsilverstein/wordpress-develop-fork/pull/25. Appreciate any feedback; I'll work on adding tests

adamsilverstein commented on PR #2393:


2 years ago
#29

A fix for the bug https://github.com/WordPress/performance/issues/358 which currently also applies here. I think a solid fix for this would be to add a file_exists check to the _save() method of the two core image editor class implementations, so that the file is not saved if one with the exact same path and name already exists. I think we could return an error in that situation. This is already in the plugin, so this is ready to be implemented here.

@felixarntz one potential issue would this will prevent overwriting existing images, for example during regenerating images. I'm not sure it is safe to change the behavior of _save which currently overwrites images.

One way to handle this more generically would be to add the original file extension into the generated filenames which most users never see anyway.

eg:
"cat.jpeg" -> "cat-jpeg.webp"
"cat.jpg" -> "cat-jpg.webp"

This would solve this issue raised in https://github.com/WordPress/performance/issues/358 without changing the behavior of _save. This should require only minimal other adjustments but mostly should just work since we store the filename and rely on that later (we don't assume the filename will be the same).

Another possible way to handle this would be to make the current checks we do for the existing file in wp_unique_filename more robust so it detected that cat.jpeg and cat.jpg were the "same" and renamed the second upload automatically. This would be pretty efficient and foolproof as well,:

eg:
"cat.jpeg" -> "cat.webp"
"cat.jpg" -> "cat-1.webp"

these filenames look nice, but the code would be messier.

What do you think?

adamsilverstein commented on PR #2393:


2 years ago
#30

Working on "The enhancement to only replace images generally when for sure in a frontend request context" and noting that _wp_post_thumbnail_html is not updated in this PR, since we are working on that elsewhere. We will want to ensure that only applies automatically in the front end context as well.

felixarntz commented on PR #2393:


2 years ago
#31

@adamsilverstein I prefer the second alternative that doesn't include a "double" extension in the file name. I'm not sure on the technical implications, maybe we can find a way to implement this in a clean way still. I'd suggest we try that.

adamsilverstein commented on PR #2393:


2 years ago
#32

The enhancement to only replace images generally when for sure in a frontend request context (see https://github.com/WordPress/performance/issues/379). The approach for that should be fairly clear based on the issue, so this is ready to be implemented here.

Added. For the helper function- I put it in media.php since we use it there, open to suggestions for a better location.

adamsilverstein commented on PR #2393:


2 years ago
#33

@adamsilverstein I prefer the second alternative that doesn't include a "double" extension in the file name. I'm not sure on the technical implications, maybe we can find a way to implement this in a clean way still. I'd suggest we try that.

I tried one approach in `93542af` (#2393) which works well for two images (cat.jpeg, cat.jpg), but not for 3 (cat.jpe, cat.jpeg, cat.jpg - yes ".jpe" is valid) because the -1 variation gets applied twice. I reverted for now, but can you take a look @felixarntz - this would be a pretty clean approach and handle most use cases (may need to be added in the rest media endpoint as well).

Ultimately I think we should apply probably go the route you suggested in _save, generating a unique filename there - we still need to figure out how overwriting will work though - perhaps we can introduce a filter? When running image regeneration users would generally want old images overwritten.

adamsilverstein commented on PR #2393:


2 years ago
#34

The tweak to only keep WebP / additional MIME type images if they are in fact smaller than the primary MIME type one, per size individually (see https://github.com/WordPress/performance/issues/372). The exact implementation is currently still being discussed in that issue. Once a path forward is clear, I think we could implement the plugin and core solution for it in parallel.

I reviewed the current work on 372 and put together this change: `c22ea9e` (#2393)

In my testing with the "edge case" images I have (from slack and https://github.com/WordPress/performance/issues/7) I can reproduce the full sized image being larger (and getting deleted), but so far all my sub sized images are smaller with WebP. I'm going to continue testing with various versions of PHP GD (also tried Imagick) to see if I can get a "larger" subsize image, if not I'll manually raise WebP quality to a very high to verify this works as expected. I'll also check the front end output to verify the correct images are being used.

Attaching some edge case images from previous posts for testing -

https://i0.wp.com/user-images.githubusercontent.com/2676022/178049570-5482be9f-4d0c-41c8-966c-cc4f845f8a62.jpg
https://i0.wp.com/user-images.githubusercontent.com/2676022/178049575-2e1459bc-18f1-4366-bc8a-61dbe0c7d058.jpg
https://i0.wp.com/user-images.githubusercontent.com/2676022/178049578-56bf6dc3-0898-43b4-9fd8-9b55b343fdf4.jpeg
https://i0.wp.com/user-images.githubusercontent.com/2676022/178049596-d6826bd1-f82a-4e54-b370-1bf34af0dc5f.jpg
https://i0.wp.com/user-images.githubusercontent.com/2676022/178049686-1ead79fe-7f33-4a26-9cc0-ada7a780addf.jpeg

adamsilverstein commented on PR #2393:


2 years ago
#35

Quick note on quality settings:

In order to test larger WebP I wrote an mu plugin to set the quality based on mime.

{{{php
add_filter( 'wp_editor_set_quality', function( $quality, $mime_type ) {

if ( 'image/webp' === $mime_type ) {

return 99;

}
return $quality;

}, 10, 2 );
}}}

Unfortunately it didn't work as expected, set_quality was never called with the mime type correctly.

This diff made the conversion work for me, not sure that is right but something isn't quite working with quality settings currently $mime_type is always === $this->mime_type -

{{{diff
diff --git a/src/wp-includes/class-wp-image-editor.php b/src/wp-includes/class-wp-image-editor.php
index 6b63d7e9fd..e17b787223 100644
--- a/src/wp-includes/class-wp-image-editor.php
+++ b/src/wp-includes/class-wp-image-editor.php
@@ -414,12 +414,10 @@ abstract class WP_Image_Editor {

$filename = trailingslashit( $dir ) . wp_basename( $filename, ".$ext" ) . ".{$new_ext}";

}


  • if ( $mime_type && ( $mime_type !== $this->mime_type ) ) {
  • The image will be converted when saving. Set the quality for the new mime-type if not already set.
  • if ( $mime_type !== $this->output_mime_type ) {
  • $this->output_mime_type = $mime_type;
  • $this->set_quality();
  • }

+ if ( $mime_type && ( $mime_type !== $this->output_mime_type ) ) {
+ The image will be converted when saving. Set the quality for the new mime-type.
+ $this->output_mime_type = $mime_type;
+ $this->set_quality();

} elseif ( ! empty( $this->output_mime_type ) ) {

Reset output_mime_type and quality.
$this->output_mime_type = null;

}}}

setting the

adamsilverstein commented on PR #2393:


2 years ago
#36

Setting the WebP quality to 99 / jpeg to 82 I can verify that no WebP images are saved (other than the full size which was still smaller). testing various values above the current 86 (which may need lowering a bit) and below 99, I found some quality/image combinations where only specific sizes were kept in WebP versions, which is what is expected:

https://i0.wp.com/user-images.githubusercontent.com/2676022/178062181-565c3d16-d787-4846-8242-8ef30a483336.png

Then, inserting that image into a post and checking the source code, I can see that the srcset is properly built from the available, smaller images (which isn't part of the PR previously):

https://i0.wp.com/user-images.githubusercontent.com/2676022/178062723-e069e9d0-a9fd-41c8-a4ae-669db18fa654.png

adamsilverstein commented on PR #2393:


2 years ago
#37

@felixarntz - I pushed a change to ensure the image files for secondary mimes get unique names in `1711d2b` (#2393)

Essentially followed your suggestion to add to the "save" function which wound up requiring adding in several other places that call "save" and adding some explicit checks:

  • Added a new $unique parameter to WP_Image_Editor::generate_filename
  • Added a new $unique parameter to each editor's make_subsize, multi_resize, save and _save functions.
  • When generating additional mime full size images, explicitly check if the destination file exists and if so, generate a new unique name. The explicit check is required, otherwise the first image is name "image-1.webp" even if "image.webp" doesn't exist
  • When generating sub-sizes, also perform an explicit check for an existing file, otherwise core will always add a "-#" suffix to all sub-sizes, even if the destination file doesn't exist - because of this code: https://github.com/WordPress/wordpress-develop/blob/5c041f5777689afe49cf99dd7c897f6b81d248c7/src/wp-includes/functions.php#L2553-L2555

I am attaching my three "cat" images here for testing. If you upload all three files, you will see that each webp gets a unique name. Because the images are different dimensions, resized images where the name does not exist already will get the regular .webp externsion. For cropped images like the thumbnail size where the fie names would match, the inter is appended(-1.web etc).

Three files named "cat.jpg", "cat.jpe" and "cat.jpg" for testing: cats.zip

Files generated after uploading all three cat images (in the order generated):

https://i0.wp.com/user-images.githubusercontent.com/2676022/178601037-60a14c2d-d1ab-4f86-9b10-dd827bef785e.png

In this example, you can see images that all have the same sub-size names - each webp gets a different extension:

https://i0.wp.com/user-images.githubusercontent.com/2676022/178601304-4b38d517-93da-4004-ac7c-dae7713ceeac.png

Images that wind up with different sub size dimensions and thus different names, don't need the "-1" to be uniquely named:

https://i0.wp.com/user-images.githubusercontent.com/2676022/178601534-df7d024b-6fce-4eb4-85d6-1a9b98c6fe38.png

felixarntz commented on PR #2393:


2 years ago
#38

@adamsilverstein I see you force-pushed a bunch of times. Please don't force-push, it makes collaboration really challenging. Also, possibly because of that now the commit you're referring to above (https://github.com/WordPress/wordpress-develop/pull/2393/commits/1711d2b1bff343e394825fed279f51e59ed60899) can no longer be found.

adamsilverstein commented on PR #2393:


2 years ago
#39

Apologies. Force pushed right after the initial push to correct a phpcs issue and didn't realize that would cause an issue. I'll fix the link.

#40 @MatthiasReinholz
2 years ago

So if I understand right, the current approach is to make WebP the default image format for all Core image sizes, correct?

Even though there seems to be a filter to opt-out, I don't believe this is the most efficient approach.

As we can see in the discussions on the various posts and comments on this topic, WebP always requires the original JPG files to be available as a fallback. Hence, we are generating duplicate files (same content, another format). This may not be an issue for small websites. But if you are managing websites with GB and TB of media, this is a factor. Further, if we take into account the global reach of WordPress with smaller websites the aggregated additional resources for this conversion are tremendous.

From a sustainability perspective, duplicate files and unnecessary computational load cannot be the desired approach for the entire WordPress ecosystem.

One could argue that we'd save resources by transmitting smaller image files. My conclusion would still not be to convert to WebP by default but to add compression to existing file formats at first.

Based on various tests and all the comments across these posts here, I don't see a vast advantage of WebP over MozJPEG JPGs and compressed PNGs. WebP is actually even larger than compressed PNGs in many cases. MozJPEG may not be a realistic solution due to the technical implications (see https://github.com/WordPress/performance/issues/294).

Still, I'd imperatively raise the question if WebP conversion should be a Core feature (plus activated by default). We already have a vibrant set of plugins about image optimization (https://wordpress.org/plugins/search/image+optimization/). Many of them are already taking care of WebP.

Every single plugin in the directory utilizing MozJPEG compression plus PNG compression (and no WebP conversion) will generate competitive (IMHO better) results than the currently planned Core solution. This is because the file sizes will be comparable but we don't generate duplicates.

Image optimization is an important topic for page performance. However, we will not be able to change JPG and PNG as the default formats for content creation (users will upload files in those formats in almost all cases). WebP support still (and most likely always will) require a fallback file. Further, there are other next-gen formats coming up that are even more efficient than WebP (e.g. AVIF).

I definitely see it as a big advantage to add Core support for additional MIME types for sub-sized image files. But I can't see adding conversion to a specific other file format as preferred behavior. This may help to optimize the market position of WebP but it will also be a serious threat to plugin authors and existing larger websites that do not pay attention to this change.

Therefore, I'm questioning why this functionality should be activated by default at this stage. IMHO, it should be opt-in only. Plus ideally, we would already start to think about adding further image formats to be supported by this feature.

adamsilverstein commented on PR #2393:


2 years ago
#41

I would advise that we don't try to tweak the naming logic here and rather just bail in case a conflict with an existing file from another attachment arises.

Ok, that makes sense and simplifies what we need to accomplish. Also agree that this is an edge case so dropping the additional image should be an issue. I reverted the changes from yesterday and will try this simpler approach.

adamsilverstein commented on PR #2393:


2 years ago
#42

@felixarntz - in https://github.com/WordPress/wordpress-develop/pull/2393/commits/8eedc381d3ecc6344423ed87561b9c673162ed9c I used your original suggestion of adding the check into each image classes _save function.

  • When a matching image is found that doesn't belong to the same attachment, it will be skipped
    • when skipping, the save has "failed" and we return a WP_Error so the image size is skipped in calling functions
  • matching images that are part of the same attachment are still overwritten which should make image regeneration work as expected (I haven't tested yet though)
  • Adding here means this works for subsizes as well as the full size
  • We don't need to worry about naming conflicts for the full sized file - WordPress already ensures this has a unique name and this continues to work
  • the lookup to find the attachment ID is a bit tricky/expensive in the image editor context because we have to do a reverse lookup by URL. Fortunately, this only happens when there is a naming conflict already so should be generally rare

Q: Should we consider a filter to control exact behavior?

ps. I noticed _wp_maybe_scale_and_rotate_image doesn't seem to be returning the scales suffix for large images, I'm going to look into this next (and check for -rotated as well.

#43 follow-up: @adamsilverstein
2 years ago

Hey @MatthiasReinholz, thanks for the feedback. Responses inline below:

So if I understand right, the current approach is to make WebP the default image format for all Core image sizes, correct?

Correct.

additional resources for this conversion are tremendous.

I'm not sure I agree. Resources for generating images when you upload an image will increase dramatically, however resources to serve an image will be lowered. Since image uploading is very rare compared to image serving, the extra effort to compress and store images should be worth it.

Every single plugin in the directory utilizing MozJPEG compression plus PNG compression (and no WebP conversion) will generate competitive (IMHO better) results than the currently planned Core solution. This is because the file sizes will be comparable but we don't generate duplicates.

I'm not sure what you mean by "MozJPEG compression plus PNG compression" - isn't PNG a separate format?

MozJPEG is not widely available on PHP servers whereas WebP is. WebP offers additional benefits over MozJPEG compression/quality wise and also offers transparency and a lossless mode so it has the potential to replace transparent PNGs for example. Additionally we are building hooks in so existing plugins can control and integrate with the core feature.

Further, there are other next-gen formats coming up that are even more efficient than WebP (e.g. AVIF).

Absolutely and this feature is built with those potential formats in mind. Still, major browsers still don't support AVIF (Safari has announced support in their next release, hopefully Edge will follow suit). In addition, AVIF support isn't available in PHP until 8.1. Future potential improvements aren't a good reason to not take advantage of the significant improvements available in WebP now.

it will also be a serious threat to plugin authors and existing larger websites that do not pay attention to this change

this is a significant change and I agree large sites and plugin authors need to pay attention to these changes to choose the best path forward. I'm not sure about the threat to plugins - the team reviewed many image/optimization plugins and WebP is typically a very small part of what they offer.

Therefore, I'm questioning why this functionality should be activated by default at this stage. IMHO, it should be opt-in only.

Appreciate the concern (although I don't completely agree). We are starting with the change only for core image sizes in part as a way to be cautious about impact and give the ecosystem time to adapt (eg. so plugins or themes that add custom sizes can decide if they want secondary mime output)

Plus ideally, we would already start to think about adding further image formats to be supported by this feature.

+1 this is already underway in the Performance Lab plugin where we have open tickets exploring AVIF and JPEGXL support, as well as adding the picture element to enable serving fallback images.

#44 in reply to: ↑ 43 @eatingrules
2 years ago

Replying to adamsilverstein:

We are starting with the change only for core image sizes in part as a way to be cautious about impact and give the ecosystem time to adapt...

Making these new features opt-in instead of opt-out would be the best way to be cautious about potential impacts.

There have been many requests for this to be opt-in (as well as some asking for a setting on the Media page, rather than only a filter for developers). So far there hasn't been any open conversation about why that's not being taken into consideration.

Perhaps there could be a proposal/conversation to solicit feedback from the community about the best way to roll this out?

adamsilverstein commented on PR #2393:


2 years ago
#45

@felixarntz In `f41c83f` (#2393) - I switched the approach used to ensuring unique filenames, removing the changes from there _save functions. When generating filenames in generate_filename, if the passed extension doesn't match the main image extension, the original extension is added to the image suffix. In addition, I added the extension to the call to "generate_filename" when generating full sized images in _wp_make_additional_mime_types.

For example, when uploading "cat.jpg", the full size WebP will be named "cat-jpg.webp". A 150px square thumbnail will be named "cat-150x150.jpe", the WebP version will be named "cat-150x150-jpe.webp". By adding this suffix to the name, we neatly avoid nearly all name conflicts, specifically those that occur when uploading several files with the same name and different extensions (ie. cat.jpg, cat.jpe and cat.jpeg). Although the file names get a little longer and are slightly different than the original, it is worth it to avoid the potential conflicts, and usage will be generally unaffected since the names stored in image meta will be correct and are the single source of truth.

felixarntz commented on PR #2393:


2 years ago
#46

@adamsilverstein Sounds great, let's make sure to test this more. Looks like PHPUnit tests are failing, could you look into that? Maybe you can also start working on tests for that part.

I will start working on test coverage for:

  • wp_image_use_alternate_mime_types()
  • wp_upload_image_mime_transforms()
  • _wp_get_primary_and_additional_mime_types()
  • _wp_filter_image_sizes_additional_mime_type_support()
  • _wp_maybe_scale_and_rotate_image()
  • _wp_get_image_suffix()

Will see how far I get there, once we have those we should eventually also have some tests for _wp_make_additional_mime_types() and additional coverage for the new parts of _wp_make_subsizes() and wp_get_missing_image_subsizes().

felixarntz commented on PR #2393:


2 years ago
#47

@adamsilverstein In https://github.com/WordPress/wordpress-develop/pull/2393/commits/7eba89f75e42bdc11ac2b1e90602346913547d2b I've added tests for wp_image_use_alternate_mime_types() - quite useful already, since I uncovered a bug and several minor enhancements in its implementation, which are now fixed. Prior to the commit, the function would not replace any JPG with WebP even if only the full / original didn't have WebP available, which is of course possible - so in that case it needs to be ensured that the sub-sizes are still replaced with WebP.

felixarntz commented on PR #2393:


2 years ago
#48

@adamsilverstein

A few minor enhancements are included, e.g. sanitization for the filter value from wp_upload_image_mime_transforms.

adamsilverstein commented on PR #2393:


2 years ago
#49

@felixarntz I'm going to look into the failing tests here and also work on adding some tests for _wp_maybe_scale_and_rotate_image() and _wp_get_image_suffix() which I don't think you worked on yet. Tests will help validate a small change to get the -scaled size working `e16256e` (#2393)

felixarntz commented on PR #2393:


2 years ago
#50

@adamsilverstein There's just one more failure on PHP 5.6 now, which I _think_ may be fixed via https://github.com/WordPress/wordpress-develop/pull/2393/commits/8e68e6a38f9afc5f6a1f6a961de9f77cbc2fdb87 - in any case, the lack of array_values there was problematic since it could mean the array would not be properly indexed (e.g. index 0 could be missing).

adamsilverstein commented on PR #2393:


2 years ago
#51

  • Added tests for _wp_get_image_suffix
  • Added tests for _wp_maybe_scale_and_rotate_image

Also:
Fix a bug that caused secondary WebP images to not get the rotation attribute of the original image because we reset orientation after creating the -rotated version (originally from https://github.com/WordPress/wordpress-develop/blob/a855e85c77f3cc978e960321e0c7a63fa808b9c1/src/wp-admin/includes/image.php#L314-L317) to indicate that the full sized image is now oriented correctly (1 means not rotated). After trying a few different approaches I settled on: when generating the additional mime types, the orientation meta is temporarily restored so the image rotation works as expected.

To test: upload the rotated image test-image-rotated-90ccw.jpg from this PR and preview the generated images to verify that all images have the correct "rotated" orientation (with the part marked "Top" at the top).

@adamsilverstein
2 years ago

latest from git

felixarntz commented on PR #2393:


2 years ago
#52

@adamsilverstein In https://github.com/WordPress/wordpress-develop/pull/2393/commits/da8fe016f9fc6a0ed12fb2b71eb8e3a3f18c37a2 I've added tests for _wp_filter_image_sizes_additional_mime_type_support().

Please review my feedback in https://github.com/WordPress/wordpress-develop/pull/2393#pullrequestreview-1043716923 and https://github.com/WordPress/wordpress-develop/pull/2393#pullrequestreview-1044049704. I think once we've addressed those, this PR should be good to go. You might want to try to add some tests for wp_delete_attachment() to cover the new parts so that it does delete the files for the additional sources as expected, but I guess this is not a strict necessity since that function didn't really have tests before either.

adamsilverstein commented on PR #2393:


2 years ago
#53

You might want to try to add some tests for wp_delete_attachment() to cover the new parts so that it does delete the files for the additional sources as expected

I see we do have it_should_remove_the_generated_webp_images_when_the_attachment_is_deleted in tests/phpunit/tests/image/editor.php (and I see some since tags need updating in there). Easy to miss this file as GitHub has it collapsed by default! I will check if this can be expanded at all.

This ticket was mentioned in Slack in #core-media by adamsilverstein. View the logs.


2 years ago

adamsilverstein commented on PR #2393:


2 years ago
#55

regarding reading EXIF data, I found another instance we we have a filterable list of types which support reading EXIF data; I wonder if we should use the same list (plus WebP) here, looks like tiff image which we support also include EXIF data:

https://github.com/WordPress/wordpress-develop/blob/57f4d6c7b27db2705721d345fd6bc0927528c663/src/wp-admin/includes/image.php#L833-L843

adamsilverstein commented on PR #2393:


2 years ago
#56

@felixarntz in https://github.com/WordPress/wordpress-develop/pull/2393/commits/253e9086a4023fc7daf6a72401ed1f199e08a122 I tried switching from set_mime_type to set_output_mime_type. Because we are only setting the output type now, the main image mime data remains the original type, so we no longer need to remove the exif check only for jpeg part. I'll do more testing to make sure this works as expected (so far, so good).

adamsilverstein commented on PR #2393:


2 years ago
#58

Addressing final feedback and looking into failing test, the output_mime_type change is breaking the test_set_quality_with_image_conversion test and I'm digging into what the best fix is.

adamsilverstein commented on PR #2393:


2 years ago
#59

Fix for the final test issue in `ebe3ad4` (#2393)

#60 @adamsilverstein
2 years ago

55443.7.diff reflects the latest small changes from the PR, mostly improving doc blocks.

#61 @adamsilverstein
2 years ago

  • Keywords commit added; needs-testing removed

This feels ready and - pending any remaining feedback on the patch from contributors (especially component maintainers) - I'm planning to commit this change tomorrow (Mountain time).

#62 follow-ups: @adamsilverstein
2 years ago

Hey Andrew thanks for your feedback and your continued tracking of this feature.

Making these new features opt-in instead of opt-out would be the best way to be cautious about potential impacts.

The feature will have widespread benefits for users by opting in core sizes (to start) - if it were entirely opt-in it would have little impact - or benefit.

There have been many requests for this to be opt-in (as well as some asking for a setting on the Media page, rather than only a filter for developers). So far there hasn't been any open conversation about why that's not being taken into consideration.

Perhaps there could be a proposal/conversation to solicit feedback from the community about the best way to roll this out?

We have discussed both suggestions in chats and issues with mixed responses. Project philosophy is regularly mentioned as aligning with the current approach.

I would suggest opening a trac ticket to propose adding a UI element after the feature has landed which would be a good place to discuss and mock up what the UI would look like.

Similarly I would decouple the conversation around opt in (or what the "default" should be instead of all core image sizes) since that is a separate question from whether there should be a UI in core. Open a separate ticket to discuss that if you feel that the default should be different.

#63 in reply to: ↑ 62 @feastdesignco
2 years ago

It's extremely clear from the community that this feature is not wanted. The response to this has been unanimously negative.

Suggesting we should "open a ticket" after this is forced out despite near total rejection is ham-fisted and demonstrates how little the community feedback actually means to WordPress.

Whether there should be opt-in or opt-out - and how that's implemented - is core to managing this, before it's released, not after.

Suggesting it gets de-coupled is a cheap tactic to force the release of this rather than building it properly in the first place.

This is not ready for production in its current form.

Replying to adamsilverstein:

Hey Andrew thanks for your feedback and your continued tracking of this feature.

Making these new features opt-in instead of opt-out would be the best way to be cautious about potential impacts.

The feature will have widespread benefits for users by opting in core sizes (to start) - if it were entirely opt-in it would have little impact - or benefit.

There have been many requests for this to be opt-in (as well as some asking for a setting on the Media page, rather than only a filter for developers). So far there hasn't been any open conversation about why that's not being taken into consideration.

Perhaps there could be a proposal/conversation to solicit feedback from the community about the best way to roll this out?

We have discussed both suggestions in chats and issues with mixed responses. Project philosophy is regularly mentioned as aligning with the current approach.

I would suggest opening a trac ticket to propose adding a UI element after the feature has landed which would be a good place to discuss and mock up what the UI would look like.

Similarly I would decouple the conversation around opt in (or what the "default" should be instead of all core image sizes) since that is a separate question from whether there should be a UI in core. Open a separate ticket to discuss that if you feel that the default should be different.

#64 in reply to: ↑ 62 @eatingrules
2 years ago

Replying to adamsilverstein:

I would suggest opening a trac ticket to propose adding a UI element after the feature has landed which would be a good place to discuss and mock up what the UI would look like.

Similarly I would decouple the conversation around opt in (or what the "default" should be instead of all core image sizes) since that is a separate question from whether there should be a UI in core. Open a separate ticket to discuss that if you feel that the default should be different.

I think the two things are actually interrelated, especially since a UI feature will help encourage adoption while further empowering users to control their sites.

But to start I just created #56263 to help facilitation the discussion about whether or not it should be enabled/disabled by default.

#65 @adamsilverstein
2 years ago

to start I just created #56263 to help facilitation the discussion

Thanks

#66 follow-up: @kirasong
2 years ago

I'm reviewing the patch now, and will leave further notes if there is time before it's committed.

My testing from the plugin so far has worked as expected.

I wanted to leave a note that I support this particular implementation / method, and think it's great that there has been as much community-wide discussion about it so far. It's made the feature much better than it would have been a cycle ago!

I also had the concern about WebP being larger in some cases (which was confirmed during image quality testing), and dropping the images that are larger solves this concerns.

It's worth noting that when WebP is smaller, which in my testing so far is common for smaller resolutions even when the largest sizes are bigger, the savings percentage-wise are quite significant.

I think this feature has the chance to make a big impact on load times and bandwidth on the web at large.

It is a big change, and I support landing it early, as is currently planned, so it can get thorough testing in core.

#67 @adamsilverstein
2 years ago

Thanks for the review and support @mikeschroder! I am also excited to get this in early and am glad to hear everything works as expected in your testing. I pushed one final change based on your feedback and update the code here in 55443.8.diff.

#68 @adamsilverstein
2 years ago

In 53751:

Media: enable generating multiple mime types for image uploads; specifically WebP versions for JPEG images by default.

This changeset adds the capability for core media uploads to generate sub sized images in more than a single mime type. The output formats for each mime type can be controlled through a filter. WebP is used as an additional output format for JPEG images by default to improve front end performance.

When generating additional mime types, only images which are smaller than the respective original are retained. By default, additional mime type images are only generated for the built-in core image sizes and any custom sizes that have opted in.

Image meta is updated with a new 'sources' array containing file details for each mime type. Each image size in the 'sizes' array also gets a new 'sources' array that contains the image file details for each mime type.

This change also increases image upload retries to accommodate additional image sizes. It also adds a $mime_type parameter to the wp_get_missing_image_subsizes function and filter.

This change adds three new filters to enable full control of secondary mime image generation and output:

  • A new filter wp_image_sizes_with_additional_mime_type_support that filters the sizes that support secondary mime type output. Developers can use this to control the output of additional mime type sub-sized images on a per size basis.
  • A new filter wp_upload_image_mime_transforms that filters the output mime types for a given input mime type. Developers can use this to control generation of additional mime types for a given input mime type or even override the original mime type.
  • A new filter wp_content_image_mimes which controls image mime type output selection and order for frontend content. Developers can use this to control the mime type output preference order for content images. Content images inserted from the media library will use the available image versions based on the order from this filter.

Thanks to the many contributors who helped develop, test and give feedback on this feature.

A haiku to summarize:

Upload a JPEG
Images of all sizes
Output as WebPs

Props flixos90, MatthiasReinholz, studiolxv, markhowellsmead, eatingrules, pbiron, mukesh27, joegrainger, mehulkaklotar, tweetythierry, akshitsethi, peterwilsoncc, eugenemanuilov, mitogh, shetheliving, clarkeemily, codekraft, mikeschroder, clorith, kasparsd, spacedmonkey, trevorpfromsandee, jb510, scofennellgmailcom, seedsca, cagsmith, karinclimber, dainemawer, baxbridge, grapplerulrich, sobatkras, chynnabenton, tonylocalword, barneydavey, kwillmorth, garymatthews919, olliejones, imarkinteractive, jeffpaul, feastdesignco, webbeetle, masteradhoc.

See #55443.

#69 @adamsilverstein
2 years ago

r53751 lays the foundation for multi-mime support; I am leaving the ticket open for several other patches (which are mostly complete already) to:

  • Add the WebP shim (which uses the jpeg instead) for non-supporting browsers
  • Add plugin integration points
  • Add additional mime type output during media edit flow
  • Add new media meta data in REST API media endpoint

One more piece that probably deserves a separate ticket is:

  • Add control over mime type support in add_image_size

and also reviewing other core media functions to see where adding a mime_type parameter makes sense.

Last edited 2 years ago by adamsilverstein (previous) (diff)

#71 @jorbin
2 years ago

#56263 was marked as a duplicate.

#72 @ocean90
2 years ago

  • Milestone changed from Future Release to 6.1

This ticket was mentioned in Slack in #core by jorbin. View the logs.


2 years ago

#74 @jorbin
2 years ago

#56263 was marked as a duplicate.

#75 in reply to: ↑ 66 @jb510
2 years ago

Replying to mikeschroder:
[...]

I also had the concern about WebP being larger in some cases (which was confirmed during image quality testing), and dropping the images that are larger solves this concerns.

It's worth noting that when WebP is smaller, which in my testing so far is common for smaller resolutions even when the largest sizes are bigger, the savings percentage-wise are quite significant.

This seems a little short-sighted to me. While I agree that we should serve the format with the smallest file size by default, it seems to me like the generated WebP, even if larger, ought to be retained for future use regardless of file size.

I really don't want a library that looks like
image-original.jpeg
image-1000x1000.jpeg
image-1000x1000.webp
image-900x900.webp
image-800x800.jpeg
image-700x700.jpeg
image-700x700.webp
image-600x600.jpeg

Where only some themnuals have sidecar webp files. Predictability is important because of how the media library works.

I for one look forward to deleting every jpeg on my site someday in the future, and wouldn't want holes where webp's weren't saved. I could care less if 1% of the files ended up being served in a format that was 5% bigger, It'd rather have the consistency to work with when managing them in bulk.

And again - this all should be opt-in on existing installs for at LEAST several versions of WP.

#76 @eatingrules
2 years ago

Here's something that we discovered just yesterday on a new (to us) client's site. I raise this because it's a very typical example of what we might see with our clients.

This is a very popular blog (tens of millions of pageviews/month), that has been around for many years. They have over 16,000 images in the media library, and their uploads folder is currently >25GB.

They are supremely talented at creating great content, but they are not "tech savvy" -- thankfully, they have the budget for an agency (and just started working with ours). They had previously been been working with a different, well-known agency. They are hosted on WP Engine.

We were having trouble getting backups to complete successfully (we use ManageWP and Code Guard). In digging deeper, we discovered that the site has many WebP images as duplicates of their JPGs. They're using Imagify to create the WebP images and add the <picture> tags.

Additionally, most of the WebP files we looked at are significantly larger (about 2-3x on average) than the original JPG.

I've downloaded a few samples from recent blog posts:

https://p128.p0.n0.cdn.getcloudapp.com/items/Wnu7WE8Y/99251237-10e5-4ca5-8409-90cbe69879e3.jpeg

The site owner had no idea the site was creating WebP images (even though it was "opt-in" on the Imagify settings! Someone must have turned that on for them at some point), and they certainly did not know the WebP images were larger than the JPGs.

(They also would not know how to add a filter to opt-out.)

We're now in the process of disabling the WebP functionality, and we will be deleting all the WebP files once we've confirmed all is well without them, so that our backups can work more reliably. (We'll then add in Cloudflare Polish instead.)

I know the plan is to discard WebP images if they're larger than the JPG (though see @jb510's point above)... but my point here is that the client was completely oblivious that this was happening -- and they had the advantage of having a support team to help them, on one of the most popular WP hosting services.

Now imagine the mess that millions of site owners are going to find themselves in when this is enabled by default without their knowledge... and without any clear way to even disable it.

Last edited 2 years ago by eatingrules (previous) (diff)

This ticket was mentioned in PR #3025 on WordPress/wordpress-develop by mukeshpanchal27.


2 years ago
#77

The PR adds new source data to the media REST API endpoint.

Trac ticket: https://core.trac.wordpress.org/ticket/55443

felixarntz commented on PR #3025:


2 years ago
#78

@mukeshpanchal27 I've added the Trac ticket link to https://core.trac.wordpress.org/ticket/55443 since that should be committed as part of that ticket, as a follow-up to https://core.trac.wordpress.org/changeset/53751.

This ticket was mentioned in Slack in #hosting-community by javier. View the logs.


2 years ago

This ticket was mentioned in Slack in #hosting-community by javier. View the logs.


2 years ago

#81 @flixos90
2 years ago

In 53786:

Media: Add information about additional MIME type sources to attachments REST endpoints.

This changeset is a follow-up to [53751] which ensures the additional sources information stored in attachment metadata are available under media_details for each image size in the REST API responses for attachments.

Props mukesh27, eugenemanuilov, mitogh, flixos90, aaemnnosttv.
See #55443.

felixarntz commented on PR #3025:


2 years ago
#82

Committed to trunk in https://core.trac.wordpress.org/changeset/53786.

I made just one small tweak I realized last minute: It's safer to use the actual image URL from wp_get_attachment_image_src() and replace the relevant file name with the file information. This is similar to how image_downsize() does it, so we should use that here rather than dirname().

This ticket was mentioned in PR #3030 on WordPress/wordpress-develop by mehulkaklotar.


2 years ago
#83

This PR adds support for multiple mime types for image edit and restore. When image is deleted, it deletes all related files for that image, when it was edited or restored.

Trac ticket: https://core.trac.wordpress.org/ticket/55443

This ticket was mentioned in Slack in #hosting-community by jadonn. View the logs.


2 years ago

adamsilverstein commented on PR #3030:


2 years ago
#85

@mehulkaklotar - I pushed a few changes to the branch and did more testing.

Please review the naming as well as testing with the 'Thumbnail" control:

https://i0.wp.com/user-images.githubusercontent.com/2676022/181365878-e39d04ab-7221-4954-a036-d7cfd90e105f.png

In my testing I noticed when selecting "thumbnail only" the generated WebP wasn't the right size (it was full sized vs being a thumbnail). I fixed that in `bb870b9` (#3030). It would be good to do additional testing with the "All sizes except thumbnail" option which I didn't test carefully yet.

Regarding the issue deleting, that was related to my previous code changes, on this new branch all meta is stored correctly and all files are deleted correctly.

This ticket was mentioned in PR #3031 on WordPress/wordpress-develop by adamsilverstein.


2 years ago
#86

Correct an issue where the secondary mime image would not be properly -scaled.

Test by uploading a large image and checking the full size generated WebP.

Trac ticket: https://core.trac.wordpress.org/ticket/55443

#87 @adamsilverstein
2 years ago

55443-fix-secondary-scaled.diff (also https://github.com/WordPress/wordpress-develop/pull/3031) is a small patch to fix an issue where the secondary mime type would not get the correct "scaled" treatment because the meta sizes passed were those returned after the original image was "scaled" (resized). The patch corrects this by temporarily setting the original meta's size (width,height) values before calling _wp_make_additional_mime_types.

This ensures the secondary mime type image (eg WebP) is properly resized and gets the -scaled name suffix. We already do the same for the exif "Orientation" attribute to ensure secondary mime images are properly rotated.

mehulkaklotar commented on PR #3030:


2 years ago
#88

@adamsilverstein I did some additional testing on latest changes. Thumbnails are now correct size. Tested "All sizes except thumbnail" as well which works correctly.

#89 @adamsilverstein
2 years ago

55443-multi-mime-with-edit-and-restore-flows.diff is based on this PR from mehulkaklotar.

This patch adds secondary mime type generation when editing an image in the media library. It subsequently handles the restore functionality and ensures backup files are correctly deleted when an edited images is deleted.

Editing images in Gutenberg already works as expected since a new image is created each time: the built in secondary mime generation works as usual.

Worth noting that core does not currently have a "retry" mechanism for image generation during edits (only for uploads), something we want to fix!

Last edited 2 years ago by adamsilverstein (previous) (diff)

This ticket was mentioned in PR #3036 on WordPress/wordpress-develop by adamsilverstein.


2 years ago
#90

Trac ticket:

This ticket was mentioned in PR #3034 on WordPress/wordpress-develop by mukeshpanchal27.


2 years ago
#91

The PR adds fallback JPEG images in the frontend when WebP is not supported by the browser.

Trac ticket: https://core.trac.wordpress.org/ticket/55443

#92 @isaumya
2 years ago

I also think that this feature should be something that is opt-in by users instead of opt-out.

Because at this point most website have some image optimization / distribution flow added to them.

For example we use Cloudflare Polish on all of our client sites to ensure all the images that are being provided to the end users are fully optimised and if the browser supports, also delivered in WebP.

Now other sites might use other tools like these - imagekit, optimole, short pixel, ewww image optimozer and so on.

The point I'm trying to make is that all of these websites which already got a proper flow of things in place are not going to take advantage of this. Instead it would be a huge extra burden to them in terms of hosting space cost

But instead of a filter if there is a proper settings for this in the WP Media Settings page which would be disabled by default and then users who want to take advantage of this can enable - that would be the best

Otherwise before 6.1 rollout we all have to go in to all of these sites and add the filter. I hope what I tried to say makes sense.

This ticket was mentioned in Slack in #core-media by adamsilverstein. View the logs.


2 years ago

#95 @adamsilverstein
2 years ago

55443-misnamed-variable.diff fixes an issue opened by @mehulkaklotar in this PR.

#96 @adamsilverstein
2 years ago

In 53847:

Media: correct a misnamed variable when creating sub sizes with multi_resize.

Props mehulkaklotar, mukesh27.
See #55443.

#97 @adamsilverstein
2 years ago

In 53848:

Media: use original image size data for full size secondary mime generation.

Correct an issue where the secondary mime type full size image would not be properly resized (and -scaled added to the name) when the original upload is over the big_image_size_threshold dimensions.

Props mukesh27.
See #55443.

mehulkaklotar commented on PR #3048:


2 years ago
#99

@adamsilverstein In the latest change, the priority mime type for application/pdf is image/webp so when we generate the preview, it defaults to webp as main image source and then jpeg. so previews are webp for everywhere else.

mukeshpanchal27 commented on PR #3030:


2 years ago
#100

Thank you for the update. @mehulkaklotar The most recent commit, https://github.com/WordPress/wordpress-develop/pull/3030/commits/c1754cae027df95e188fed051326d4d36c6e277c, generates a bug for the thumbnail edit flow. It creates webp image without the original image extension.

Is there any specific use case for these changes?

mukeshpanchal27 commented on PR #3030:


2 years ago
#101

Thank you for the update. @mehulkaklotar The most recent commit, https://github.com/WordPress/wordpress-develop/pull/3030/commits/c1754cae027df95e188fed051326d4d36c6e277c, generates a bug for the thumbnail edit flow. It creates webp image without the original image extension.

Is there any specific use case for these changes?

mehulkaklotar commented on PR #3030:


2 years ago
#102

@mukeshpanchal27 I have pushed changes to fix the issue you have mentioned. Please have a look.

This ticket was mentioned in Slack in #core-media by antpb. View the logs.


2 years ago

This ticket was mentioned in Slack in #core-media by antpb. View the logs.


2 years ago

This ticket was mentioned in PR #3060 on WordPress/wordpress-develop by mukeshpanchal27.


2 years ago
#105

Unit tests coverage for the WebP implementation.

Trac ticket: https://core.trac.wordpress.org/ticket/55443

#106 @adamsilverstein
2 years ago

55443-tests.diff is based on https://github.com/WordPress/wordpress-develop/pull/3060 and ensures newly added tests run successfully by reaming them with a test_ prefix and moving them to tests/phpunit/tests/media.php

adamsilverstein commented on PR #3030:


2 years ago
#107

@mehulkaklotar looks like some tests are still failing here, can you take a look?

mehulkaklotar commented on PR #3030:


2 years ago
#108

@adamsilverstein It was a environment error when running checks. Fixed now!

SergeyBiryukov commented on PR #3034:


2 years ago
#109

Looked into the test failures:

There was 1 error:

1) Tests_Media::test_wp_print_image_mime_fallback_script_if_content_has_updated_images
file_get_contents(/var/www/src/wp-includes/js/wp-image-mime-fallback-loader.js): failed to open stream: No such file or directory

/var/www/src/wp-includes/media.php:1930

This is caused by the file being present in the build directory, but the tests running from the src directory. To resolve the failure, the tests should copy the file to the expected place. We already use that approach in some other places, for example in emoji tests.

See 0560afb, the tests should pass now.

adamsilverstein commented on PR #3060:


2 years ago
#110

@mukeshpanchal27 when giving this a final review I noticed a few tests that I believe were removed by this PR, was that intentional? can you re-add these please or let me know if I am missing something.

it_should_remove_the_webp_version_of_the_image_if_the_image_is_force_deleted_and_empty_trash_days_is_set_to_zero
it_should_remove_the_attached_webp_version_if_the_attachment_is_force_deleted_but_empty_trash_day_is_not_defined
it_should_not_create_the_sources_property_if_the_mime_is_not_specified_on_the_transforms_images
it_should_not_create_the_sources_property_if_no_transform_is_provided
it_should_replace_the_references_to_a_jpg_image_to_a_webp_version (and provider)

adamsilverstein commented on PR #3060:


2 years ago
#111

  • The functions can call wp_image_use_alternate_mime_types instead of webp_uploads_img_tag_update_mime_type
  • Some tests may need adjusting to pass, for example, -scaled suffix should be -scaled-jpeg.webp for webp in test runs.

#112 @azaozz
2 years ago

This ticket introduces the capability to generate more than a single mime type. For example, when users upload JPEG images, WordPress can automatically generate both WebP and JPEG sub-sized images.

Like @MatthiasReinholz, @eatingrules, and others I think this approach is perhaps lacking. Why would there be twice as many image files taking up a lot of extra space when half of them will never ever be used anywhere?

IMHO a better approach would be to just make all image sub-sizes WEBP. If JPEGs are indeed needed, these can be generated on-the-fly as needed. There is no point of clogging the web servers storage with all these useless files.

On the other hand, if the WEBP file sizes are actually larger than the JPEGs, that would probably mean that better tools are needed, and this patch is premature.

BTW, this functionality (making all sub-sizes WEBP) has been available in WordPress for a few releases. Is there any data to how much it is used, and how well it works. Think such data would be crucial in deciding whether to keep working on this for now.

Another shortcoming seems to be that only some image sub-sizes will be converted to WEBPs. Not sure why, doesn't seem to make sense? Then the users will end up with a "soup" of images, some different sizes, some identical sizes but different formats. Why the mess? :)

Another thing that seems to not make sense is the idea to replace images by default. Why would this be needed? Only newly uploaded images will have WEBP sub-sizes, right? Then they will be used when composing new content, right? Why would WP spend so much resources looking at places where old images are used?

(I understand, given enough storage space some users will probably "regenerate images" and create WEBP sub-sizes. But this will be the exception, not the rule. Such exception should be handled by a plugin. Doesn't seem a good idea to slow down millions of sites just because of that possibility.)

Resources for generating images when you upload an image will increase dramatically, however resources to serve an image...

Actually that dramatic increase of resources usage when uploading an image is a very bad side effect here. It means a lot of uploads will fail, and leave the users stranded. It also will increase support requests for both WordPress and the hosting companies dramatically. Don't think this is acceptable. Because of this, even if image multi-mime support is needed in WordPress, the current approach doesn't seem like a good solution.

Last edited 2 years ago by azaozz (previous) (diff)

mukeshpanchal27 commented on PR #3034:


2 years ago
#113

@felixarntz Thank you for the feedback.

After the changes, browserStack testing results for different browsers and devices.

https://www.browserstack.com/screenshots/4788bd672a483f82bb22812b6bf110a55f0ec9e3
https://www.browserstack.com/screenshots/ca40b72a081bb481f67d6c99140ab86667dc7b84

#114 follow-up: @flixos90
2 years ago

Providing an update here: The WebP fallback mechanism to JPEG for older browsers that don't support WebP (<2% usage globally altogether) looks ready for commit. Any additional feedback on https://github.com/WordPress/wordpress-develop/pull/3034 would be appreciated.

I plan to commit this mid next week.

Last edited 2 years ago by flixos90 (previous) (diff)

#115 in reply to: ↑ 114 ; follow-up: @azaozz
2 years ago

Replying to flixos90:

I plan to commit this mid next week.

Please do not commit any more code here unless it is to address the dramatic increase of resources needed to create image sub-sizes after upload. As I said above such increase is unacceptable.

Is there any data about how much more resources (memory, processing time, etc.) are needed when uploading different image sizes? Any estimate about how many sites may be affected by that? Any suggestions on how to deal with it? Do you know what happens when post-processing of an uploaded image fails?

Frankly for now it seems that this patch will have to be reverted and refactored in order to address this.

#116 @azaozz
2 years ago

  • Keywords needs-patch 2nd-opinion needs-testing added; has-patch commit removed

#117 in reply to: ↑ 115 @jb510
2 years ago

Replying to azaozz:

Please do not commit any more code here unless it is to address the dramatic increase of resources needed to create image sub-sizes after upload. As I said above such increase is unacceptable.

Is there any data about how much more resources (memory, processing time, etc.) are needed when uploading different image sizes? Any estimate about how many sites may be affected by that? Any suggestions on how to deal with it? Do you know what happens when post-processing of an uploaded image fails?

Frankly for now it seems that this patch will have to be reverted and refactored in order to address this.

I raised the issue of the UX impact at the time of image upload several times (Slack and GH). few people seemed to take it seriously but then it got mostly dismissed as OOS.

Initially: https://github.com/WordPress/performance/issues/289#issuecomment-1114880692
Response: https://github.com/WordPress/performance/issues/289#issuecomment-1126135778
Follow up: https://github.com/WordPress/performance/issues/289#issuecomment-1126271943

When a user uploads an image the time spent waiting for sub-sizes to generate. Yes, that is happening in the background, but they're still stuck waiting on the upload screen. This is always created a bad UX, and the time required waiting has doubled (someone verified this I think in that GH issue or on Slack). Obviously, CPU impact is bigger, but I presume that is a single-threaded/single-worker process. That probably not a big impact overall on the site. But is in my opinion a big impact on the UX from an author's point of view.

To me the existing sub-size generation is and has always been fundamentally flawed. Generating dozens of thumbnails many of which _never_ get used ought to have been addressed BEFORE introducing this feature into core to literally double the number of thumbnails. Including doubling the number of never used thumbnails. It just doesn't make sense to keep charging forward on a flawed foundation.

We ought to be first finding a way to stop generating every thumbnail that "might" get used someday and instead generate ALL thumbnails the first time they're used and then cache/permanently store the resulting thumbnail for the future use. Work has already been done on that approach, it has just never got much traction on bringing it into the core.

It's also pretty easy to cause an out-of-disk space scenario with this as is. The critical case should not be the 90% of sites on shared hosting, it ought to be the cases like 100GB sites on DO/AWS/GC/etc.

I think the beneficial impact of implementing WebP by default on existing sites also gets grossly over-estimated. Most existing sites have hardcoded image links, to JPGs. A WebP existing isn't going to make old content serve the WebP. Yet, these sites are going to get burdened with even more never used thumbnails. It just doesn't make any sense to force that.

#118 follow-up: @adamsilverstein
2 years ago

@azaozz - thanks for the review and feedback, responses inline below.

This ticket introduces the capability to generate more than a single mime type. For example, when users upload JPEG images, WordPress can automatically generate both WebP and JPEG sub-sized images.

Like @MatthiasReinholz, @eatingrules, and others I think this approach is perhaps lacking. Why would there be twice as many image files taking up a lot of extra space when half of them will never ever be used anywhere?
IMHO a better approach would be to just make all image sub-sizes WEBP.

I tend to agree with your assesment that all sub-sizes should be generated as WebP only, that was the shape of the original proposal. For the vast majority of use cases/users this will work the best. I would be open to considering this for the default (with some mitigations, see below).

The reason we decided to generate both formats was for backwards compatibility considerations for the few edge cases we identified where WebP images will might not work: namely emailed images (some older outlook/windows clients), Open Graph tags (some services don't support WebP) and older Safari browsers. One possibility we considered would be to keep only the full sized JPEG so it is always available for those edge cases.

The "multi-mime" support here is built to generating multiple formats so you sites can provide a primary and fallback format with something like the picture element. This is less important for WebP since it is so widely supported, but would be helpful for adopting newer formats like AVIF by plugins or core.

If JPEGs are indeed needed, these can be generated on-the-fly as needed. There is no point of clogging the web servers storage with all these useless files.

Generating images on the fly something we need to figure out but felt out of scope for this effort.

On the other hand, if the WEBP file sizes are actually larger than the JPEGs, that would probably mean that better tools are needed, and this patch is premature.

Image sizes are complicated! On average, I have seen ~28% reduction in sizes for WebP version using current WordPress settings in lab testing and in the wild using httparchive data. That said, in some cases the WebP can actually be larger than the JPEG equivelent; for that reason the current code only keeps the WebP (or secondary mime) if it is fact smaller.

BTW, this functionality (making all sub-sizes WEBP) has been available in WordPress for a few releases. Is there any data to how much it is used, and how well it works. Think such data would be crucial in deciding whether to keep working on this for now.

We don't have data about how much this exact feature is used, but we do have data about how many WordPress site pages use WebP images by WordPress version (and the average weight of those images). I am attaching screenshots of the report I created, the main query I used is here. This data tells us that WebP is increasingly being adopted by WordPress sites, even though this currently requires installing a plugin or manually generating WebP images before uploading them.

Another shortcoming seems to be that only some image sub-sizes will be converted to WEBPs. Not sure why, doesn't seem to make sense?

I agree in general all sub-sizes should be generated, with the exception of sizes you know aren't for the front end. For example, if you created a 256x256 pixel logo for your OG tag, you might want to opt that image size out of generating a secondary (or alternate) mime type.

Then the users will end up with a "soup" of images, some different sizes, some identical sizes but different formats. Why the mess? :)

Is it really a mess for the user though? I can see with your media developer hat it is a bit messy, but for end users nothing actually changes in the UI or when using images. The meta data accurately reflects the available images, and deleting the image correctly deletes all the sub sizes. Naming conflicts have also been carefully addressed. Even currently: if you edit an image in media, the filenames all have a hash appended and after multiple edits you end up with many hashed files making a "mess".

Another thing that seems to not make sense is the idea to replace images by default. Why would this be needed? Only newly uploaded images will have WEBP sub-sizes, right? Then they will be used when composing new content, right? Why would WP spend so much resources looking at places where old images are used?

The would be used for new content if they were the only type, but since we are keeping both, the original type (JPEG) will be used in the content by default. An additional consideration is image regeneration - if users regenerate their past images, the secondary type would be generated (WebP) and the on the fly replacement will upgrade their past content images as well.

You bring up a good point here though, there is a potential extra load cost to replacing these images that would be worth quantifying. Worth noting that this work takes place in the context of existing per-image content processing in wp_filter_content_tags where we already prime caches on the attachment IDs (so the cost should be low).

(I understand, given enough storage space some users will probably "regenerate images" and create WEBP sub-sizes. But this will be the exception, not the rule. Such exception should be handled by a plugin. Doesn't seem a good idea to slow down millions of sites just because of that possibility.)

We can investigate how much this slows down sites. As I mentioned, the meta call should return cached data and one of the very check looks for empty( $metadata['sources'] ) - returning early in that case.

Also, this would still be required for new content unless we make "only WebP" the default output.

Resources for generating images when you upload an image will increase dramatically, however resources to serve an image...

Actually that dramatic increase of resources usage when uploading an image is a very bad side effect here. It means a lot of uploads will fail, and leave the users stranded. It also will increase support requests for both WordPress and the hosting companies dramatically. Don't think this is acceptable. Because of this, even if image multi-mime support is needed in WordPress, the current approach doesn't seem like a good solution.

This change also doubles the number of times WordPress will "retry" the image regeneration, so while processing time will increase, I don't think we'll see a jump in failures necessarily. I know we had trouble adding new sizes in the past, however that was before we added the retry mechanism.

Last edited 2 years ago by adamsilverstein (previous) (diff)

#119 @adamsilverstein
2 years ago

Is there any data about how much more resources (memory, processing time, etc.) are needed when uploading different image sizes? Any estimate about how many sites may be affected by that? Any suggestions on how to deal with it? Do you know what happens when post-processing of an uploaded image fails?

In general WebPs are a little more resource intensive to generate vs. JPEGs. Only sites that have WebP support will be affected though, and while we expect support is widespread since it has been available for so long, but we don't have hard data (we will start gathering data since #48116).

The additional resources at upload time need to be weighed against the reduced resources of serving the smaller WebP image, especially since serving typically occurs several orders of magnitude more frequently than uploading.

If uploading fails after all the retries, the user has the same experience as currently: they are left with a broken, unusable image. That can probably be fixed, although I don't think this change will increases failure rates.

#120 @eatingrules
2 years ago

Is it really a mess for the user though? I can see with your media developer hat it is a bit messy, but for end users nothing actually changes in the UI or when using images.

Serving WebP images when they're not expected by the user will indeed cause confusion for non-developers.

We use Cloudflare Polish for our clients' sites, which will serve WebP if it's smaller than the original JPG... our clients (or their Assistants) will try to download an image from the front-end of the site. So they'll Right-Click->Save As... and then be very confused and frustrated when the file that they think is a JPG actually downloads as a WebP.

#121 in reply to: ↑ 118 @jb510
2 years ago

Replying to adamsilverstein:

Is it really a mess for the user though? I can see with your media developer hat it is a bit messy, but for end users nothing actually changes in the UI or when using images. The meta data accurately reflects the available images, and deleting the image correctly deletes all the sub sizes. Naming conflicts have also been carefully addressed. Even currently: if you edit an image in media, the filenames all have a hash appended and after multiple edits you end up with many hashed files making a "mess".

I'm going to disagree and say this is messy for both developers and users. Years ago we taught users that they could truncate the webp extension to get the jpg if they needed a jpeg link Don't get me wrong, we tell people all the time they shouldn't be right-click downloading images from the web, especially when they're usually the person that uploaded the original in the first place, yet we have a lot of authors that regularly do just that to reuse the images associated with their article when they re-post to social.

ie.
/wp-content/uploads/2022/08/0_AbbrLEQpkYjNgWkr.jpg.webp
/wp-content/uploads/2022/08/0_AbbrLEQpkYjNgWkr.jpg

It's just human nature to assume things like that will "just work". Hence my comment 75 above. https://core.trac.wordpress.org/ticket/55443#comment:75

Finally, a user story about this:

As a user/blogger of WordPress, I want to run head first into WebP (and hopefully soon AVIF). While I appreciate the use cases that still require JPEG, I don't really care about them or the breakage (OG tags, feed, newsletters). I'm totally willing to accept those breakage edge cases given the choice. I'd much rather have the original + 100% WebP thumbnails stored. Then I'd delete 100% of Jpeg thumbnail images (not the original). I could also see OG and others eventually supporting webp, at which point how do we tidy up our already bloated media libraries at that point?

However, this is a very different story as a developer working on very large publisher sites. There, I definitely need BOTH image types "forever" because there are millions of inbound links (hotlinking) to those images. Legacy 3rd party integrations from 15 years ago are critically important that they keep working.

All this is why a simple media settings option is needed to enable this on demand for old sites instead of by default. It's a single checkbox that gives site owners needed controls. "Decisions not options" is about avoiding complexity, it's not about having zero options at all.

#122 @adamsilverstein
2 years ago

  • Description modified (diff)

#123 @dd32
2 years ago

Just noting here, that these additional webp conversions appear to have been the leading cause of upload failures on the WordPress Photo Directory in recent weeks. See #meta6142 and tickets closed as duplicate of it.

The errors were generally along the lines of Allowed memory size of 256M bytes exhausted (tried to allocate 90M bytes (obviously with byte values) while attempting to perform the initial full-size-original jpeg -> webp conversion.

It hasn't affected every upload, only that of certain images. Potentially related to the $quality value being passed for webp requests (IIRC the default of 82 was optimized for jpeg?).

I've disabled the jpeg->webp conversion there [meta12023], temporarily at least, as we don't currently utilise the webp in the directory, however, this might be a sign that it might be worth considering only generating webp's for the resized images, rather than for the original file too.

#124 @costdev
2 years ago

  • Keywords changes-requested added

As an aside, please note:

  • All tests that are prefixed with it_ need to be updated to use the test_ prefix instead. Ref.
  • All data providers using the provider_ prefix need to be updated to use data_ instead. Ref 1. Ref 2.
  • All data providers using yield need to instead return an array of arrays - named datasets with keys. Ref.

i.e.

public function data_name_of_test_method() {
    return array(
        'the first dataset description' => array(
            'the_first_argument_name'  => 'value',
            'the_second_argument_name' => 'value',
         ),
     );
}
Last edited 2 years ago by costdev (previous) (diff)

#125 follow-up: @adamsilverstein
2 years ago

@costdev thanks for raising this, work to correct the unit tests is happening in https://github.com/WordPress/wordpress-develop/pull/3060

Just noting here, that these additional webp conversions appear to have been the leading cause of upload failures on the WordPress Photo Directory in recent weeks. See #meta6142 and tickets closed as duplicate of it.

@dd32 Thanks for adding this important data point and the details on the ticket.

Out of memory for the smaller images would be surprising, I wonder if this is exposing a bug - the memory should be reclaimed after each image resize (even if they happen as part of a long running process), but maybe that isn't how it currently works (the smaller core image sizes are processed last). I'll have to try some memory profiling to see if we can improve the processing and free up memory between resizes.

It hasn't affected every upload, only that of certain images. Potentially related to the $quality value being passed for webp requests (IIRC the default of 82 was optimized for jpeg?).

We are investigating that setting in https://github.com/WordPress/performance/issues/7 and so far my research indicates the WebP level should actually match JPEGs, eg. we should lower it to 82. This might help a little but wouldn't solve your problem.

I've disabled the jpeg->webp conversion there [meta12023], temporarily at least, as we don't currently utilise the webp in the directory, however, this might be a sign that it might be worth considering only generating webp's for the resized images, rather than for the original file too.

Thanks again for the feedback and for "dogfooding" trunk - this is exactly the type of data we need to make decisions around this feature. I was also happy to see your elegant fix disabling the feature. I assume that since adding the fix, reports of the issue have dissipated?

#126 @azaozz
2 years ago

Replying to adamsilverstein:

Thanks for the clarifications!

I tend to agree with your assessment that all sub-sizes should be generated as WebP only, that was the shape of the original proposal.
...
The reason we decided to generate both formats was for backwards compatibility considerations

Yes, makes sense. Seems it would be better to make these images "on demand". The advantages would be:

  1. Faster processing of uploaded images. This is critical.
  2. Reduced space requirements. JPEG images won't be generated unless they are needed.

The "on demand" part can probably be handled by the code that would determine which image format to output.

The "multi-mime" support here is built to generating multiple formats so you sites can provide a primary and fallback format with something like the picture element. This is less important for WebP since it is so widely supported, but would be helpful for adopting newer formats like AVIF by plugins or core.

To me it seems that the "multi-mime" support is not currently a critical feature. I agree, once the AVIF format gets wide support in the browsers, and in the image editing tools available in PHP, multi-mime support will be very desirable. But for now it doesn't seem needed.

Also thinking that implementing support for several image mime types would be best done as part of a larger refactoring of the images handling code. Perhaps even moving the image editing to the client and uploading the sub-sizes, and/or creating sub-sizes only when needed, etc. (I know, seems a bit far fetched, but one can dream, right?)

Generating images on the fly something we need to figure out but felt out of scope for this effort.

Yes, it seems out of scope here. On the other hand it seems it will significantly improve this implementation. The first access of a particular image may be slowed down significantly (as it's being generated), but the space and processing time savings would bring far greater benefits.

Image sizes are complicated! On average, I have seen ~28% reduction in sizes for WebP version using current WordPress settings in lab testing and in the wild using httparchive data. That said, in some cases the WebP can actually be larger than the JPEG equivalent; for that reason the current code only keeps the WebP (or secondary mime) if it is fact smaller.

Yes, you're right. I'm thinking more along the lines of having a "sensible mid-point" cutoff. Perhaps something like 800px or maybe 1000px. If an image's width and height are smaller, chances are WebP will be a better format for it. If not, make a JPEG. The advantages of such approach seem pretty good. Again, less space needed and greatly reduced processing time/resources (considering WebP images are not created and then deleted for the larger sizes).

I agree in general all sub-sizes should be generated, with the exception of sizes you know aren't for the front end.

Yep, agreed.

Is it really a mess for the user though?

What I mean is that there are some (subtle but noticeable) differences between JPEGs and WebPs when used on the web. A good example is that when a site visitor downloads an image, WebP will be a lot less usable than a JPEG. However the users/site visitors won't be able to tell which is which, hence some confusion.

You bring up a good point here though, there is a potential extra load cost to replacing these images that would be worth quantifying.
...
Also, this would still be required for new content unless we make "only WebP" the default output.

Yea, agreed. If the implementation is changed to generate fallback JPEGs on demand, that will be happening here, right? Then it makes sense to keep and even expand this imho.

This change also doubles the number of times WordPress will "retry" the image regeneration

The problem is that "retry" for image post-processing doesn't work as well as expected. It is on my (ever growing) TO-DO list to revisit it soon, but... Not sure exactly when I'll get a chance. The bad news is that if the post processing fails, chances are the originally uploaded file will remain. Then it will be used everywhere as most of the code in WP falls back to available sizes, and the only size will be the original. That means we will be serving huge (4MB - 8MB average) images. A serious drawback.

Last edited 2 years ago by azaozz (previous) (diff)

#127 in reply to: ↑ 125 @dd32
2 years ago

Replying to adamsilverstein:

I've disabled the jpeg->webp conversion there [meta12023], [...]

[...] I assume that since adding the fix, reports of the issue have dissipated?

Looking at the PHP Fatal error logs, yes, fatals related to image upload has ceased. Multiple failures a day to 1 in 3 days, and that 1 OOM error was needing an extra 32 Kilobytes of memory, not 90+ Megabytes.

edit: for reference, WordPress.org is currently running PHP 7.4.30 w/ GD Version => bundled (2.1.0 compatible), ImageMagick is not currently enabled.

Last edited 2 years ago by dd32 (previous) (diff)

#128 follow-up: @adamsilverstein
2 years ago

Thanks for the feedback @azaozz! Responses and sone suggested ways forward below...

First: on the specific point of the default behavior, I agree with your suggestion of only generating WebP format images. Also, I like the possibility of having a (filterable) “threshold” value above which (only) JPEGs are generated, for example making anything over 1024 or 2048 pixels in either dimension default to JPEG - similar to the way we handle big_image_size_threshold/-scaled images currently.

The performance team will research this more directly to test what the threshold should be (and the assumption that we should have a threshold). Anecdotally in the field data we can see that images on desktop WordPress pages using WebP images are about 8% smaller, while images on mobile pages using WebP are around 38% smaller (based on HTTP Archive data using this query). One advantage of retaining some sizes is we can also use these JPEGs as fallback images for the edge cases where WebP still isn’t supported (eg. Open Graph tags).

Second: on the infrastructure added here to support multiple mimes. This includes the code added to support multiple mimes directly in image generation, as well as changes to core media functions and on the fly updating on post content. We need to weigh the pros and cons of these changes separately.

Pros:

  • Prepares media for supporting multiple types using the picture element
  • Enables seamless support for multiple mimes useful for plugins managing media.
  • Allows for maximum backwards compatibility because current data/types remain unchanged (when generating both types).
  • Requested by contributors: in the original development issue, only WebP images were generated; the multi-mime type support was added based on a contributor issue and subsequent group decision (worth reading for context).
  • Supports any use case where multiple mimes are desired by the site, for example generating JPEG fallback images on demand

Cons:

  • Adds additional complexity/maintenance burden to core media (which is already complex)
  • Single/alternate mime type output can still achieve a high degree of backwards compatibility as long as we keep some versions as JPEGs.
  • If we generate multiple MIMES:
    • Consumes significantly more resources on upload (increasing the chance for failures)
    • Small overhead of run-time content image swapping
    • Degraded user experience (slower uploads)

Given these points I can see two paths forward we should consider:

  1. Keep the current multi-mime infrastructure, but change the defaults so only WebP files are generated, possibly up to a threshold size beyond which only JPEGs would be generated. Most existing work would continue; content filtering could possibly be removed.
  2. Revert the multi-mime infrastructure and switch back to a single mime approach, using WebP for images up to a threshold size and adjusting the compatibility layer to use the JPEGs we keep.

When making this decision, we should also consider what is coming in the next few years: namely widespread AVIF support (support will land shortly in Safari). If we might be adding AVIF support in a few years, would we want the multi-mime support then, or would the “single mime” output support suffice?

My own inclination is to go with option 1 because of the pros listed above and the community process that led to the approach (and possibly I will admit some attachment to all the hard work that went into developing it). It works well, adds flexibility to media handling and sets us up for the future.

I recognize option 2 would be a much smaller change and would involve less risk and maintenance burden while still achieving the main goal of improving WebP adoption and hence lifting WordPress site performance. If we go with this option, we can still consider adding multi-mime support in the future, including the async “on demand” generation you suggested so we could actually improve upload success and user experience.

Let me know what you think for the best path forward and my points/questions above. Did I miss any pros/cons of multi-mime support? I'd like to agree on an acceptable approach soon so we can land this before beta (Scheduled for Sept. 20 according to https://make.wordpress.org/core/2022/06/23/wordpress-6-1-planning-roundup/).

#129 follow-up: @jb510
2 years ago

Maybe already added to the conversation. But most major services (FB, TWTR, do already support WebP for open graphs tags, even if not officially in the spec yet.

This link has a nice recent compatibility chart for reference: https://www.ctrl.blog/entry/webp-ogp.html

Perhaps instead of accommodating those outliers, WordPress should be pushing them to catch up with the others.

Regarding multi-mime. All this seems to be hamstrung by the face that these are all being generated upon upload, rather than on first use.

I really feel before we accept the storage and performance impacts this would cause that some serious consideration needs to be given to approaches that would modernize thumbnail generation and the needless preemptive generation of so many thumbnails:

  1. Simply store the original in whatever format it's uploaded
  2. Upon first request of an image, serve whatever is closest and then attempt to generate the correct size and format for next time.

Timthumb did this 15 years ago, surely we can do better today?

This ticket was mentioned in Slack in #core-media by adamsilverstein. View the logs.


2 years ago

#131 in reply to: ↑ 128 @azaozz
2 years ago

Replying to adamsilverstein:

Thanks for the reply!

I agree with your suggestion of only generating WebP format images. Also, I like the possibility of having a (filterable) “threshold” value above which (only) JPEGs are generated

Great! Think this will make it faster and simpler.

One advantage of retaining some sizes is we can also use these JPEGs as fallback images for the edge cases where WebP still isn’t supported (eg. Open Graph tags).

Yes, exactly. This will further simplify the implementation and improve image post-processing making it less likely to run out of memory/fail.

Given these points I can see two paths forward we should consider:

  1. Keep the current multi-mime infrastructure, but change the defaults so only WebP files are generated, possibly up to a threshold size beyond which only JPEGs would be generated. Most existing work would continue; content filtering could possibly be removed.
  2. Revert the multi-mime infrastructure and switch back to a single mime approach, using WebP for images up to a threshold size and adjusting the compatibility layer to use the JPEGs we keep.

When making this decision, we should also consider what is coming in the next few years: namely widespread AVIF support (support will land shortly in Safari). If we might be adding AVIF support in a few years, would we want the multi-mime support then, or would the “single mime” output support suffice?

My own inclination is to go with option 1 because of the pros listed above and the community process that led to the approach (and possibly I will admit some attachment to all the hard work that went into developing it). It works well, adds flexibility to media handling and sets us up for the future.

Multi-mime support sounds pretty good, but also has some drawbacks. At first look it seems it's worth the effort to add now, but looking at the details, it adds a lot of complexity and possible bugs and edge cases. In addition it is a large update to most of the images handling code that is not critical for the current task at hand. As you said WebP image sub-sizes can be implemented without it, and the implementation will probably be simpler, faster, and with less edge cases/bugs.

Images handling in WordPress is old, legacy code that has been fixed and enhanced hundreds of times over the last 15 years. It is pretty hard to work with and there are many chances of introducing edge cases and regressions. I think that adding multi-mime support would probably be better as part of a larger re-imagining/refactoring of how WordPress works with images.

Also I think it would be better if the multi-mime support is added after wider testing. For that it would be great to try to make it into a feature plugin if possible. Also, it would need a separate ticket. Thinking the current code is a great start, and it would need some more testing and polishing.

I recognize option 2 would be a much smaller change and would involve less risk and maintenance burden while still achieving the main goal of improving WebP adoption and hence lifting WordPress site performance. If we go with this option, we can still consider adding multi-mime support in the future, including the async “on demand” generation you suggested so we could actually improve upload success and user experience.

Yes, exactly. In my mind this is the right approach. WebP by default can be added now. Multi-mime support can be added in the near future, most likely before it is needed for supporting AVIF. Ideally by then WordPress will be able to generate image sub-sizes on the fly as needed.

This is starting to sound like a pretty good plan for this part of WordPress for the next few years. I hope we will be able to implement it well :)

#132 in reply to: ↑ 129 @azaozz
2 years ago

Replying to jb510:

  1. Upon first request of an image, serve whatever is closest and then attempt to generate the correct size and format for next time.

Timthumb did this 15 years ago, surely we can do better today?

Unfortunately it's not as simple to implement this on all servers for everybody, everywhere. Part of the problem is finding the image meta from the image URI. Another part is that it will probably need a bit more in .htaccess which would probably not work everywhere.

But generally I agree, it's time to revisit. Any takers that would like to help? :)

This ticket was mentioned in Slack in #core-media by mike. View the logs.


2 years ago

#134 @snoringdragon
2 years ago

Hi, there.

I agree with your suggestion of only generating WebP format images. Also, I like the possibility of having a (filterable) “threshold” value above which (only) JPEGs are generated.

This is a great measure.

One advantage of retaining some sizes is we can also use these JPEGs as fallback images for the edge cases where WebP still isn’t supported.

Yes, it's important for backward compatibility.

Hope that it helps.
Thanks.

Last edited 2 years ago by snoringdragon (previous) (diff)

#135 follow-ups: @spacedmonkey
2 years ago

Just a thought. How about this.

Allow users to upload jpgs are normal. Add a warning / check in the editor. Something like.

“It appears you are using images that are not optimised, would you like to optimise them?”

Then when you click okay, covert images to webp and remove jpeg version.

Then add a check setting to “optimise all images by default”.

For what it is worth, I think the server space issue is a little over blown. Lots of popular plug-ins add image sizes that create more files and no one talks about that.

#136 follow-up: @eatingrules
2 years ago

For what it is worth, I think the server space issue is a little over blown. Lots of popular plug-ins add image sizes that create more files and no one talks about that.

It's precisely because of our experience with image optimization plugins and the massive bloat from (largely unused) thumbnails piling up that I know that this will make an existing problem much worse.

I run a WordPress maintenance agency that currently supports about 750 sites, ranging from mid-size e-commerce to some high-traffic sites seeing 20-30 million pageviews/month. The majority of our clients have been publishing for many years, and their media libraries typically have anywhere from 3,000 to 20,000 images, and are growing every day.

We don't provide hosting, and we have experience with all the major hosts and many others as well. We run into storage issues on a near-daily basis. Either their hosting account is running out of storage, or we're having issues with our nightly backups timing out because there are too many files.

Often, it's directly due to image optimization. We use Shortpixel or Imagify on many sites, and keep the backups of the originals...this is indeed a large part of the usage (though we often exclude those backups from our backups), but we have found it's even more problematic not to save the originals, should anyone want to reoptimize in the future (so at that point we recommend our clients upgrade their plans to get more storage).

At least in the case of these image optimization plugins, they are opt-in. The clients (or their support company) choose to install these tools, and therefore they can be aware of the ramifications.

I know I've been one of the most outspoken critics of this proposal, but I'm only chiming in so emphatically because I have deep experience on this topic. I also understand the end user extremely well, since I work with literally hundreds of publishers on a daily basis and see all the issues that come up...and know that this is going to confuse the heck out of most site owners.

Should things move forward as proposed, the disk space issues may not surface right away on most sites, but over time (or when someone or something regenerates thumbnails), it will absolutely be a problem on many, many sites.

#137 in reply to: ↑ 135 ; follow-up: @eatingrules
2 years ago

Replying to spacedmonkey:

Just a thought. How about this.

Allow users to upload jpgs are normal. Add a warning / check in the editor. Something like.

“It appears you are using images that are not optimised, would you like to optimise them?”

Then when you click okay, covert images to webp and remove jpeg version.

Then add a check setting to “optimise all images by default”.

Interesting ideal. Maybe a bit clunky, but at least this way it's not just creating extra images in the background with a site owner oblivious to what's going on, and instead gives them the control and can help them make an informed decision.

It seem to me that all the concerns people have raised that haven't been addressed yet could be solved in one of two ways:

  1. Overhaul how WordPress deals with thumbnails, generating as-needed on first use. (A huge undertaking, but would be an awesome outcome!). This would provide plenty of headroom for both JPG and WebP files to be saved to disk as-needed.
  1. Add an option in the Media Settings, disabled by default, that can enable WebP generation... it could explain both the pros and cons so people can decide what's best for them. There could even be a dashboard notification suggesting people turn it on.
Version 0, edited 2 years ago by eatingrules (next)

#138 in reply to: ↑ 135 @azaozz
2 years ago

Replying to spacedmonkey:

“It appears you are using images that are not optimised, would you like to optimise them?”
...
Then add a check setting to “optimise all images by default”.

I think I understand where you're coming from, but how many people would choose to not optimize their images? We're talking about the great majority of users that probably haven't heart about WebP or what advantages and disadvantages it has. Imho doing this optimization by default for newly uploaded images is the right choice here. Of course there should be an easy way to disable it from a plugin, either globally or per image.

For what it is worth, I think the server space issue is a little over blown. Lots of popular plug-ins add image sizes that create more files and no one talks about that.

I see @eatingrules already replied that adding larger storage requirements by a plugin is acceptable as the site owner can make an informed decision about it. Yes, generally server disk space is cheaper than bandwidth when an image is used somewhat frequently (SSD prices have come down but still not that cheap).

It becomes more expensive for rarely used images, and eventually can become "wasted space" for images that are used very rarely or almost never. (The actual calculation depends on several factors, but 2-3 sub-sizes would be enough for rarely used images. If there isn't a good match in the srcset the requested image size will likely be larger, and the bandwidth will increase a bit. However that image will be loaded only few times per year. Chances are the increased bandwidth cost will be less than the storage cost fro extra sub-sizes.)

Last edited 2 years ago by azaozz (previous) (diff)

#139 in reply to: ↑ 136 ; follow-up: @azaozz
2 years ago

Replying to eatingrules:

I run a WordPress maintenance agency that currently supports about 750 sites, ranging from mid-size e-commerce to some high-traffic sites seeing 20-30 million pageviews/month. The majority of our clients have been publishing for many years, and their media libraries typically have anywhere from 3,000 to 20,000 images, and are growing every day.

Thanks for sharing! This is exactly the feedback that's needed here imho.

Should things move forward as proposed, the disk space issues may not surface right away on most sites, but over time (or when someone or something regenerates thumbnails), it will absolutely be a problem on many, many sites.

It would be great if you could post some aggregated data and perhaps some (estimates) on how storage requirements change after installing an image optimization plugin, or perhaps how much more space is taken by the WebP images.

Also the current proposal is to not create duplicate images at all. Instead there would be a threshold. Smaller sizes will be WebP as it seems that's most efficient there. Larger sizes will continue to be JPEG. These larger sizes will also be used as fallback where WebP is not supported.

If this is accepted (and I hope it will be) the overall storage requirements will actually decrease as the WebP image sizes will be smaller.

Last edited 2 years ago by azaozz (previous) (diff)

#140 in reply to: ↑ 139 @eatingrules
2 years ago

Replying to azaozz:

Thanks for sharing! This is exactly the feedback that's needed here IMHO.

You bet. I'm incredibly grateful you've jumped in here, too. Thank you.

It would be great if you could post some aggregated data and perhaps some (estimates) on how storage requirements change after installing an image optimization plugin, or perhaps how much more space is taken by the WebP images.

In previous conversations, I think the rough estimate has been that WebPs are about 70% the size of JPG (on average)... so the proposal (before the recent threshold change), would likely increase image library sizes by 70% (assuming thumbnails get regenerated at some point).

So.. if someone has, say, a 10GB folder of images, that would turn into roughly 17GB.

For our clients, we don't have these plugins create WebP images -- we just optimize the existing files (JPGs stay JPGs) and keep the originals in a backup folder. I don't have an easy way to aggregate data, but I can at least try to find some representative samples of the upload folder size vs. the Shortpixel/Imagify Backups folder sizes. I might not be able to get that done until the weekend, but I'll follow up with that ASAP.

The sheer number of files is a concern as well...Hosts often have inode limits in individual folders, and some also implement overall limits on the total number of inodes.

(To be fair: it's rare for us to run into these inode limits, as we're generally working with higher-level hosting. I'm guessing that would be more of an issue on lower-budget, shared hosting plans.)

Another example: We use ManageWP as one of our backup tools; it has a limit of 1 Million files per backup/site. We're actually struggling to be below that threshold right now on a site we recently started working with... they don't have any WebP files and we've excluded the Shortpixel backup folder.

And it'll be especially bad if a user does not have "Organize my uploads into month- and year-based folders" enabled. (Now that's a setting I would like to see go away! 😆)

Also the current proposal is to not create duplicate images at all. Instead there would be a threshold. Smaller sizes will be WebP as it seems that's most efficient there. Larger sizes will continue to be JPEG. These larger sizes will also be used as fallback where WebP is not supported.

I believe the Performance Team's original proposal was to do this (though without the JPG cutoff threshold), and it was met with some significant resistance as well.

Reading the comments on these posts may be helpful if you haven't seen them yet:

https://make.wordpress.org/core/2022/03/28/enabling-webp-by-default/
https://make.wordpress.org/core/2022/04/12/follow-up-on-webp-by-default-proposal/

I do think the current plan you referenced could work, and be of significant benefit to many sites while mitigating the possible negative impacts, if all of the following are true:

1. It's opt-in, not opt-out.
2. The site owner is made aware of the pros/cons before enabling it.
3. There is an easy setting (not just a filter) to turn it on/off.

This way, if the site owner finds there's an issue, they will (presumably) understand why and could turn the feature off.

There would still be some other challenges to overcome, though. A few off the top of my head:

If it's turned off, how would it revert gracefully? Would they have to regenerate thumbnails?

What would happen when they turn WebP on and then later regenerate thumbnails? If the system then generates WebP thumbs, but doesn't delete the old JPG thumbs, we may have similar (and very sudden) disk space issues.

One idea to solve that could be to delete all the old JPG thumbnails as the WebPs are generated, but that would likely cause other issues. For example, if JPG thumbs are indexed in Google Image Search, or linked to directly from other sites, there could be many broken links.

Format conversion behind-the-scenes can also confuse users. I shared this earlier in the thread:

We use Cloudflare Polish for our clients' sites, which will serve WebP if it's smaller than the original JPG... our clients (or their Assistants) will try to download an image from the front-end of the site. So they'll Right-Click->Save As... and then be very confused and frustrated when the file that they think is a JPG actually downloads as a WebP.

One last anecdote for now: Here's a story I shared in Slack in #core-performance last May, about a client whose site was generating WebP images (due to a performance plugin installed by his host), and he was unaware. It was an SEO disaster when he changed hosts, ultimately costing him many thousands of dollars of lost revenue. It's another example of how there can be catastrophic unintended consequences to these changes, which is why I think it's best to tread lightly.

#141 @costdev
2 years ago

@azaozz Also the current proposal is to not create duplicate images at all. Instead there would be a threshold. Smaller sizes will be WebP as it seems that's most efficient there. Larger sizes will continue to be JPEG. These larger sizes will also be used as fallback where WebP is not supported.

Query: If I, for example, upload a JPEG that's 1500x1500 and 512KB, and the feature creates thumbnails, does that mean that the 1500x1500 and 512KB JPEG would be used as a fallback if WebP isn't supported, even if the display size is 150x150?

Last edited 2 years ago by costdev (previous) (diff)

#142 @kirasong
2 years ago

I think the idea for a threshold is a great one -- and from the tests so far, 1024 and smaller benefit most from the conversation to WebP.

Anything above that sometimes, and sometimes does not benefit from the conversion at the same quality level.

#143 follow-up: @adamsilverstein
2 years ago

Research on WebP Sizes

We have completed a round of research on image sizes to investigate the best/any threshold for creating WebP images. In summary, the research took a sample group of images, ran them through WordPress sub-sized image generation, then output data about the WebP and JPEG version’s file size to a spreadsheet (see summary of results and this repo branch with testing code).

Note that we are currently discussing the best compression quality to use for WebP on this ticket, and are likely to change the setting from the current 86 to either match the JPEG setting (82) or somewhere in between (e.g. 83 or 84 or...). For that reason the research was conducted repeatedly with WebP quality settings of 82, 84 and 86. For my comments below I am using the numbers from the 82 setting. Regardless of the quality setting though, the conclusion is the same.

With quality for WebP set to 82: for all image sizes, the average savings for WebP was ~36% and only 4.75% of generated WebPs were larger than the equivalent JPEG . In those cases, they exceeded the JPEG size by an average of 3.75% and maximum of 9.64% (and an even lower 6.25% maximum for large images). For comparison, the savings at 84 quality averaged ~29%, with 7.19% being larger; at 86 quality the savings averaged ~23% with around 16.99% being larger. In all cases though, the large savings for most images outweigh the small increase for a small percentage of images.

Conclusion

To summarize, the research shows that across all image sizes, WebPs benefits far outweigh any adverse results (even for the current quality of 86, and much more so for 82). Based on these averages, it makes sense to generate WebPs for all image sizes. The vast majority of generated images will be smaller, and the few that are larger will be only slightly larger.. Therefore, I don’t think we need a threshold filter. Is there a use case for it I missed?

One addition to consider is enhancing add_image_size similar to what is proposed in https://core.trac.wordpress.org/ticket/56288. This would let developers opt out of the mime transform for specific sizes, so a newsletter plugin could add their own custom size that would always be JPEGs.

JPEG for compatibility

To ensure compatibility for use cases where WebP may not be supported (OG tags, older safari and email), we can rely on the “original_image” meta when present (or the full size image when missing). Developers can use wp_get_original_image_url to retrieve the JPEG version.

We may want to consider always creating a full sized WebP version of uploaded images to ensure we always have a “full” sized WebP (even for those below the big_image_size_threshold; above that size we would generate a WebP image already). In this case, original_image meta would always be present.

What do you think?

Additional questions

Regarding the question from @costdev

Query: If I, for example, upload a JPEG that's 1024x1024 and 512KB, and the feature creates thumbnails, does that mean that the 1024x1024 and 512KB JPEG would be used as a fallback if WebP isn't supported, even if the display size is 150x150?

True, however the JPEG would only be used for ~2% of Safari users. For Open Graph tags, the consumer site will likely create its own sub-sized front end variations. Email or newsletter platforms probably/can do the same, we can test that to make sure.

This was part of the issue multi-mime format generation aimed to address. If WordPress generates both WebP and JPEG, we can provide maximum compatibility - Safari users could always count on getting a similar sized JPEG as a fallback. The trade off here is increased file storage and upload processing cost to generate the extra images and we previously decided that trade off was worth it. For reference this is the original issue where adding multi-mime support was discussed.

Infrastructure

Regarding the inclusion of the infrastructure for multiple mimes: this would no longer be strictly required for only outputting WebPs. Still, the multi-mime feature does offer some advantages worth considering: mainly the ability for site owners to maximize compatibility by always having a jpeg version of every size (at the expense of work at upload time).

Also I think it would be better if the multi-mime support is added after wider testing. For that it would be great to try to make it into a feature plugin if possible

Worth noting that the multi-mime architecture has been available/tested in the Performance Lab Plugin (which is a collection of performance features being testing for core) since it's initial release and with over 10k installs. Not sure this addresses your concern about added complexity, but this is definitely getting some widespread testing already!

I think that adding multi-mime support would probably be better as part of a larger re-imagining/refactoring of how WordPress works with images.

One enhancement I see is if we introduced multi-mime support along with async image generation, we could enable generating non-default mime types only when we need them, e.g. adding a $mime_type parameter to media functions, then generating the requested mime type if missing.

Although I would love to see a larger refactor, that shouldn’t be a blocker for including this feature if we are confident of its value.

Yes, exactly. In my mind this is the right approach. WebP by default can be added now. Multi-mime support can be added in the near future, most likely before it is needed for supporting AVIF. Ideally by then WordPress will be able to generate image sub-sizes on the fly as needed.

This is starting to sound like a pretty good plan for this part of WordPress for the next few years. I hope we will be able to implement it well :)

Based on this I will prepare two commits: one to revert the existing milti-mime commits, and one to introduce the WebP output change (so that it can be reviewed separately). Once this is in place we can follow up and evaluate if we want to add additional “full” size image handling, and update the shim we developed to swap out images for Safari.

Thoughts?

#144 @costdev
2 years ago

Thanks for clarifying this @adamsilverstein!

Just to check:

True, however the JPEG would only be used for ~2% of Safari users.

Based on this comment, those getting the fallback JPEG are ~2% of global users, not ~2% of Safari users. These are MacOS users not on MacOS 11 (Big Sur)/MacOS 12 (Monterey), who are using Safari 14-15.6 (the latest released version), correct?

Also, I wasn't able to find a link to the specific usage data on this, partially since Big Sur is lumped in with Catalina for usage. Is there a reference available for the ~2% falling into this category?

Last edited 2 years ago by costdev (previous) (diff)

#145 @eatingrules
2 years ago

Thanks for doing those comparisons, Adam.

Creating only WebP thumbs definitely tempers my concerns about disk space issues, as I'm coming around to the idea that it could solve some problems by not having both types of images stored on disk. But, questions remain:

How will dealing with thumbnail regeneration work?

If regenerating is suddenly going to add WebP thumbs for all images in an existing library, that will suddenly use ~70% more disk space. Regeneration is common enough that this really needs to be considered and planned for.

I don't think it will be safe to delete the old JPG thumbs as new WebPs are generated, since there could very likely be links directly to the JPG images. So how to prevent a sudden and unexpected massive increase in disk space for site owners?

(And it's not just when installing a thumbnail regeneration plugin. WooCommerce, for example, can regenerate thumbnails in the background -- completely unannounced. Woo is running on 5+ million sites!)

Could you maybe share some sample images from the tests so we can get a sense of the visual quality loss (if any) for each of the compression levels?

I know each site owner may have a different tolerance for quality loss, but seeing the differences may help us choose a default compression setting -- rather than basing it solely on % of space savings.

What about images other than photos?

From the repo it looks like you tested with photos from unsplash.com. What about other image types, like drawings, line-art, logos, or screenshots? Or "Pinterest images" (which are typically photos with text overlaid)?

What happens when users Right-Click->Save as, and they get a WebP instead of a JPG?

Of my concerns, this is the most minor - but it speaks to the client's understanding what their site is doing and having control over it. If they opt-in to the WebP feature then they can be informed at that time what will be happening, and will expect to see this new/different format, rather than just being confused or frustrated by it.

Is the plan still for this to be enabled by default?

Making these features opt-in rather than opt-out solves a lot concerns, since it puts the site owner back in the driver's seat.

Will there be any user interface for turning this on or off?

What happens to existing images/thumbnails when it's enabled? Disabled?

Last edited 2 years ago by eatingrules (previous) (diff)

#146 @bradt
2 years ago

Did someone mention on the fly image generation!? I wrote a blog post on this and provided a proof of concept in 2017. I even packaged it up as a plugin and published on WordPress.org as per Matt's request in the comments.

What I've really wanted in core since roughly a decade now is a background processing queue that all plugins and themes can use instead of each implementing their own and taxing the CPU at the same time. On the fly image generation is an excellent first use case for this.

Yes, it seems out of scope here. On the other hand it seems it will significantly improve this implementation.

Agreed.

The first access of a particular image may be slowed down significantly (as it's being generated), but the space and processing time savings would bring far greater benefits.

In my blog post, I suggest serving the best fit image on the first request and queuing the image generation, which is done in the background.

#147 follow-up: @adamsilverstein
2 years ago

  • Keywords needs-patch removed

Did someone mention on the fly image generation!? I wrote a blog post on this and provided a proof of concept in 2017. I even packaged it up as a plugin and published on WordPress.org as per Matt's request in the comments.

Hey @bradt nice to see you here!

The performance team has reviewed your post and plugin as part of the inspiration for what we are working to build for core. As you suggest, the idea is a new API for background tasks (A "background queue") and the first direct use of that API in core would be async media generation (https://core.trac.wordpress.org/ticket/6814).

With this in place, user experience uploading images would greatly improve. We could show them a thumbnail immediately (uploading images in the block editor does this already), then generate the sub-sized images in the background without requiring the presence of the user agent.

#148 @adamsilverstein
2 years ago

@eatingrules thanks for the questions, answers below inline...

How will dealing with thumbnail regeneration work?

By default, regeneration will use the current generation scheme, so you would get new WebPs as you suggested.

If space is a concern: to avoid extra files, you could either use the 'delete previous images' option in the Regenerate images plugin, or disable WebP before regenerating images (for example, with this plugin https://wordpress.org/plugins/disable-webp-by-default/). For the specific case of woo commerce, they may want to consider an option for disabling WebP output for their regeneration code.

Are you aware of any other plugins that regenerate images?

Could you maybe share some sample images from the tests so we can get a sense of the visual quality loss (if any) for each of the compression levels?

I can dig up some or file names at least. You could also re-test with your own set of images using the tool.

What about images other than photos?

Image quality testing provides a spectrum of results based on the input and we don't have any data on what users actually upload. We picked images that were mostly hires photos, something that is very common on the web. You will get different results testing images like line art, drawings or logos, however my guess is you will still see an overall improvement in WebP sizes.

Line art should probably be SVGs. Logos and product images are typically served as PNG images on the web today because of the alpha transparency capability. WebP makes a much better format for these images because it supports lossy images with alpha transparency, something that isn't possible with JPEG. Compared to a transparent PNG, the WebP lossy compression version will save ~80%!

What happens when users Right-Click->Save as, and they get a WebP instead of a JPG?

Those saved WebP images will be slightly less usable (presumably, some programs still don't support WebP). Really though, this isn't much different from what happens when you save an image from your iphone in HEIC format (which is arguably less compatible). The same will be true of any new format and this shouldn't stop us from adopting them. Ultimately, I expect the user agent (eg the browser) to provide a 'save as' option (similar to other programs), so users can get a JPEG if they need one.

Is the plan still for this to be enabled by default?

Yes, it doesn't make sense to introduce a new feature that is not enabled.

Will there be any user interface for turning this on or off?

No. I have yet to see a strong argument for having a UI. I remain open to considering it, although as previously mentioned, project philosophy strongly suggests no UI. Also, this question remains: would average users know how to choose the best option? Instead, users can install a plugin to disable the feature, see https://wordpress.org/plugins/disable-webp-by-default/.

What happens to existing images/thumbnails when it's enabled? Disabled?

Nothing. This feature only effects the generation of images, no change is made to existing media.

#149 in reply to: ↑ 147 @jb510
2 years ago

Replying to adamsilverstein:

Did someone mention on the fly image generation!? I wrote a blog post on this and provided a proof of concept in 2017. I even packaged it up as a plugin and published on WordPress.org as per Matt's request in the comments.

Hey @bradt nice to see you here!

The performance team has reviewed your post and plugin as part of the inspiration for what we are working to build for core. As you suggest, the idea is a new API for background tasks (A "background queue") and the first direct use of that API in core would be async media generation (https://core.trac.wordpress.org/ticket/6814).

With this in place, user experience uploading images would greatly improve. We could show them a thumbnail immediately (uploading images in the block editor does this already), then generate the sub-sized images in the background without requiring the presence of the user agent.

@adamsilverstein apologies for being a little OT, but is there a ticket for that background image processing queue? The most important thing many have expressed is stopping the needless generation of never used thumbnails just because the size definition was added. I’m really hopping this is being taken into account and it’s not just preemptively generating all sizes in the background. We used Brad’s solution for many years on several large sites with great success. Ironically the reason we stopped was we adopted WebP.

I feel support for WebP keeps getting understated, or that the lack of support given undue concern. The only two places it’s not supported are Safari when running on an earlier version of MacOS than 11. That’s a tiny number I think can be ignored. And mobile safari running on iOS 13 and earlier, that’s a tiny number that probably should be given some consideration if only because serving a larger fall back on mobile would be very bad practice.

Click on the “relative usage tab” to get a good visual https://caniuse.com/webp

Threshold - i didn’t see it get discussed but WebP preserves more detail at the same file size than JPEG assuming you start with an uncompressed high detail source image. Comparing SSIM is a bit more complicated than just looking at file size but there is data out there on this. Basically if you start with a medium compressed JPEG that detail is already gone and converting to WebP you end get the same file size, but also the same degraded quality. Taking a raw image source image and then look at SSIM between a WebP and JPEG compressed to the same file size, WebP looks better. Anyway, I agree, thresholds don’t make sense at this point.

I do think you need to generate a full size image (not original) in jpeg format as a fallback. Just one though. I then think there ought to be a filter to enable generating Jpegs of all sizes, vs the default which should just be the full size. Unlike enabling the feature entirely which I still feel needs a user facing checkbox on the media setting page for “Enable modern image format conversion [ ] WebP [ ] AVIF, etc.., With those off by default on upgrades for at least a few versions, and on by default on new installs.

#150 follow-up: @adamsilverstein
2 years ago

Also, I wasn't able to find a link to the specific usage data on this, partially since Big Sur is lumped in with Catalina for usage. Is there a reference available for the ~2% falling into this category?

@costdev I believe this is a rough calculation based on caniuse data combined with data like https://gs.statcounter.com/macos-version-market-share/desktop/worldwide. I'm not sure we can get an accurate accounting of this, in part because of bugs around the OS reporting the version. 2% globally is probably too high, given Safari's market share and the percent of Safari users this effects.

#151 @eatingrules
2 years ago

@adamsilverstein The fundamental issue here is that the vast majority of users will have absolutely no idea all of this is happening, but the proposed workarounds to avoid/solve issues keep putting the onus on users.

Most end users probably don't even know these forums exist, let alone visit them. I'm sure only a miniscule fraction even read the big release notes on the Core "About" page after an update. Most of our users completely ignore dashboard notices, too...

So if this feature gets released in 6.1, I'd wager that ~99% of users won't ever realize it...and they certainly won't understand that, sometime down the road, they are going to blow up their storage when thumbs get regenerated.

If users don't know that their site is even generating WebP, how would they know they could do any of these things to avoid issues on their sites?

#152 follow-up: @adamsilverstein
2 years ago

stopping the needless generation of never used thumbnails just because the size definition was added

The tricky part there is knowing which images are actually needed. Waiting until they are requested probably requires .htaccess changes, which while possible, will not work for all setups. We might be able to improve on that, perhaps we can combine some knowledge from the block editor (and its breakpoint driven previews) to derive the images we expect to need.

What do you see as the main problem with pre-emptively generating all sizes, primarily a storage concern? Certainly something to consider, but challenging to build in a way that works across WordPress installs.

Click on the “relative usage tab” to get a good visual https://caniuse.com/webp

Neat, I hadn't seen this before.

mobile safari running on iOS 13 and earlier, that’s a tiny number that probably should be given some consideration if only because serving a larger fall back on mobile would be very bad practice.

this would suggest keeping a much smaller size as JPEG, right?

Comparing SSIM is a bit more complicated than just looking at file size but there is data out there on this.

yes, we wanted to repeat the studies in a WordPress context though, since our image generation stack is somewhat unique.

I do think you need to generate a full size image (not original) in jpeg format as a fallback. Just one though. I then think there ought to be a filter to enable generating Jpegs of all sizes, vs the default which should just be the full size

Right, we have discussed this before, maybe adding a new full-original-mime "size".

I still feel needs a user facing checkbox on the media setting page for “Enable modern image format conversion [ ] WebP [ ] AVIF, etc.., With those off by default on upgrades for at least a few versions, and on by default on new installs.

I don't agree we need a checkbox, to me this would be confusing for users.

Last edited 2 years ago by adamsilverstein (previous) (diff)

#153 @flixos90
2 years ago

@costdev @adamsilverstein To chime in here regarding the 2% number, I calculated that a few months ago, indeed based on StatCounter. This number is global, however the real number is somewhere between 1% and 3% global usage, which we can't determine exactly because Mac OS still reports the Catalina user agent for some newer versions like Big Sur (unfortunately it's exactly that difference between Catalina and Big Sur which defines whether WebP is supported in Safari or not).

To summarize: Because the value is somewhere between 1 and 3%, we have sometimes been speaking about 2%, but no matter where in that range the real value lies, it is still above 1%. Per WordPress policy, any browser with less than 1% global usage does not need to be supported anymore. So while this is global usage, and while we do have to support that use-case, it is also very small and certainly almost at the point where we wouldn't even need to support it anymore, so if performance there isn't perfect, it's a very low impact at the large scale we're speaking.

#154 follow-up: @adamsilverstein
2 years ago

So if this feature gets released in 6.1, I'd wager that ~99% of users won't ever realize it

Thats right. When this is released in 6.1, users will get a faster site that works and looks exactly as it did before, without having to do anything on their end. 99% of users won't notice anything, except maybe that their site seems faster.

they are going to blow up their storage when thumbs get regenerated

I think your concerns about storage are exaggerated. We are only generating a single mime type and regenerating images always has the potential to add new files, for example when new sizes are added. That is why the regenerate images plugin has the "delete previous images" option already.

#155 in reply to: ↑ 154 @eatingrules
2 years ago

Replying to adamsilverstein:

I think your concerns about storage are exaggerated.

With all due respect, I'm not exaggerating one bit. I'm basing these concerns on having worked through image optimization (via Shortpixel and Imagify) on many hundreds of sites. Sites with anywhere from 3,000 to 40,000+ images in the library. Often there are hundreds of thousands of thumbnail files, and, in some cases, the numbers reach into the millions.

As I've mentioned before, on a near-daily basis we have clients running out of disk space on their hosts because of thumbnails. In our situation, it's often the backups from Shortpixel or Imagify... and that roughly tracks with the same amount of disk space that we're talking about with the WebP Conversion (should all the thumbs be duplicated). The savings is roughly 30-50%, depending on the original images, but these plugins keep the originals so, overall, the uploads folder's disk space usage can grow by 30-50% or more.

These aren't "extreme" outliers -- they are simply active, reasonably successful blogs (most are food/recipe blogs, but we work with a variety of other publishers and e-commerce sites).

Moreover, your own research indicated that up to 14% of sites could run into hosting space issues!

So no, I don't think I'm exaggerating or overreacting here. You're proposing something that will absolutely cause issues on a significant percentage of sites over time -- possibly up to 14%, far far above the threshold that should be considered acceptable.

Last edited 2 years ago by eatingrules (previous) (diff)

#156 follow-up: @eatingrules
2 years ago

Also, the April 12 "Follow up on WebP by Default Proposal" blog post said:

Once we’ve completed our investigation and determined next steps on these two issues, we will work with the community to reassess two other concerns that were raised:

  1. Having the feature on/off by default
  2. Having a UI-based control to turn the feature on/off

I've tried to have a substantive discussion on both of these issues (including opening #56263, as @adamsilverstein suggested, which was promptly closed and directed back here). Each time the issue has been raised, it's been summarily dismissed with, effectively, "we know best" or "decisions not options."

But from what I've seen on this ticket, on the other proposal blog posts, and in Github issues, many people have been calling for this to be opt-in and to have a UI control, and those comments are being ignored as well. It seems the only people pushing against it are on the performance team (though perhaps I've missed comments from others who are in favor of keeping this hidden from end users?).

It's time for that follow up blog post to solicit more community feedback on these two questions.

#157 follow-up: @adamsilverstein
2 years ago

14%... if storage requirements were doubled. They won't be, we are actually reducing storage requirements.

Every comment from a core committer or media component maintainer has agreed about the default and UI.

#158 in reply to: ↑ 157 @eatingrules
2 years ago

Replying to adamsilverstein:

14%... if storage requirements were doubled. They won't be, we are actually reducing storage requirements.

...until thumbnails are regenerated!! And, unless someone knows they're about to create WebP duplicates of all their existing thumbnails -- and most people will not know, because you're not making it clear to them -- they won't know they're about to increase their usage by ~70%.

This will be a severe problem over time...but fine. Do what you want. I give up. 😢

#159 @seedsca
2 years ago

As somebody that has been using WordPress since 2006 or so and helps maintain hundreds of high traffic websites, I'm really appreciating the time and energy that is going into implementing this. It, hopefully, won't break many many sites or cost lots of money to unsuspecting users. Maybe it'll be a cost to them in the form of developer time to add a very simple bit of code to disable this feature.

I don't know of a single person who decided to go with this CMS over others because it's fast (and it can easily be made fast). Everybody I've heard from is here because of the options, ability to customize, and awesome community. I truly hope this makes WordPress faster and even better. However I fear it may break enough sites to cause bad press and an exodus to other CMSs. All for what, in my opinion, seems like a features pushed prematurely/forcefully.

Please be careful with this tool that has helped democratize publishing for so many and sorry for this possibly nonconstructive comment but I feel this needed to be said.

#160 in reply to: ↑ 150 @jb510
2 years ago

Replying to adamsilverstein:

Also, I wasn't able to find a link to the specific usage data on this, partially since Big Sur is lumped in with Catalina for usage. Is there a reference available for the ~2% falling into this category?

@costdev I believe this is a rough calculation based on caniuse data combined with data like https://gs.statcounter.com/macos-version-market-share/desktop/worldwide. I'm not sure we can get an accurate accounting of this, in part because of bugs around the OS reporting the version. 2% globally is probably too high, given Safari's market share and the percent of Safari users this effects.

I agree we'll never get exact numbers on some of these because the data is combined.

What we can say is based on https://caniuse.com/webp
IOS 3.2-13.7 Mobile Safari/Chrome 1.14%
1.14% matters some.

macOS < 11 Safari is some fraction of the 2.68% using Safari.
Based on https://gs.statcounter.com/macos-version-market-share/desktop/worldwide 10.10 is 1.18%. So take a wild guess what fraction of that 1.18% is ALSO using safari and not chrome. We don't know. A best guess would be browser usage on per 10.11 is the same ratio as after 10.11. So 2.68% of 1.18%, which would be 0.03%. It's hard to guess if these legacy Macos users skew more towards using Safari, or towards Chrome. Hypothetically even if they skwed towards safari 2:1, that's still insignificant.

Conclusion - What are we doing about the 1.14% of users on iOS 13.7 or earlier? Are we werving them broken webp images? are we serving them oversized jpeg images? Something else?

#161 in reply to: ↑ 152 @jb510
2 years ago

Replying to adamsilverstein:

stopping the needless generation of never used thumbnails just because the size definition was added

The tricky part there is knowing which images are actually needed. Waiting until they are requested probably requires .htaccess changes, which while possible, will not work for all setups. We might be able to improve on that, perhaps we can combine some knowledge from the block editor (and its breakpoint driven previews) to derive the images we expect to need.

What do you see as the main problem with pre-emptively generating all sizes, primarily a storage concern? Certainly something to consider, but challenging to build in a way that works across WordPress installs.

Storage (KB and file count) and CPU cycles. I think absent from this conversation is that every KB of storage is actually 10-20x that much when you take backups into account. Now, I'd prefer versioned backups that don't store really-big-image.jpg uniquely in ever single backup, but most don't do that. File count because at scale it affects performance. CPU cycles because while "cheap" in this context that too affects performance. All together, all this also has an avoidable environmental impact.

htaccess is one way, but I imagine a JS based worker could handle it as well independent of server config.

Click on the “relative usage tab” to get a good visual https://caniuse.com/webp

Neat, I hadn't seen this before.

mobile safari running on iOS 13 and earlier, that’s a tiny number that probably should be given some consideration if only because serving a larger fall back on mobile would be very bad practice.

this would suggest keeping a much smaller size as JPEG, right?

See my comment immediately above. Looks like 1.18% for iOS without WebP support. MacOS certainly can be completely ignored, but 1.18% for older iOS devices seems significant. I loathe the idea of a JPEG for every thumbnail size and am _very_ happy we've moved away from that. I don't love the solution, but generating 1 full size jpeg and serving that anytime a jpeg fallback is needed seems like an option. The problem with that I imagine is dealing with the various hard/soft crops. To me, the right solution remains, only generate the JPEG if/when it's actually been requested.

Comparing SSIM is a bit more complicated than just looking at file size but there is data out there on this.

yes, we wanted to repeat the studies in a WordPress context though, since our image generation stack is somewhat unique.

Well, "WordPress context" means a user has uploaded an image with anywhere between 50% quality and 100% quality. You just never know... But the photographers most concerned already dislike t

I do think you need to generate a full size image (not original) in jpeg format as a fallback. Just one though. I then think there ought to be a filter to enable generating Jpegs of all sizes, vs the default 82% jpeg quality feelings it's too aggressive (I disagree, but such is the case with subjective opinions). My point is simple that is a 100% jpeg gets uploaded, the WebP is going to come out much better than the jpeg if both files end up being 200KB. The SSIM score on the WebP will be way closer to the original than the jpeg will be. Anyway, think it's been discussed ad nauseam. I don't think we need jpeg fallback because in a tiny number of cases they're larger than jpegs.

This is a good example of decisions not options. Making decisions (webp only thumbnails) is great when it actually simplifies things for the end user. Users shouldn't be concerned whether their thumbnails are jpeg or webp, just that they work.

Right, we have discussed this before, maybe adding a new full-original-mime "size".

I still feel needs a user facing checkbox on the media setting page for “Enable modern image format conversion [ ] WebP [ ] AVIF, etc.., With those off by default on upgrades for at least a few versions, and on by default on new installs.

I don't agree we need a checkbox, to me this would be confusing for users.

The problem with saying "decisions, not options" here is that it's being misapplied (in my opinion) when is about not providing a very simple and useful option from the user. In this case enabling/disabling specific image format support. That's a single checkbox that does something simple. The alternative here, a filter and/or plugin to effect a filter is WAY more complicated. You're actually forcing a very complicated solution on users to avoid a single check box. Decide to provide a simple option, instead of deciding to force a complicated one.

I don't see how this would be a consuming option. It's a check box that says "do you want WP to convert your images into a more performant format, WebP. A lot of users are going to say "no, I want to keep using JPEGs" for whatever reason.

Again, in terms of the 99% impact debate. Sites with old images aren't going to benefit from WebP at all, because those images are hard-coded URLS pointing at jpegs. It's wildly wrong to presume existing content will benefit from WebP. Keep in mind just how recent blocks are and how little content has been built with blocks vs the classic editor. However, all most new content going forward will benefit!

I believe @eatingrules concern is that WebP images are still going to be generated for existing images, images that already have JPEG thumbnails. So if I have an existing media library that is 100GB, what is the impact on regenerating thumbnails on that library the day after this launches?

If WP was smart enough to only generate images that were actually requested, this would be non-issue. But if WP is going to say "well I need to generate webp thumbnails of every old image just in case someone inserts it in the future", that's the problem.

#162 in reply to: ↑ 143 @azaozz
2 years ago

Replying to adamsilverstein:

We have completed a round of research on image sizes to investigate the best/any threshold for creating WebP images.
...
Note that we are currently discussing the best compression quality to use for WebP on this ticket, and are likely to change the setting from the current 86 to either match the JPEG setting (82) or somewhere in between

As far as I know older versions of the WebP encoding libraries may produce images with poorer quality when this is lower. Thinking perhaps it would be better to err on the side of the better quality and a bit larger size here to compensate.

To summarize, the research shows that across all image sizes, WebPs benefits far outweigh any adverse results (even for the current quality of 86, and much more so for 82). Based on these averages, it makes sense to generate WebPs for all image sizes. The vast majority of generated images will be smaller, and the few that are larger will be only slightly larger.. Therefore, I don’t think we need a threshold filter.

Thanks for looking at this again.

One addition to consider is enhancing add_image_size similar to what is proposed in https://core.trac.wordpress.org/ticket/56288. This would let developers opt out of the mime transform for specific sizes, so a newsletter plugin could add their own custom size that would always be JPEGs.

Yes, think this makes sense for now. Eventually emails/newsletters will also start to support the newer image formats and this won't be needed but there may be other edge cases where forcing specific image format may be good.

To ensure compatibility for use cases where WebP may not be supported (OG tags, older safari and email), we can rely on the “original_image” meta when present (or the full size image when missing). Developers can use wp_get_original_image_url to retrieve the JPEG version.

We may want to consider always creating a full sized WebP version of uploaded images to ensure we always have a “full” sized WebP (even for those below the big_image_size_threshold; above that size we would generate a WebP image already). In this case, original_image meta would always be present.

Thinking this will probably be more complex. When the uploaded image is larger than the "big image" threshold (2560px) WP creates a "full" size for use on the internet. When created by WP that full size should be WebP according to the above research. Perhaps it can be JPEG and used as fallback, but seems quite large. In addition it won't work for cropped image sub-sizes.

On the other hand having the original image (regardless if it was made by WP or not) always be the same mime type as the uploaded image makes sense, and makes it quite easier for back-compat.

When processing uploaded images the basic logic is:

  • Never change the originally uploaded image file except to rotate it according to EXIF.
  • If the image file is large, make a smaller version that's more suitable for use on the internet and set it as "original image".
  • If not too large, use the originally uploaded file as the largest size.

To satisfy all fallback requirements without generating images on the fly it seems that it will need:

  • One image with the original ratio. Thinking perhaps a mid-sized image would work well here, perhaps the "large" 1024px.
  • A JPEG version for each cropped image sub-size. Alternatively all cropped images can be JPEG (thinking that's better?) or only some known sub-sizes can be WebP.

Regarding the inclusion of the infrastructure for multiple mimes: this would no longer be strictly required for only outputting WebPs.

Yes agreed. The multi-mime support should be added from its own ticket where it can be researched/tested/reviewed easier.

Worth noting that the multi-mime architecture has been available/tested in the Performance Lab Plugin

I think what's missing here is more docs/examples/explanations of how it is implemented and why. For example how is the image meta changed. How is backwards compatibility maintained, what ensures the changes are good enough when going forward, etc.

One enhancement I see is if we introduced multi-mime support along with async image generation, we could enable generating non-default mime types only when we need them, e.g. adding a $mime_type parameter to media functions, then generating the requested mime type if missing.

Yes, definitely. If WP can generate image sub-sizes on the fly, I think most will be generated that way. Only one or two will probably be still generated on upload, likely the 1024px as it's used the most in the editor, and maybe the thumbnail.

Although I would love to see a larger refactor, that shouldn’t be a blocker for including this feature if we are confident of its value.

I'm still thinking that it is a bit premature to add it now. It will be needed before adding more image formats for sure. It will probably also be needed to enhance and extend how WP handles JPEGs, PNGs, and WebPs in the future (eventual <picture> tag support, etc.). But think it will be better to add it when it is going to be used more widely both in wp-admin and on the front-end.

This is starting to sound like a pretty good plan for this part of WordPress for the next few years. I hope we will be able to implement it well :)

Based on this I will prepare two commits: one to revert the existing milti-mime commits, and one to introduce the WebP output change (so that it can be reviewed separately). Once this is in place we can follow up and evaluate if we want to add additional “full” size image handling, and update the shim we developed to swap out images for Safari.

Thoughts?

Sounds great :)

#163 follow-up: @davidbaumwald
2 years ago

First, I'd like to say that I appreciate the vast amount of work put into this feature by everyone involved.

After monitoring this ticket for some time and gauging input from the community, I think it's fair to request a formal discussion about adding a Media setting to control this behavior. Most of the community feedback around this feature is that it should be opt-in.

I see some responses mentioning the "Decisions, not options" philosophy, but, as @jb510 stated, this is not an absolute. There are many settings for various features in Core. Some much more esoteric, yet potentially less impactful than this. I think a rationale should be put forth to the community as to why a setting is off the table here, other than it being "confusing". That could apply to any percentage of users for a given, existing setting in Core.

Listening to and acting upon community feedback plays an important role in open source. The general sense I gather is that the community understands that WebP could potentially make their sites faster, but an even greater number are requesting an easy, familiar way to control favoring WebP output over JPEG. I understand that the "vocal minority" may apply here, but that could honestly apply to any feature-inclusion discussion here on Trac because those of us that do engage here are many orders of magnitude less in numbers than the user base as a whole.

#164 in reply to: ↑ 156 ; follow-up: @azaozz
2 years ago

Replying to eatingrules:

But from what I've seen on this ticket, on the other proposal blog posts, and in Github issues, many people have been calling for this to be opt-in and to have a UI control, and those comments are being ignored as well.

Don't think they are being ignored, but the "to UI or not to UI" question is not that simple. It has been discussed many times over the last 15 years.

The general ideas are like in MacOS and Windows. One has few options in the UI, most things have to be set from cli (pretty complex for a lot of people). The other has tons and tons of options. Most things can be set from the UI with some digging, but many users have no idea what to choose.

So, which is better? :)

WordPress has chosen the "MacOS" way long ago. It is the philosophy that things should "just work", and not ask the users questions they do not know the answers to.

Of course, the site owners should be able to control how their sites work and behave, if they are so inclined. For that a plugin would be best. It takes just few more seconds to download and install it, and instead of a checkbox the site owners will be able to enable/disable it.

It's true that "many people have been calling for this" on GitHub and Trac, but consider who these people are. Web Developers or otherwise engaged in the development of WordPress. A good rule-of-thumb would probably be to ask few people that have nothing to do with software development, like perhaps family and friends. Would be good to see what they know about WebP and its advantages and disadvantages :)

#165 in reply to: ↑ 163 @azaozz
2 years ago

Replying to davidbaumwald:

Listening to and acting upon community feedback plays an important role in open source.

Right, but what is the feedback here? I see about 10-12 people having the opinion that a checkbox may be a good idea. Another 5-6 thinking it may not be.

Looking a bit deeper, what is the difference between a checkbox and a plugin? Not that large. In both cases the user will have to know what they want before changing the setting or installing the plugin.

The other idea, default to "off", makes this useless imho. Usually 95% of options are never changed from their default values. This is a new feature that would enhance millions of web sites and bring this part of the WordPress image handling to 2022. Why would it be potentially enabled on only 5% of sites? Makes no sense to add to core at all in that case.

#166 @jb510
2 years ago

UI - the folks being vocal about the UI for this are perhaps are developers, that's kind of aways the case here, however they're developers voicing a concern on behalf of their many clients none of who participate directly in these discussions.

For my personal use I could care less about a UI for this. I've said before I'd personally turn this on and delete every single JPEG on my blog. I don't care about image quality, the worse the better maybe fewer people will steal the photos from my blog. However, I support 100s of clients/users and I am certain the vast majority will want a basic UI to turn this on/off on existing sites.

People are very sensitive about what happens with their images after they upload them.

Again "Decisions, not options" is about keeping it simple, like MacOS, but it's not about zero options ever. It'd be like saying "well we shouldn't give users settings for media sizes, or permalinks, or date format, or whatever. Seriously, people need to re-evaluate how they use that phrase.

Folks here are only arguing that we should have a toggle to enable format conversion (ie. upload a jpeg but generate WebP). I do believe changing image formats is a big enough thing it deserves an option to control if it happens or not. One check box. That's simple and sensible.

Nobody (I don't think) is arguing that we should have a checkbox, plus a quality control slider, plus another checkbox to save originals and another to delete originals on upload. That indeed would go way too far.

IMHO nothing is a worse answer to this sort of situation than "well there is a plugin for that". I shouldn't have to install 5 freaking plugins to disable comments, enable media replace, set up an SMTP server, AND now also control if my uploaded images change the file format or not after being uploaded.

These aren't complicated options to be avoided, they're very basic site controls.

EDIT TO ADD: I, and I think others, are saying it should default to off on existing sites at the time up WP upgrade. However, that it should default to on on new installs. I'd even say the upgrade ought to prompt users to turn it on like we did for enabling Gutenberg at first. However, that need is in part based on HOW it interacts with existing images should regenerate thumbnails be run by a user or a plugin. That's the core of that concern, is does this have the potential to generate webp sidecars for millions of existing images, sidecar thumbnails that won't actually be used since those old images were inserted into the classic editor.

Last edited 2 years ago by jb510 (previous) (diff)

#167 in reply to: ↑ 164 @eatingrules
2 years ago

Replying to azaozz:

It's true that "many people have been calling for this" on GitHub and Trac, but consider who these people are. Web Developers or otherwise engaged in the development of WordPress. A good rule-of-thumb would probably be to ask few people that have nothing to do with software development, like perhaps family and friends. Would be good to see what they know about WebP and its advantages and disadvantages :)

Thanks again for jumping in on this. I couldn't agree with you more -- and that's exactly why I've been pushing back so strongly. I am particularly well-qualified to represent the WordPress users who have nothing to do with software development. I know them well; I've built my entire business on helping them.

To be clear, I'm not fundamentally opposed to WebP. I understand the positives here. Smaller images are better. I also get that for most sites, this will be beneficial and likely not cause issues.

But there will be negative consequences for a very significant number of sites -- and those consequences have not been addressed sufficiently.

It has also felt like the Performance Team is going to barrel ahead no matter what, and now the ONLY possible thing we have left is to try to get it to be off by default and/or have a UI component. It's a hail-Mary pass, frankly.

With the current proposal, the user will be completely unaware of that this is now happening on their site, and that will actually make the likelihood of issues even greater.

However, I know from experience with our clients that the vast majority of "non-technical" people, when presented with clear information and corresponding pros/cons, will be able to make the choice that is right for them and their sites.

So at least if it's off by default and has a toggle, we can say "Hey, there's this new feature, it's has these benefits, but just be aware of XYZ potential downsides" then they can choose to turn it on and try it.

I am still certain that there will be a significant number of problems down the road (most acutely due to future thumbnail regeneration) -- and, as currently proposed, it will be particularly bad because site owners won't have a clue as to why their site's disk usage has skyrocketed sometime in the future.

The other idea, default to "off", makes this useless imho. Usually 95% of options are never changed from their default values. This is a new feature that would enhance millions of web sites and bring this part of the WordPress image handling to 2022. Why would it be potentially enabled on only 5% of sites? Makes no sense to add to core at all in that case.

You're right: This shouldn't be in core! At least, not yet.

But we're really going in circles here because there's a fundamental problem with how WordPress deals with thumbnail images. Until that's fixed with some kind of "on first use" system (therefore reducing a huge amount of unused thumbnails), any changes to how core deals with images/thumbnails is just going to exacerbate the issue.

So how about putting this on pause, and instead creating a "Media Team" tasked with overhauling the fundamental issues with thumbnails? WebP thumbs could then be incorporated properly (and any future formats could as well). This would be a MASSIVE WIN for WordPress!

Last edited 2 years ago by eatingrules (previous) (diff)

This ticket was mentioned in Slack in #core-media by mike. View the logs.


2 years ago

#169 @snoringdragon
2 years ago

Hi,
What about making it opt-in for WordPress v6.1 by providing UI.
We can add another suggestion in site health for giving it more exposure.
Even if only 1% users enable it as soon as it is available, it will help us to see if there are any bugs or edge cases left to be covered.
For WordPress v6.2, we can make it opt-out once asyc media generation is ready.
That time, we can also introduce multi mime support as all things will be in place.

#170 @TweetyThierry
2 years ago

Heya all,

Just jumping in and adding my two cents here. I hear everyone's concern and ultimately there are some tradeoffs in every path we could take. With that in mind, we could get lost discussing the tradesoffs for days/weeks/months without moving forward. We are surely not going to find a solution that suits everyone, but should find one that suits the majority.

I will share my perspective on the things discussed in this ticket:

  • Not generating both JPEG and WebP version for image sizes: +1, @azaozz (and others) raised some valid concerns and it makes sense not to do so
  • Removing mime types infrastructure: -1, @adamsilverstein shared the benefits of the infrastructure which I agree with. To me it is an elegant architecture to support WebP but more importantly for future enhancements. With that said, it seems like a reasonable outcome to remove it for now if folks feel strongly about it.
  • Having an opt-in/out UI: -1000, this is a total no go from my perspective. This makes this feature useless, puts the burden on the user, bloats the admin UI, prevents WordPress as a platform to move forward (by the way, many other CMSes already have WebP by default which proved to be successful) and frankly only cater for some use cases which some folks may feel strongly about but is not applicable to the vast majority.

The other idea, default to "off", makes this useless imho. Usually 95% of options are never changed from their default values. This is a new feature that would enhance millions of web sites and bring this part of the WordPress image handling to 2022. Why would it be potentially enabled on only 5% of sites? Makes no sense to add to core at all in that case.

Strong +1, for all the reason I mentioned above (at the risk of sounding like a broken record). It also feels to me that this was already discussed and agreed upon by the majority and by core committers/leads not to introduce a UI option.

Some general thoughts regarding the importance of this feature: WordPress as a platform has to move forward, "don't touch it if it is not broken" motto is a failure state which prevents evolution and leads to stagnation. This may be ok in some cases, but not for a platform which powers a big chunk of the web, failing to do so puts the entire ecosystem at risk. Evaluated tradeoffs are ok and inevitable.

Last edited 2 years ago by TweetyThierry (previous) (diff)

#171 @johnbillion
2 years ago

I've not been following this feature too closely so I won't comment whether it's ready for inclusion or not, but I want to make one important point about introducing a checkbox in the UI to enable or disable the feature. This can only be done if the user is able to make an informed decision about the impact of the option, and I don't believe that a setting such as "Enable WebP image format" is something that a non-technical user is able to make an informed decision about.

On the surface, of course they should enable WebP because it makes their website faster for their users, but understanding the downsides is a complex issue, as seen above. For that reason introducing a checkbox in the UI to enable or disable this feature is a no-go. Either the feature is stable and reliable enough to be enabled for all users on all platforms, or it's not.

#172 follow-up: @flixos90
2 years ago

I agree with the latest two comments, and to me this discussion has increasingly become about general sentiment and opinion rather than facts. Additionally, I see some arguments being raised again which aren't relevant anymore based on the updated direction we've been leaning towards most recently. Most of the points here have been discussed for a long time already, though I'd like to point out the main new aspect that had not been considered prior is the increased resources needed for an upload as pointed out by @azaozz and @dd32 just a few weeks ago. As @TweetyThierry already mentioned above, there is no decision in this which will satisfy everyone, so we will need to make a decision that benefits the vast majority (>80%) based on the data we have. So I'd like to compare the approaches here, listing the benefits and tradeoffs over "doing nothing" for now (thus keeping the status quo).

Generating WebP instead of JPEG:
(while keeping at least the original JPEG around)

Benefits:

  • roughly 30% smaller image files (29% if we choose quality 84, 36% if we choose quality 82) for ~98% of global users, leading to faster performance
  • reduced filesystem storage needs to ~70% of what it was before
  • similar resource needs on image upload

Tradeoffs:

  • regression in performance for the remaining ~2% of global users who are on old browsers (due to loading a higher resolution JPEG image)

Generating WebP in addition to JPEG:

Benefits:

  • roughly 30% smaller image files (29% if we choose quality 84, 36% if we choose quality 82) for ~98% of global users, leading to faster performance
  • similar experience for the remaining ~2% of global users who are on old browsers (as JPEG for every size will be available)

Tradeoffs:

  • increased resource needs on image upload, leading to timeouts more often
  • increased filesystem storage needs to ~170% of what it was before

Based on the above, and also based on much of the recent conversation here, it makes most sense for the vast majority to generate WebP instead of JPEG.

Regarding the multiple MIMES infrastructure: I am undecided on whether to keep it or not at this point, though I definitely think in the long term it will be beneficial. The main benefit for keeping it now would be that it would allow site owners to go with the 2nd approach above, essentially giving them a third option in addition to "Only WebP" vs "No WebP". Many contributors were also requesting that in earlier discussions of the feature, and arguably on a good tier host with solid resources this should be possible. With that said though, I also see that it may be safer to not ship it until we are more confident there will be no timeouts on upload (e.g. by implementing fully asynchronous upload, as suggested above already).

#173 in reply to: ↑ 172 @eatingrules
2 years ago

Replying to flixos90:

Generating WebP instead of JPEG:
(while keeping at least the original JPEG around)

Benefits:

  • roughly 30% smaller image files (29% if we choose quality 84, 36% if we choose quality 82) for ~98% of global users, leading to faster performance
  • reduced filesystem storage needs to ~70% of what it was before
  • similar resource needs on image upload

Tradeoffs:

  • regression in performance for the remaining ~2% of global users who are on old browsers (due to loading a higher resolution JPEG image)

You left off this tradeoff with the current plan (Generating WebP instead of JPEG) -- which will happen when thumbnails are regenerated:

  • increased filesystem storage needs to ~170% of what it was before

Sorry to keep bringing this up, but it seems like some people still don't recognize how significant this problem will be.

@adamsilverstein pointed out that the Regenerate Thumbnails plugin can delete old thumbs during the process - and that's potentially a solution, provided the user knows they should do that.

Perhaps user education and other solution(s) for this can be worked into the most common tools that regenerate thumbnails, before WebP is included in Core?

(Here's a start to that list: Regenerate Thumbnails, reGenerate Thumbnails Advanced, WP-CLI, and WooCommerce.)

#174 follow-up: @jb510
2 years ago

TY @flixos90 for that summary. I agree some opinions are getting repeated based on older proposals, so the summary is helpful.

If I may, these are questions I haven't seen answered based on the latest proposal (WebP only).

Q1) What happens on a site with 100GB of existing images when thumbnails are regenerated (wp media regenerate, by a plugin, or by woo-commerce)? If this doesn't "blow up large sites" I'll begrudgingly drop my constant pleas for an option.

Q2) Is anything being done to where WebP would actually benefit existing content (ie. old posts/pages written with the classic editor)?

Q3) How are the < 2% of browsers that don't support WebP going to be addressed (again, that's predominantly old iOS devices)

Why I think these questions matter.

If the answer to 1 is that it is going to generate WebP images, than we still have a problem with increasing file storage unexpectedly on existing large sites that we still need to clearly address, and if not with an opt-in, then how? My only suggestion here would be to use NOT generate WebP sidecars for existing images by default nor when media regenerate is run unless a parameter is passed. Instead pass in an opt-in style parameter to force WebP generation for existing images, something like wp media regenerate --webp-existing. This becomes complicated in terms of code but maybe the image meta helps differentiate.

If the answer to 2 is no, there is no benefit to old content, then again, the benefits are being for existing sites are grossly overestimated. That doesn't mean we should do WebP, just we should better consider the cost/benefit of this affecting existing images and/or consider how we could replace those hard-coded images with WebP versions. And just to be clear, I totally think that becomes plugin territory. Someone can write a plugin that 1) generate WebP sidecar files for existing image AND does the search and replace on existing content to replace hard-coded jpegs with WebP. But without that S&R we're burdening millions of sites with WebPs for old images that will never be used.

And 3, it's just unclear what is the plan to address that 2% at this point. I'm not terribly concerned about it, just want to hear a plan articulated so debates around it can stop.

And just to be clear. I do think we've all together made tremendous progress in improving how this is going to be implemented in the last few weeks, so thank you all for that.

#175 @seedsca
2 years ago

Like @johnbillion (Core Committer) said:
"Either the feature is stable and reliable enough to be enabled for all users on all platforms, or it's not."

Well, is it? If not (I don't think it is) then this introduces breaking changes, so we don't commit. If it is still merged then there should be an opt in option. Pretty simple. Then we can gather info from there and maybe, down the road, enable it by default for new images/sites.

The argument that we should only concern ourselves with the majority is concerning... Specially when we don't really have good numbers/data on this. In my opinion the amount of hosts that replied to form our numbers are lackluster. I would not bet on those numbers...

#176 in reply to: ↑ 174 @adamsilverstein
2 years ago

Q1) What happens on a site with 100GB of existing images when thumbnails are regenerated (wp media regenerate, by a plugin, or by woo-commerce)? If this doesn't "blow up large sites" I'll begrudgingly drop my constant pleas for an option.

I took a look at this a bit more carefully. With default options, you wouldn't get extra files due to WebP:

  • wp media regenerate deletes old thumbnails by default unless you pass --skip-delete. You can also pass --only-missing to regenerate only for new or changed sizes, sizes with existing meta wouldn't be regenerated. new sizes would add files regardless of WebP output or not.
  • regenerate-thumbnails plugin (no longer supported) by default "Skips regenerating existing correctly sized thumbnails" - meaning WebPs would only be generated for newly added sizes.
  • regenerate-thumbnails-advanced similarly only generates missing/changed sizes by default ("Don't redo existing").
  • woocommerce - as far as i can tell looking at the code only regenerates sizes that have changed dimensions, or new sizes.

Q2) Is anything being done to where WebP would actually benefit existing content (ie. old posts/pages written with the classic editor)?

yes. by default, WordPress filters the content, so images inserted from the media library (not by URL) will get updated on the fly to newer subsizes, if you wp media regenerate older content should start using the new WebPs (and your old JPEGs will be deleted).

Q3) How are the < 2% of browsers that don't support WebP going to be addressed (again, that's predominantly old iOS devices)

using a tiny bit of feature dection JS on pages with WebPs we can swap out images for the smallest JPEG we have (possibly the original upload). The approach would be similar to https://github.com/WordPress/wordpress-develop/pull/3034

Last edited 2 years ago by adamsilverstein (previous) (diff)

#177 @eatingrules
2 years ago

Thanks for looking into those @adamsilverstein. That goes a long way to alleviate my concerns about the thumbnail regeneration!

Quick question, as I've lost track of a detail in the current plan:

If someone uploads a JPG image, will the original file stay as-is? Or is it going to be converted to WebP and then deleted? Or will a WebP also be generated (in the same size), and the JPG will stay on the server?

#178 @jb510
2 years ago

@adamsilverstein thank you for answering/clarifying those cases.

#179 follow-up: @adamsilverstein
2 years ago

If someone uploads a JPG image, will the original file stay as-is? Or is it going to be converted to WebP and then deleted? Or will a WebP also be generated (in the same size), and the JPG will stay on the server?

WordPress always keeps the original upload untouched.

We are currently discussing if we should always create a WebP "full size" image as well - thats for small images. For large images, WordPress already creates a -scaled version and this will now be a WebP by default.

#180 @adamsilverstein
2 years ago

  • Keywords has-patch commit added; 2nd-opinion needs-testing changes-requested removed

revert-53786-53848-r53847-r53845-r53751.diff Reverts the initial multi-mime support and several follow up commits, restoring media to state prior to the initial commit. Tests running for this patch are green in https://github.com/WordPress/wordpress-develop/pull/3161 and I verified media uploads work as expected. A few doc block improvements were retained, see the PR conversation/commits for more details.

reintroduce-webp-default-output.diff reintroduces WebP by default (as well as the name collision fix we introduced).

Follow up work including add_image_size support and a capability shim is already underway and will be added in follow up tickets. Tests are green in https://github.com/WordPress/wordpress-develop/pull/3162 which includes this + the revert code.

#181 in reply to: ↑ 179 @eatingrules
2 years ago

Replying to adamsilverstein:

If someone uploads a JPG image, will the original file stay as-is? Or is it going to be converted to WebP and then deleted? Or will a WebP also be generated (in the same size), and the JPG will stay on the server?

WordPress always keeps the original upload untouched.

We are currently discussing if we should always create a WebP "full size" image as well - thats for small images. For large images, WordPress already creates a -scaled version and this will now be a WebP by default.

Ok, thanks Adam!

#182 @adamsilverstein
2 years ago

reintroduce-webp-default-output.2.diff adds some additional tests, see PR comments for details.

#183 @adamsilverstein
2 years ago

reintroduce-webp-default-output.3.diff includes all the final changes from the PR and is ready for commit immediately after the revert commit.

#184 @adamsilverstein
2 years ago

In 54085:

Media: revert the multi-mime feature.

This feature isn't quite ready to land.

Reverts r53786, r53848, r53847, r53845, r53751.

Props flixos90, azaozz, dd32.
See #55443.

#185 @adamsilverstein
2 years ago

  • Resolution set to fixed
  • Status changed from assigned to closed

In 54086:

Media: Output WebP by default when uploading JPEGs.

Uploaded JPEGs will automatically be converted to WebP sub-sizes instead of JPEG, saving space and making sites faster.

The original JPEG upload is always retained and can be accessed by calling wp_get_original_image_url.

Props azaozz, flixos90.
Fixes #55443.

This ticket was mentioned in PR #3208 on WordPress/wordpress-develop by felixarntz.


2 years ago
#186

This is a follow up to https://core.trac.wordpress.org/ticket/55443: The WP_Image_Editor classes are loaded also in the frontend and could theoretically be used there, so while I suggested otherwise on the original pull request, on second thought I think it's more appropriate to hook in the filter also in the frontend.

Trac ticket:

adamsilverstein commented on PR #3208:


2 years ago
#187

+1 makes sense to me to load this way.

#188 @flixos90
2 years ago

In 54094:

Media: Move wp_default_image_output_mapping() filter callback to frontend scope.

While the image_editor_output_format filter is primarily used in WP Admin, it can also be executed in frontend scope, as the related WP_Image_Editor class and wp_unique_filename() function are being loaded in that scope.

Follow up to [54086].

See #55443, #56526.

#190 @azaozz
2 years ago

Just a nitpick but seems the change to generate_filename() first param handling is not used here. The part that changes if ( ! $suffix ) to if ( null === $suffix ).

Best to put in a separate ticket as allowing to pass '' (empty string) in order to not auto-generate a suffix seems to be an enhancement.

adamsilverstein commented on PR #3048:


2 years ago
#191

This is no longer needed since multi-mime was reverted.

I tested on trunk and verified that the JPEG output for PDF uploads are already converted to WebP output using the current approach:

https://i0.wp.com/user-images.githubusercontent.com/2676022/188965775-cd6d8a3b-ee9b-451a-9b7e-fdd315b7a858.png

adamsilverstein commented on PR #3034:


2 years ago
#192

Nice work @mukeshpanchal27!

This is ready to be refactored based on the new approach in core, a few thoughts on what is needed:

  • Only the 'sizes' meta array exists, we no longer create the 'sources' data, so we should look through all sizes to find any potential JPEG replacements
  • We should also include the URL of the original image with wp_get_original_image_url since that is the only image we can be sure will be available in the original JPEG form.
  • When replacing image URLs in non supporting browsers, we should choose the next largest available size with the same aspect ratio, always falling back to the original upload as a last resort
  • I feel we should drop the complexity of handling srcset at all, this is a "minimum viability" shim designed to ensure (a very small % of) users see images instead of broken boxes. I suggest just swapping the image out with a new image with best image URL we can find. Creating new elements is probably safer as well.
  • the logic to hook in will be different, although the $_wp_image_mime_fallback_should_load global approach might still makes since since I think we want to consider every image on the page and only enqueue the shim when some images include WebP mime sub-sizes.
  • also, I wonder if we should run $_wp_image_mime_fallback_should_load through a filter right before using it so developers can fine tune the behavior. not sure that is needed but worth reviewing how similar shims are loaded and controlled in core.

adamsilverstein commented on PR #3060:


2 years ago
#193

@mukeshpanchal27 Since we reverted multi-mime support this is not currently needed, so I am closing this out. Thanks for your work here, should/when we reintroduce multi-mime support this will come in handy.

adamsilverstein commented on PR #3030:


2 years ago
#194

@mehulkaklotar - Since we reverted multi-mime support this is not currently needed, though it will come in handy if we reintroduce the feature so thanks for your work here! Closing for now.

#195 @flixos90
2 years ago

In 54097:

Media: Generate WebP only for certain registered image sizes.

The existing filter image_editor_output_format receives an additional parameter $size_name which is populated whenever it controls the output format for a specific registered image size to create. Otherwise, it remains empty. In order to achieve this, a low level change has been added in bringing a new $size_name class property to the WP_Image_Editor base class, which is introduced in a backward compatible way that will not cause conflicts with custom implementations.

This parameter is then used in new logic inside the wp_default_image_output_mapping() callback function for the filter, controlling whether image/jpeg should map to image/webp output or not. By default, this is enabled for all WordPress core image sizes by default, and this list can be modified using a new wp_image_sizes_with_additional_mime_type_support filter, e.g. to remove core sizes or add custom sizes.

The customization per image size may be further enhanced by providing a more declarative API via a new parameter on the add_image_size() function.

Props eugenemanuilov, flixos90, adamsilverstein, joegrainger.

Fixes #56526.
See #55443, #56288.

#196 @flixos90
2 years ago

  • Keywords commit removed
  • Resolution fixed deleted
  • Status changed from closed to reopened

Reopening this just for the needs-dev-note aspect.

The eventual dev note for this should probably encompass the follow up tickets to this as well:

felixarntz commented on PR #3034:


2 years ago
#197

@mukeshpanchal27 This is now ready to proceed, however it has to be updated to the new approach where only WebP or JPEG is available per image size. No sources data is present anymore, so this needs to be fully dependent on the single image that is available for each size. I've opened a new Trac ticket specifically for this functionality: https://core.trac.wordpress.org/ticket/56529

I believe most of the code in this PR can remain unchanged. The two main things that need to be updated are:

  • The wp_image_use_alternate_mime_types() function no longer exists. So we need to handle setting the global $_wp_image_mime_fallback_should_load in a different way. Since we now no longer do frontend replacements of JPEG at all, I suggest we instead incorporate this somewhere into wp_filter_content_tags() or potentially a new helper function where we check whether the content includes any .webp image, and if so set the global so that the minimal fallback script is included.
  • The actual JS code in the wp-image-mime-fallback.js script needs to be changed according to the new conditions. We will now have to replace every WebP version with the best available JPEG version, which is
    • the smallest available JPEG with at least the dimensions of the WebP
    • using the same aspect ratio

cc @adamsilverstein

#198 in reply to: ↑ 137 ; follow-up: @CyberCr33p
2 years ago

Replying to eatingrules:

Replying to spacedmonkey:

Just a thought. How about this.

Allow users to upload jpgs are normal. Add a warning / check in the editor. Something like.

“It appears you are using images that are not optimised, would you like to optimise them?”

Then when you click okay, covert images to webp and remove jpeg version.

Then add a check setting to “optimise all images by default”.

Interesting idea. Maybe a bit clunky, but at least this way it's not just creating extra images in the background with a site owner oblivious to what's going on, and instead gives them control and can help them make an informed decision.

It seems to me that all the concerns people have raised that haven't been addressed yet could be solved in one of two ways:

  1. Overhaul how WordPress deals with thumbnails, generating as-needed on first use. (A huge undertaking, but would be an awesome outcome!). This would provide plenty of headroom for both JPG and WebP files to be saved to disk as-needed.
  1. Add an option in the Media Settings, disabled by default, that can enable WebP generation... it could explain both the pros and cons so people can decide what's best for them. There could even be a dashboard notification suggesting people turn it on.

As a webhost hosting many thousand of wordpress websites I agree 100% with everything you said.

I would like only to add these:

  1. Servers will use more CPU to create the extra images. More CPU usage = more energy cost
  1. More servers will needed for webhosting / keeping backups. More servers = more energy cost
  1. Webhosting customers in many cases will have to pay more to host their websites.
  1. Developers will spend a lot of time to remove the webp images and disable this "feature".

#199 in reply to: ↑ 198 ; follow-up: @azaozz
2 years ago

Replying to CyberCr33p:

  1. Servers will use more CPU to create the extra images. More CPU usage = more energy cost

I think you may be misunderstanding this a bit. The latest decision here was to not create two images for each sub-size, only make WebPs. This would not increase the server resources needed after uploading an image, and will reduce storage space requirements as WebPs are smaller in size than JPEGs.

#200 @eatingrules
2 years ago

Just FYI, in an effort to help keep everything consolidated in this Trac ticket:

On Sunday Matt said this should not be included in Core for 6.1 and instead should be in a Canonical Plugin.

The Performance Team is currently deciding how best to proceed. See: Performance team meeting summary 13 September 2022

#201 in reply to: ↑ 199 @CyberCr33p
2 years ago

Replying to azaozz:

Replying to CyberCr33p:

  1. Servers will use more CPU to create the extra images. More CPU usage = more energy cost

I think you may be misunderstanding this a bit. The latest decision here was to not create two images for each sub-size, only make WebPs. This would not increase the server resources needed after uploading an image, and will reduce storage space requirements as WebPs are smaller in size than JPEGs.

That sounds good. Thanks for the reply.

#202 @desrosj
2 years ago

In 54210:

Coding Standards: Various alignment fixes from composer format.

Follow up to [53874], [54097], [54110], [54155], [54162], [54184].

See #39210, #55443, #56288, #56092, #56408, #56467, #55881.

#203 @davidbaumwald
2 years ago

In 54226:

Media: Revert WebP generation.

Given Matt's recent post about removing WebP from core and possibly implementing the feature in a future Canonical Plugin, this change reverts changesets [54086], [54094], and [54097]. Additionally, [54210] contained a coding standards follow-up in one of the affected files that is no longer needed.

Reverts [54086], [54094], and [54097].

Props SergeyBiryukov.
See #55443.

#204 @davidbaumwald
2 years ago

  • Milestone changed from 6.1 to Future Release

With 6.1 Beta 1 releasing today and having had a brief conversation with @adamsilverstein and @flixos90 about the way forward here, this ticket is being moved to Future Release for continued discussion and work.

#205 @Alignak
2 years ago

I support the webp by default. Of course it will be more expensive on the cpu, but it's expected and acceptable to have webp than not to have it. At least as an option, it should be available.

#206 @Oceanwatcher
21 months ago

As far as I can see right now, WP create WebP sub-sizes if I upload WebP. Excellent. If I understand it correct, there is only one browser that does not display WebP, and that is Safari on OSX 10 or earlier.

I would actually settle for a solution where I can make a default image to be displayed for this browser where I will put "Please switch to a modern browser".

As far as comparing image sizes for formats, this should be done first in a desktop program and then on upload. If the uploaded sizes are not comparable, then something needs to be changed.

We need to be able to set the quality for generation of sub-sizes.

I just took one of my pictures that I developed from RAW to TIFF and converted to JPG, PNG and WebP just to see the difference. The original is a full 24 MP image, TIFF uncompressed. The results were interesting:

PNG - 109 MB
JPG - 16.9 MB
WebP - 4.8 MB

This matches with all other tests I have done. PNG is not good for photos. Graphics on the other hand is very good with PNG.

All of the tests were exported in Affinity Photo with quality settings at max. This is the right way to do it. I also did a 50% setting between JPG and WebP, and the result was the same (ranking, not size).

Then do an upload and compare the sub-sizes. If the LPG is smaller, then the quality settings are not correct.

This ticket was mentioned in Slack in #core-performance by joemcgill. View the logs.


5 months ago

Note: See TracTickets for help on using tickets.