WordPress.org

Make WordPress Core

#27389 closed task (blessed) (fixed)

MCE Views for Audio/Video

Reported by: wonderboymusic Owned by: azaozz
Milestone: 3.9 Priority: normal
Severity: normal Version: 3.9
Component: Media Keywords: has-patch commit dev-reviewed
Focuses: javascript Cc:

Description

Turns out, you can render players in TinyMCE. This was not easy.

Attachments (27)

good-times.png (34.5 KB) - added by wonderboymusic 15 months ago.
27389.diff (17.5 KB) - added by wonderboymusic 15 months ago.
27389-02.patch (19.2 KB) - added by gcorne 15 months ago.
27389.2.diff (23.2 KB) - added by wonderboymusic 15 months ago.
27389.3.diff (24.2 KB) - added by wonderboymusic 15 months ago.
27389.4.diff (24.3 KB) - added by wonderboymusic 15 months ago.
27389.5.diff (29.7 KB) - added by wonderboymusic 15 months ago.
Screen Shot 2014-03-14 at 2.06.32 AM.png (125.6 KB) - added by wonderboymusic 15 months ago.
27389.6.diff (11.3 KB) - added by wonderboymusic 15 months ago.
27389.7.diff (10.5 KB) - added by wonderboymusic 15 months ago.
27389.8.diff (13.0 KB) - added by wonderboymusic 14 months ago.
reinit-views.patch (1.9 KB) - added by gcorne 14 months ago.
27389-why-god-why.diff (5.1 KB) - added by wonderboymusic 14 months ago.
27389-why-god-why.2.diff (4.6 KB) - added by wonderboymusic 14 months ago.
27389-wpviews-undo-revamp.patch (2.2 KB) - added by gcorne 14 months ago.
27389-wpviews-undo-revamp-02.patch (4.9 KB) - added by gcorne 14 months ago.
27389-timeout.diff (1.0 KB) - added by wonderboymusic 14 months ago.
27389.combined.diff (8.0 KB) - added by wonderboymusic 14 months ago.
empty-editor.png (18.6 KB) - added by wonderboymusic 14 months ago.
one-audio.png (17.7 KB) - added by wonderboymusic 14 months ago.
27389.combined.2.diff (5.0 KB) - added by wonderboymusic 14 months ago.
27389.combined.3.diff (5.1 KB) - added by wonderboymusic 14 months ago.
27389.combined.4.diff (5.1 KB) - added by wonderboymusic 14 months ago.
27389-unbind.patch (3.5 KB) - added by azaozz 14 months ago.
27389-unbind-02.patch (4.4 KB) - added by gcorne 14 months ago.
27389.9.diff (5.6 KB) - added by wonderboymusic 14 months ago.
27389.10.diff (6.0 KB) - added by azaozz 14 months ago.

Download all attachments as: .zip

Change History (64)

@wonderboymusic15 months ago

comment:1 @ircbot15 months ago

This ticket was mentioned in IRC in #wordpress-dev by wonderboymusic. View the logs.

@wonderboymusic15 months ago

@gcorne15 months ago

@wonderboymusic15 months ago

comment:2 @wonderboymusic15 months ago

This bent my brain in half: 27389.2.diff

  • Pauses all players when toggling between editor modes
  • Pauses all players when popping open the media modal
  • When "Update" is pressed: the MCE view's data-wpview-text attr is updated, the views are refreshed, and the specific type of media's render method is called to remove the player and re-add it
  • wp.media.audio|video's shortcode method is now called update, which is more semantically representative.
  • Triggers a ready event when a wp.mce.View is constructed
  • Bind ready handlers for wp.mce.audio|video.View for when the page initially loads so we have a reference to the view's root element
  • Removed the event attributes that were present in a previous patch's Underscore templates
Last edited 15 months ago by wonderboymusic (previous) (diff)

@wonderboymusic15 months ago

comment:3 @wonderboymusic15 months ago

27389.3.diff has many improvements:

  • Previously, if you had the same shortcode in your post twice, it would only get rendered once. Fixed that.
  • When rendering a wp.mce.View, the current node context is passed to a ready event, so when you do have multiple shortcodes that are equal, you know which one you are dealing with.
  • For videos in the editor, force preload="none" - will let the video play immediately. Any of the other preload values cause the spinner to appear for many seconds. This defies logic.

@wonderboymusic15 months ago

comment:4 @wonderboymusic15 months ago

In 27528:

Add MCE Views for audio and video. Please clear your browser cache so that you get the latest TinyMCE stylesheet.

  • Move TinyMCE shortcode handling from wpgallery plugin to mce-view.js
  • Force preload="none" when rendering media in the editor to ensure fast loading (I realize this sounds illogical)
  • Move audio and video tag builder logic in media-template.php into PHP funcs that can be reused by any code passing data.model to an Underscore template
  • Pause all players when moving between editor tabs and when moving from the editor to editing in the media modal.
  • Rename wp.media.audio|video.shortcode() to the more appropriate wp.media.audio|video.update() that now returns a wp.shortcode object instead of a string.
  • Add necessary MediaElement css files to $mce_css
  • In wp.mce.View.render(), support multiple instances of the same shortcode
  • When rendering wp.mce.Views, fire a ready event that passes the current MCE View root element as context

See #27389.

comment:5 @wonderboymusic15 months ago

In 27529:

Version bump for TinyMCE stylesheet.

See #27389, [27528].

comment:6 @wonderboymusic15 months ago

In 27530:

Revert [27528] until Flash in Firefox behaves :(

See #27389.

@wonderboymusic15 months ago

comment:7 @wonderboymusic15 months ago

I tested this on every browser with every supported audio/video type: 27389.5.diff

tl;dr MediaElement's Plugin Bridge (Flash/Silverlight) and TinyMCE do not intermingle well.

Subsequently, I added an isCompatible method to whitelist types that MEjs should parse, otherwise replace with a placeholder and bail.

comment:8 @wonderboymusic15 months ago

posted a screenshot that shows intermingling of supported/unsupported videos in TinyMCE. What sucks: almost every type plays just fine in the media modal. I think there is something in the global space that doesn't translate from the MEjs bridge to TinyMCE - the errors propagate from the JS VM though, not from the app #yay

comment:9 @wonderboymusic15 months ago

In 27533:

When rendering Underscore templates for Audio/Video details, use 2 new PHP functions to render the markup for audio and video tags: wp_underscore_audio_template() and wp_underscore_video_template().

Future JS templates can follow the convention of expecting data to be passed to them containing a model property.

See #27389.

comment:10 @wonderboymusic15 months ago

In 27534:

Load MediaElement's CSS when TinyMCE is loaded via $mce_css in editor_settings(). Add some baseline styles in wp-content.css for audio, video, and embed tags to ensure their max-width is 100%, regardless of whether MEjs is implemented in TinyMCE.

See #27389.

comment:11 @wonderboymusic15 months ago

In 27535:

Make media.view.VideoDetails.prepareSrc a static class method instead of an instance method. Properly pairs it with its incrementing instances property.

See #27389.

comment:12 @wonderboymusic15 months ago

In 27536:

In media.model.PostMedia, for these 2 scenarios:

  • src is set, and 'Add a Source' results in the same file (or a file with the same extension) being added
  • src is set, and 'Replace Audio|Video' is used, which will add a model property named by the attachment's extension

... call model.unset( 'src' ).

See #27389.

comment:13 @wonderboymusic15 months ago

In 27537:

Rather than extending wp.media.mixin, only borrow coerce when necessary in wp.media.audio|video|collection. wp.media.mixin will make sense to be mixed in for classes that expect to interact with the player. More universal methods can be added and inherited by all those who extend their prototype with it.

Assuming someone had implemented players in the editor, pause all players when switching between editor tabs. A method, pauseAllPlayers, has been added to wp.media.mixin.

See #27389.

comment:14 @wonderboymusic15 months ago

In 27538:

Move removePlayer() and unsetPlayer() to wp.media.mixin so that other classes can take advantage of them.

  • removePlayer() is an alternative to MediaElement's player removal method that does not re-add the audio|video element back to the DOM.
  • unsetPlayer() will check for an existing player property on the passed instance and pause all players before calling unsetPlayer()

See #27389.

comment:15 @wonderboymusic15 months ago

In 27539:

The canPlayType property for audio and video in JS is so bad that the official valid responses are "probably" and "maybe". There are many cases where we might want to know if an audio|video tag is gonna blow up in our face before even attempting to make a MediaElementPlayer instance out of it.

The best (and most cautious) way to tackle this is to whitelist types by browser. Imagine that one implemented MEjs in TinyMCE's rich editor mode, this would be very helpful.

Add isCompatible( $media ) to wp.media.mixin. Future features will use this.

See #27389.

@wonderboymusic15 months ago

comment:16 @ircbot15 months ago

This ticket was mentioned in IRC in #wordpress-dev by wonderboymusic. View the logs.

comment:17 @wonderboymusic15 months ago

In 27542:

In wp.media.mixin, add ua and a compat map to separate and clarify browser detection and support. Leverage Underscore list iterators in isCompatible().

See #27389.

@wonderboymusic15 months ago

comment:18 @wonderboymusic15 months ago

27389.7.diff is a refresh against all of the code churn. Does not take [27544] into account yet.

@wonderboymusic14 months ago

comment:19 @wonderboymusic14 months ago

In 27615:

Add MCE views for audio and video shortcodes. When the shortcode does not contain a source that supports native playback, just show the filename.

  • Remove the audio/video shortcode parsing from the wpgallery plugin.
  • Make mce-view a dependency of media-audiovideo
  • Introduce wp.mce.video, wp.mce.audio, wp.mce.media, and wp.mce.media.View
  • Rename wp.media.audio|video.shortcode() to wp.media.audio|video.update() since it is called on Update and returns a wp.shortcode object now.
  • In wp.mce.View.render(), fire a ready event when the placeholder is being parsed and pass the current node to the event handler.

See #27389, #27437.

comment:20 @wonderboymusic14 months ago

  • Resolution set to fixed
  • Status changed from new to closed

New tickets for specific bugs.

comment:21 @wonderboymusic14 months ago

In 27658:

When an HTML5 fallback button is pressed in the Audio or Video Details state, filter the library by that specific mime-type when the Add Audio|Video Source is activated.

See #27389.

@gcorne14 months ago

comment:22 @ircbot14 months ago

This ticket was mentioned in IRC in #wordpress-dev by wonderboymusic. View the logs.

comment:23 @ircbot14 months ago

This ticket was mentioned in IRC in #wordpress-dev by wonderboymusic. View the logs.

comment:24 @wonderboymusic14 months ago

  • Resolution fixed deleted
  • Status changed from closed to reopened

There are at least 2 patches that will need to be applied - one is the timeout on initial page load, the other is related to cut-copy-paste-undo and memory leaks

@wonderboymusic14 months ago

comment:25 @wonderboymusic14 months ago

27389.combined.diff combines the patches. As far as memory management goes, TinyMCE itself leaks memory. My initial profiling (empty-editor.png) was done against an empty Visual editor, each additional profile was done by going from Visual-Text-Visual then taking another snapshot. The second set of snapshots (one-audio.png) was done with one audio player in the editor. I did some other tests with 21 players in the editor and the ratios were the same.

My code in the patch ensures that MediaElement's cleanup routine is run on every player when TinyMCE rebuilds itself, even when multiple players are bound to the same view - this happens when duplicate shortcodes exists. I tested this with 20 players, 7 being the exact same.

On page load, the players are initialized after a 500ms timeout to ensure that the stylesheets are loaded. Subsequent builds happen after a 10ms refresh. The broken looking players in FF were due to a race condition with the editor stylesheet.

The patch also contains @gcorne's view refresh fixes in wp.mce.View

comment:26 @gcorne14 months ago

A change to the UndoManager was committed upstream that adds the additional context to the BeforeAddUndo hook that will allow us to prevent undo steps from being added when the body of a wpview changes and some sort of action triggers TinyMCE to attempt to add an undo level.

comment:27 follow-up: @wonderboymusic14 months ago

27389.combined.3.diff cleans up all player instances in a view when the view refreshes and the first new player is being parsed.

Since we are blowing away the inner DOM for MCE views when transition away from Visual or Undo-ing, this is necessary to clean up MEjs cruft from previously created players. They all have to be done at once to prevent extras from leaking - say you go from Visual tab to Text, delete a shortcode, and then tab back to Visual, we don't want orphaned and unbound player instances.

This is a joy to debug. Patch also removes the TinyMCE changes which are forthcoming from upstream.

comment:28 in reply to: ↑ 27 @azaozz14 months ago

Replying to wonderboymusic:

There is no need to use jQuery inside the editor (TinyMCE includes Sizzle too). This bit:

_.each( tinymce.editors, function( editor ) { 
  var doc; 
  if ( editor.plugins.wpview ) { 
    doc = editor.getDoc(); 
    size += $( doc ).find( '[data-wpview-text="' + self.encodedText + '"]' ).length; 
  } 
} );

would be better as:

_.each( tinymce.editors, function( editor ) { 
  if ( editor.plugins.wpview ) { 
    size += editor.dom.select( '[data-wpview-text="' + self.encodedText + '"]' ).length; 
  } 
} );

Also self.encodedText can be quite long, might slow down or even break the selector? If you need to count only specific views, perhaps select all views first with views = editor.dom.select( '[data-wpview-text]' ); then loop over them and count only the specific views by matching the attribute value with editor.dom.getAttrib( viewNode, 'data-wpview-text' ) === self.encodedText.

comment:29 @azaozz14 months ago

Thinking about this more: would it be better to have a "global" unbind event triggered in wpview? Seems the three events that force DOM rebuilding in the editor are undo, redo, and loadContent. We can easily fire an view.unbind() on these.

comment:30 @wonderboymusic14 months ago

An unbind event would negate a lot of the code in my patch, which is a good thing. All of the memory cleanup could happen in the same place, before ready fires.

Once that is exposed, I can refactor my patch.

comment:31 @ircbot14 months ago

This ticket was mentioned in IRC in #wordpress-dev by wonderboymusic. View the logs.

@azaozz14 months ago

@gcorne14 months ago

comment:32 @azaozz14 months ago

27389-unbind-02.patch looks good, @wonderboymusic could you confirm it works as expected.

@wonderboymusic14 months ago

comment:33 @wonderboymusic14 months ago

27389.9.diff adds a few of the things from my patch that are still necessary.

  1. Need to set a longer timeout on initial page load to avoid the race condition with the editor stylesheet setting content width
  2. Need to push players onto a stack, a single view can render multiple instances if the shortcode is duplicated. Thus, the unbind routine needs to iterate and remove and then reset.

Everything looks good, and this approach is much cleaner. John Resig envisioned code-heroes/cowboys like you two when he wrote Secrets of a JavaScript Ninja.

comment:34 @ircbot14 months ago

This ticket was mentioned in IRC in #wordpress-dev by wonderboymusic. View the logs.

@azaozz14 months ago

comment:35 @azaozz14 months ago

  • Keywords commit added

27389.9.diff looks good. 27389.10.diff just adds couple more inline comments.

comment:36 @nacin14 months ago

  • Keywords dev-reviewed added

If azaozz, gcorne, and wonderboymusic are fine with this, me too.

comment:37 @azaozz14 months ago

  • Owner set to azaozz
  • Resolution set to fixed
  • Status changed from reopened to closed

In 28084:

TinyMCE wpViews:

  • Prevent undo steps from being added when the body of a wpview changes.
  • Add unbind() to handle cleanup on DOM rebuilding in TinyMCE.
  • Ensure that MediaElement's cleanup routine is run on every player in all instances of the editor.
  • Initialize the players after some delay to ensure CSS is loaded.

Props gcorne and wonderboymusic, fixes #27389

Note: See TracTickets for help on using tickets.