24 Sep 2013

Gecko and vtt.js

After three weeks of coding the initial patches and nearly two months of review, Nightly finally plays basic WebVTT captions again! If you'd like to see a demo you can head over to my previous blog post where I put together one.

Gecko Integration Setup

The set up of our integration of vtt.js into Gecko is fairly simple.

First, we have a WebVTTParserWrapper that allows us to wrap a vtt.js parser. It implements the nsIWebVTTParserWrapper IDL which allows us to work with the WebVTTParserWrapper (that was implemented in JS) in C++.

Secondly, we have a WebVTTListener class which primarily does two things:

  • Listens to the data stream of an HTMLTrackElement and passes the data to the WebVTTParserWrapper to be parsed.
  • Receives parsed VTTCues and VTTRegions from vtt.js and adds them to the HTMLTrackElement's Track.
Since the WebVTTParserWrapper is written in JS we need to define an IDL that the WebVTTListener class can implement in order to expose its callbacks to JavaScript. The IDL that it implements is the nsIWebVTTListener IDL which defines two callbacks that vtt.js needs—onCue(cue) and onRegion(region).

And that's pretty much it for our current integration of vtt.js into Gecko. Since vtt.js implements the parsing section of WebVTT and the DOM tree construction of rendered subtitles, there is very little we need to add. To get the DOM tree represenation of a cue we can just call convertCueToDOMTree() on vtt.js, get the div containing the DOM tree, and add it to the video's caption overlay.

Moving Forward

The next major step for WebVTT in Gecko is to get what's called the WebVTT processing model implemented. This is the section of the spec that positions the cue on the video with respect to the cue-settings that have been applied to the cue.

Work on that has already begun, and after some major hurdles having to do with the spec being very buggy, vtt.js now applies horizontal or vertical positioning depending on the text orientation of the cue. We've yet to integrate that into Gecko though. We still need to finish the part of the processing model that accounts for overlap avoidance, i.e. positioning cues so that they don't overlap eachother. After that we can start preparing a patch to get that functionality into Gecko.

- Tagged in mozilla, seneca college, open-source, and cdot.

comments powered by Disqus

Wow coding Copyleft Richard Eyre 2013

Code for this site can be found on GitHub.