Deprecate embedded audio or video as targets for Media Overlay <text> reference #2397
Description
My apologies for the verbose background below; I believe it is necessary to make an informed decision. This is not a widely known area of the spec…
This is the usual way of using Media Overlays (MO):
<html>
…
<p id="txt">...</p>
</html>
<smil>
…
<par>
<text src="...#txt">
<audio src="link_to_audio_file" clipBegin="…" clipEnd="…">
</par>
…
<smil>On Reading Systems implementing MO, if read-aloud is switched on, the content of the audio file is read while the text in the XHTML content is rendered.
However, what happens if we have, instead, the following:
<html>
…
<video id="movie">...</video>
</html>
<smil>
…
<par>
<text src="...#movie">
<audio src="link_to_audio_file" clipBegin="…" clipEnd="…">
</par>
…
<smil>The answer is… it is complicated. There are indeed two time-dependent media rendering jobs in parallel (the movie and the audio clip), and the spec should be clear about which one has the upper hand in timing control. The problem is that it isn't
- §9.2.2.4.1 in the EPUB 3.3 content spec does not talk about this case, only the case when there is no audio element in the MO file. Specifying what is and isn't allowed would require further work.
- §9.3.2 in the Reading System is a bit clearer; what it specifies is that the audio file's timing is in control: the video must be started by the RS when the audio starts, it must be stopped when the audio ends, the reader has no control over the video (only the RS has). But of course there are some other cases, see that section for details, which makes this specification much more complicated. In my view that section requires further editorial work to make it clearer.
Some tests have been created for §9.3.2 (see mol-embed or mol-embed_deactivate_playback) but, at the moment, there are no known implementations around, and we may be looking at a problem at the end of the CR phase. Also: it is not clear what the use case may be for such a structure in the EPUB context. (If the goal is to add an audio track to a video as, say, background music, there are widely available video editing programs to do that more easily these days.)
(Note that this introduction only referred to video but, in fact, it could also be an audio track within the XHTML content document. The spec used the term "embedded media" as a general term, although that also includes still images.)
Sorry for the long introduction; here is the real issue I indent to raise. I think we should consider deprecating the feature of "embedded media" (maybe with the exception of still images). It is an overly complex part of the spec (mainly the RS spec) without known use cases and implementations; we can preempt the problems we will have at the end of the CR phase by doing so.
I can see several ways we could proceed:
- declare the whole of Content spec §9.2.2.4.1 as well as Reading System §9.3.2 a SHOULD or MAY (and avoid using MUST statements in the text, much like the §9.3.3 TTS section); or
- we extract the use case whereby video or audio in a
<par>without an accompanying<audio>means playing that media under the implementation control (without user control, that is), and we declare all other cases as deprecated; or - declare the whole feature as deprecated: the content document MO file disallows a reference to a video/audio from the
<text>element (either directly or indirectly with a, say, video in the middle of a text flow with read loud) and the RS specification just says that handling those cases is implementation dependent.
I do not think we would create problems with existing EPUB deployment; indeed, I do not believe there is any out there... And it would make our spec cleaner.