Skip to content

Commit

Permalink
Update explainer with interrupted state info (#882)
Browse files Browse the repository at this point in the history 
* Adding interrupted state info

* Use link aliases to make the markdown more readable
  • Loading branch information
@gabrielsanbrito
gabrielsanbrito authored Oct 4, 2024
1 parent 4eaa4e0 commit e12f73f
Showing 1 changed file with 95 additions and 53 deletions.
148 changes: 95 additions & 53 deletions IframeMediaPause/iframe_media_pausing.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ There are scenarios where a website might want to just not render an iframe. For

## Proposed solution: media-playback-while-not-visible Permission Policy

We propose creating a new "media-playback-while-not-visible" [Permission Policy](https://www.w3.org/TR/permissions-policy/) that would pause any media being played by iframes which are not currently rendered. For example, this would apply whenever the iframe’s `"display"` CSS property is set to `"none"` or when the the `"visibility"` property is set to `"hidden"` or `"collapse"`.
We propose creating a new "media-playback-while-not-visible" [permission policy] that would pause any media being played by iframes which are not currently rendered. For example, this would apply whenever the iframe’s `"display"` CSS property is set to `"none"` or when the the `"visibility"` property is set to `"hidden"` or `"collapse"`.

This policy will have a default value of '*', meaning that all of the nested iframes are allowed to play media when not rendered. The example below show how this permission policy could be used to prevent all the nested iframes from playing media. By doing it this way, even other iframes embedded by "foo.media.com" shouldn’t be allowed to play media if not rendered.

Expand Down Expand Up @@ -58,21 +58,21 @@ Permissions-Policy: media-playback-while-not-visible=(self)

In this case, `example.com` serves a document that embeds an iframe with a document from `https://foo.media.com`. Since the HTTP header only allows documents from `https://example.com` to inherit `media-playback-while-not-visible`, the iframe will not be able to use the feature.

In the past, the ["execution-while-not-rendered" and "execution-while-out-of-viewport"](https://wicg.github.io/page-lifecycle/#feature-policies) permission policies have been proposed as additions to the Page Lifecycle draft specification. However, these policies freeze all iframe JavaScript execution when not rendered, which is not desirable for the featured use case. Moreover, this proposal has not been adopted or standardized.
In the past, the `"execution-while-not-rendered"` and `"execution-while-out-of-viewport"` permission policies have been proposed as additions to the [Page Lifecycle API] draft specification. However, these policies freeze all iframe JavaScript execution when not rendered, which is not desirable for the featured use case. Moreover, this proposal has not been adopted or standardized.

## Interoperability with other Web API's

Given that there exists many ways for a website to render audio in the broader web platform, this proposal has points of contact with many API's. To be more specific, there are two scenarios where this interaction might happen. Let's consider an iframe, which is not allowed to play `media-playback-while-not-visible`:
- Scenario 1: When the iframe is not rendered and it attempts to play audio; and
- Callers should treat this scenario as if they weren't allowed to start media playback. Like when the [`autoplay` permission policy](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Permissions-Policy/autoplay) is set to `'none'` for an iframe.
- Callers should treat this scenario as if they weren't allowed to start media playback. Like when the [`autoplay` permission policy] is set to `'none'` for an iframe.
- Scenario 2: When the iframe is already playing audio and stops being rendered during media playback.
- Callers should treat this scenario as if the user had paused media playback.

The following subsections covers how this proposal could interact with Web APIs that render audio.

### HTMLMediaElements

HTMLMediaElement media playback is started and paused, respectively, with the [`play`](https://html.spec.whatwg.org/multipage/media.html#dom-media-play) and [`pause`](https://html.spec.whatwg.org/multipage/media.html#dom-media-pause) methods. For scenario 1, the media element shouldn't be [allowed to play](https://html.spec.whatwg.org/multipage/media.html#allowed-to-play) and `play` should return a promise rejected with `"NotAllowedError"`. In this case, the website could easily handle this like shown below.
HTMLMediaElement media playback is started and paused, respectively, with the [`play()`] and [`pause()`] methods. For scenario 1, the media element shouldn't be [allowed to play] and `play()` should return a promise rejected with `"NotAllowedError"`. In this case, the website could easily handle this like shown below.

```js
const videoElem = document.querySelector("video");
Expand Down Expand Up @@ -109,9 +109,11 @@ videoElem.addEventListener("pause", (event) => {

### Web Audio API

_This Web Audio API integration is dependent on the successful specification and implementation of the new AudioContextState `"interrupted"` proposed in this [explainer][`"interrupted"`]._

The Web Audio API renders audio through an [AudioContext](https://webaudio.github.io/web-audio-api/#AudioContext) object. We propose that the `AudioContext` shouldn't be [allowed to start](https://webaudio.github.io/web-audio-api/#allowed-to-start) whenever it is not rendered and disallowed by the `media-playback-while-not-visible` policy.

For scenario 1, if the iframe is not rendered, any `AudioContext` will not be [allowed to start](https://webaudio.github.io/web-audio-api/#allowed-to-start). Therefore, attempting to [create](https://webaudio.github.io/web-audio-api/#dom-audiocontext-audiocontext) a new `AudioContext` or start playback by calling [`resume()`](https://webaudio.github.io/web-audio-api/#dom-audiocontext-resume) shouldn't output any audio and put the `AudioContext` into a [`"suspended"`](https://webaudio.github.io/web-audio-api/#dom-audiocontextstate-suspended) state. It would be recommended for the iframe to wait for a new user interaction event before calling [`resume()`](https://webaudio.github.io/web-audio-api/#dom-audiocontext-resume) - e.g., `click`.
For scenario 1, if the iframe is not rendered, any `AudioContext` will not be [allowed to start]. Therefore, [creating][AudioContext constructor] a new `AudioContext` should initially put it into the [`"suspended"`] state. Consequently, attempting to start playback by calling [`resume()`] shouldn't output any audio and move the `AudioContext` to the [`"interrupted"`] state. In this case, the webpage can then listen to [`statechange`] events to determine when the interruption is over.

```js
// AudioContext being created in a not rendered iframe, where
Expand All @@ -120,62 +122,72 @@ let audioCtx = new AudioContext();
let oscillator = audioCtx.createOscillator();
oscillator.connect(audioCtx.destination);

const resume_button = document.querySelector("resume_button");
resume_button.addEventListener('click', () => {
if (audioCtx.state === "suspended") {
audioCtx.resume().then(() => {
console.log("Context resumed");
});
}
})
audioCtx.onStateChange = () => {
console.log(audioCtx.state);
}

oscillator.start(0);
// should print 'suspended'
console.log(audioCtx.state)
oscillator.start(0);
// `audioCtx.onStateChange` should print 'interrupted'
```

Similarly, for scenario 2, when the iframe becomes not rendered during audio playback, the user agent should run the [`suspend()`](https://webaudio.github.io/web-audio-api/#dom-audiocontext-suspend) steps. The audio context state should change to `'suspended'` and the website can monitor this by listening to the [`statechange`](https://webaudio.github.io/web-audio-api/#eventdef-baseaudiocontext-statechange) event.
Similarly, for scenario 2, when the iframe becomes not rendered during audio playback, the user agent should interrupt the `AudioContext` by moving it to the [`"interrupted"`] state. Likewise, when the interruption is over, the UA should move the `AudioContext` back to the [`"running"`] state. Webpages can monitor these transitions by listening to the [`statechange`] event.

```js
let audioCtx = new AudioContext();
let oscillator = audioCtx.createOscillator();
oscillator.connect(audioCtx.destination);
The snippet below show this could work for scenario 2. Let's assume that the `AudioContext` in the iframe is [allowed to start]. When the web page is initialized, the `AudioContext` will be able to play audio and will transition to the [`"running"`] state. If the user clicks on the `"iframe_visibility_btn"`, the frame will get hidden and the `AudioContext` should be put in the [`"interrupted"`] state. Likewise, pressing again the button will show the iframe again and audio playback will be resumed.

const playback_control_btn = document.querySelector("playback_control_button");
playback_control_btn.textContent = "start";
playback_control_btn.addEventListener('click', () => {
if (audioCtx.state === "suspended") {
audioCtx.resume().then(() => {
console.log("Context resumed");
playback_control_btn.textContent = "suspend";
});
} else if (audioCtx.state === "running") {
audioCtx.suspend().then(() => {
console.log("Context suspended");
playback_control_btn.textContent = "resume";
});
}
})
```html
<!-- audio_context_iframe.html -->
<html>
<body>
<script>
let audioCtx = new AudioContext();
let oscillator = audioCtx.createOscillator();
oscillator.connect(audioCtx.destination);
audioCtx.addEventListener("statechange", (event) => {
if(audioCtx.state === "suspended"){
// Context has been suspended, because either suspend() has been
// called or the document is not-rendered.
console.log("Context suspended");
playback_control_btn.textContent = "resume";
}
});
audioCtx.onStateChange = () => {
console.log(audioCtx.state);
}
oscillator.start(0);
console.log(audioCtx.state)
oscillator.start(0);
</script>
</body>
</html>

<!-- Top-level document -->
<html>
<body>
<iframe id="media_frame"
src="audio_context_iframe.html"
allow="media-playback-while-not-visible 'none'">
</iframe>
<button id="iframe_visibility_btn">Hide Iframe</button>
<script>
const BTN_HIDE_DISPLAY_NONE_STR = 'Hide Iframe';
const BTN_SHOW_DISPLAY_NONE_STR = 'Show Iframe';
const iframe_visibility_btn = "iframe_visibility_btn"
let display_btn = document.getElementById(iframe_visibility_btn);
display_btn.innerHTML = BTN_HIDE_DISPLAY_NONE_STR;
display_btn.addEventListener('click', () => {
if (display_btn.innerHTML == BTN_HIDE_DISPLAY_NONE_STR){
iframe.style.setProperty('display', 'none')
display_btn.innerHTML = BTN_SHOW_DISPLAY_NONE_STR
} else {
iframe.style.setProperty('display', 'block')
display_btn.innerHTML = BTN_HIDE_DISPLAY_NONE_STR
}
});
</script>
</body>
</html>
```

### Web Speech API

The [Web Speech API](https://wicg.github.io/speech-api/) proposes a [SpeechSynthesis interface](https://wicg.github.io/speech-api/#tts-section). The latter interface allows websites to create text-to-speech output by calling [`window.speechSynthesis.speak`](https://wicg.github.io/speech-api/#dom-speechsynthesis-speak) with a [`SpeechSynthesisUtterance`](https://wicg.github.io/speech-api/#speechsynthesisutterance), which represents the text-to-be-said.
The [Web Speech API] proposes a [SpeechSynthesis interface]. The latter interface allows websites to create text-to-speech output by calling [`window.speechSynthesis.speak`] with a [`SpeechSynthesisUtterance`], which represents the text-to-be-said.

For both scenarios, the iframe should listen for utterance errors when calling `window.speechSynthesis.speak()`. For scenario 1 it should fail with a [`"not-allowed"`](https://wicg.github.io/speech-api/#dom-speechsynthesiserrorcode-not-allowed) SpeechSyntesis error; and, for scenario 2, it should fail with an [`"interrupted"`](https://wicg.github.io/speech-api/#dom-speechsynthesiserrorcode-interrupted) error.
For both scenarios, the iframe should listen for utterance errors when calling `window.speechSynthesis.speak()`. For scenario 1 it should fail with a [`"not-allowed"` SpeechSynthesisErrorCode]; and, for scenario 2, it should fail with an [`"interrupted"` SpeechSynthesisErrorCode].

```js
let utterance = new SpeechSynthesisUtterance('blabla');
Expand All @@ -199,12 +211,12 @@ This proposal does not affect autoplay behavior unless the media-playing iframe

Both `execution-while-not-rendered` and `execution-while-out-of-viewport` permission policies should take precedence over `media-playback-while-not-visible`. Therefore, in the case that we have an iframe with colliding permissions for the same origin, `media-playback-while-not-visible` should only be considered if the iframe is allowed to execute. The user agent should perform the following checks:

1. If the origin is not [allowed to use](https://html.spec.whatwg.org/multipage/iframe-embed-object.html#allowed-to-use) the [`"execution-while-not-rendered"`](https://wicg.github.io/page-lifecycle/#execution-while-not-rendered) feature, then:
1. If the iframe is not [being rendered](https://html.spec.whatwg.org/multipage/rendering.html#being-rendered), freeze execution of the iframe context and return.
2. If the origin is not [allowed to use](https://html.spec.whatwg.org/multipage/iframe-embed-object.html#allowed-to-use) the [`"execution-while-out-of-viewport"`](https://wicg.github.io/page-lifecycle/#execution-while-out-of-viewport) feature, then:
1. If the iframe does not intersect the [viewport](https://www.w3.org/TR/CSS2/visuren.html#viewport), freeze execution of the iframe context and return.
3. If the origin is not [allowed to use](https://html.spec.whatwg.org/multipage/iframe-embed-object.html#allowed-to-use) the [`"media-playback-while-not-visible"`](#proposed-solution-media-playback-while-not-visible-permission-policy) feature, then:
1. If the iframe is not [being rendered](https://html.spec.whatwg.org/multipage/rendering.html#being-rendered), pause all media playback from the iframe context and return.
1. If the origin is not [allowed to use] the [`"execution-while-not-rendered"`] feature, then:
1. If the iframe is not [being rendered], freeze execution of the iframe context and return.
2. If the origin is not [allowed to use] the [`"execution-while-out-of-viewport"`] feature, then:
1. If the iframe does not intersect the [viewport], freeze execution of the iframe context and return.
3. If the origin is not [allowed to use] the [`"media-playback-while-not-visible"`](#proposed-solution-media-playback-while-not-visible-permission-policy) feature, then:
1. If the iframe is not [being rendered], pause all media playback from the iframe context and return.

## Alternative Solutions

Expand Down Expand Up @@ -244,4 +256,34 @@ Similarly to the [HTMLMediaElement.muted](https://html.spec.whatwg.org/multipage
</html>
```

This alternative was not selected as the preferred one, because we think that pausing media playback is preferable to just muting it.
This alternative was not selected as the preferred one, because we think that pausing media playback is preferable to just muting it.

[AudioContext constructor]: https://webaudio.github.io/web-audio-api/#dom-audiocontext-audiocontext
[allowed to play]: https://html.spec.whatwg.org/multipage/media.html#allowed-to-play
[allowed to start]: https://webaudio.github.io/web-audio-api/#allowed-to-start
[allowed to use]: https://html.spec.whatwg.org/multipage/iframe-embed-object.html#allowed-to-use
[`autoplay` permission policy]: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Permissions-Policy/autoplay
[being rendered]: https://html.spec.whatwg.org/multipage/rendering.html#being-rendered
[`"execution-while-not-rendered"`]: https://wicg.github.io/page-lifecycle/#execution-while-not-rendered
[`"execution-while-out-of-viewport"`]: https://wicg.github.io/page-lifecycle/#execution-while-out-of-viewport
[`"interrupted"`]: https://github.com/MicrosoftEdge/MSEdgeExplainers/blob/main/AudioContextInterruptedState/explainer.md
[`"interrupted"` SpeechSynthesisErrorCode]: ttps://wicg.github.io/speech-api/#dom-speechsynthesiserrorcode-interrupted
[`"not-allowed"` SpeechSynthesisErrorCode]: https://wicg.github.io/speech-api/#dom-speechsynthesiserrorcode-not-allowed
[Page Lifecycle API]: https://wicg.github.io/page-lifecycle/#feature-policies
[permission policy]: https://www.w3.org/TR/permissions-policy/
[`pause()`]: https://html.spec.whatwg.org/multipage/media.html#dom-media-pause
[`play()`]: https://html.spec.whatwg.org/multipage/media.html#dom-media-play
[`SpeechSynthesisUtterance`]: https://wicg.github.io/speech-api/#speechsynthesisutterance
[SpeechSynthesis interface]: https://wicg.github.io/speech-api/#tts-section
[`statechange`]: https://webaudio.github.io/web-audio-api/#eventdef-baseaudiocontext-statechange
[`"suspended"`]: https://webaudio.github.io/web-audio-api/#dom-audiocontextstate-suspended
[`resume()`]: https://webaudio.github.io/web-audio-api/#dom-audiocontext-resume
[`"running"`]: https://webaudio.github.io/web-audio-api/#dom-audiocontextstate-running
[viewport]: https://www.w3.org/TR/CSS2/visuren.html#viewport
[Web Speech API]: https://wicg.github.io/speech-api/
[`window.speechSynthesis.speak`]: https://wicg.github.io/speech-api/#dom-speechsynthesis-speak


The [Web Speech API](https://wicg.github.io/speech-api/) proposes a [SpeechSynthesis interface](https://wicg.github.io/speech-api/#tts-section). The latter interface allows websites to create text-to-speech output by calling [`window.speechSynthesis.speak`](https://wicg.github.io/speech-api/#dom-speechsynthesis-speak) with a [`SpeechSynthesisUtterance`](https://wicg.github.io/speech-api/#speechsynthesisutterance), which represents the text-to-be-said.

For both scenarios, the iframe should listen for utterance errors when calling `window.speechSynthesis.speak()`. For scenario 1 it should fail with a [`"not-allowed" SpeechSynthesisErrorCode`](https://wicg.github.io/speech-api/#dom-speechsynthesiserrorcode-not-allowed) SpeechSyntesis error; and, for scenario 2, it should fail with an [`"interrupted" SpeechSynthesisErrorCode`](https://wicg.github.io/speech-api/#dom-speechsynthesiserrorcode-interrupted) error.

0 comments on commit e12f73f

Please sign in to comment.