Feat: add an option to play without audio or video if audio or video failed #1624

Florent-Bouisset · 2025-01-06T09:28:11Z

Currently, the player throws an error if either all audio or all video tracks are not playable. This PR introduces two new options, onAudioTrackNotPlayable and onVideoTrackNotPlayable, to the loadVideo method. These options allow the player to continue playback even when all audio or video tracks are not playable. In such cases, the content will play without audio or video, respectively, instead of failing.

API

loadVideo(options) options object has two more keys:

[optional]onAudioTrackNotPlayable : "continue" or "error". Default if not set: "continue"
[optional]onVideoTrackNotPlayable : "continue" or "error". Default if not set: "continue"

Usage

  player.loadVideo({
    transport: "dash",
    url: "https://dash.akamaized.net/dash264/TestCasesDolby/1/Living_Room_1080p_20_96k_25fps.mpd",
    autoPlay: true,
    onAudioTrackNotPlayable: "continue",
    onVideoTrackNotPlayable: "error",
  });

Example

https://codesandbox.io/p/sandbox/amazing-pare-kdf65f

Florent-Bouisset · 2025-01-06T16:12:53Z

relates to #1602

peaBerberian · 2025-01-06T17:12:20Z

onAudioTrackNotPlayable and onVideoTrackNotPlayable

Shouldn't it be "tracks" here?

Default if not set: "continue"

Doesn't it break the API this way? Its complex but it changes the default behavior on which some applications may rely? Wouldn't it be safer to default it to "error"?

doc/api/Loading_a_Content.md

Florent-Bouisset · 2025-01-07T13:14:29Z

onAudioTrackNotPlayable

Yes I think I should default it to "error" instead.

Florent-Bouisset · 2025-01-09T10:22:34Z

onAudioTrackNotPlayable and onVideoTrackNotPlayable

Shouldn't it be "tracks" here?

Default if not set: "continue"

Doesn't it break the API this way? Its complex but it changes the default behavior on which some applications may rely? Wouldn't it be safer to default it to "error"?

I have updated it.

src/core/stream/period/period_stream.ts

src/main_thread/tracks_store/tracks_store.ts

peaBerberian · 2025-01-09T11:20:22Z

As this adds a similar logic in the TrackStore + Manifest + Init, and as the manifest is more or less intended to just regroup metadata, I'm wondering if it would not be easier to follow if that option was mainly handled by a part of the Init (and TrackStore I guess for its own thing) : Like the Manifest would parse the content regardless, and the Init would investigate the issues seen while parsing and decide whether to error or warn and continue based on its options.

What do you think?

doc/api/Player_Events.md

src/main_thread/api/public_api.ts

src/manifest/classes/period.ts

src/main_thread/tracks_store/tracks_store.ts

github-actions · 2025-02-18T15:57:39Z

Automated performance checks have been performed on commit 1b34a6119995d99e0732abd3996f0fce7e07590d with the base branch dev.

Tests results

✅ Tests have passed.

Performance tests 1st run output

No significative change in performance for tests:

Name	Mean	Median
loading	21.10ms -> 21.14ms (-0.037ms, z: 0.38653)	29.25ms -> 29.25ms
seeking	23.68ms -> 20.27ms (3.405ms, z: 1.50470)	11.70ms -> 11.55ms
audio-track-reload	28.00ms -> 28.13ms (-0.129ms, z: 0.56610)	40.80ms -> 40.95ms
cold loading multithread	51.53ms -> 50.05ms (1.481ms, z: 7.90735)	74.70ms -> 73.50ms
seeking multithread	30.21ms -> 26.14ms (4.070ms, z: 0.96041)	10.50ms -> 10.50ms
audio-track-reload multithread	27.91ms -> 27.77ms (0.140ms, z: 1.45816)	41.10ms -> 40.80ms
hot loading multithread	18.11ms -> 18.14ms (-0.026ms, z: 0.84623)	26.40ms -> 26.40ms

If you want to skip performance checks for latter commits, add the skip-performance-checks label to this Pull Request.

github-actions · 2025-02-19T17:12:10Z

Automated performance checks have been performed on commit d11562dbaee84888c15cb1f7b7ed4e4e3b8ba66e with the base branch dev.

Tests results

✅ Tests have passed.

Performance tests 1st run output

No significative change in performance for tests:

Name	Mean	Median
loading	19.27ms -> 19.31ms (-0.049ms, z: 0.03714)	26.55ms -> 26.40ms
seeking	15.23ms -> 10.58ms (4.641ms, z: 0.14786)	11.10ms -> 11.10ms
audio-track-reload	26.13ms -> 26.17ms (-0.042ms, z: 0.58250)	37.95ms -> 37.95ms
cold loading multithread	47.53ms -> 46.67ms (0.864ms, z: 13.13265)	69.45ms -> 68.10ms
seeking multithread	20.57ms -> 15.20ms (5.369ms, z: 1.28316)	10.35ms -> 10.35ms
audio-track-reload multithread	25.93ms -> 25.87ms (0.062ms, z: 1.08056)	38.10ms -> 37.95ms
hot loading multithread	16.27ms -> 15.99ms (0.282ms, z: 4.61710)	23.70ms -> 23.25ms

If you want to skip performance checks for latter commits, add the skip-performance-checks label to this Pull Request.

… only

github-actions · 2025-02-27T11:22:10Z

Automated performance checks have been performed on commit b1f4c12bc8cb49a7598cfcfc5473edbac5ee5817 with the base branch dev.

Tests results

✅ Tests have passed.

Performance tests 1st run output

No significative change in performance for tests:

Name	Mean	Median
loading	20.49ms -> 20.58ms (-0.097ms, z: 1.01770)	28.35ms -> 28.20ms
seeking	21.60ms -> 24.85ms (-3.252ms, z: 1.61675)	11.55ms -> 11.55ms
audio-track-reload	27.73ms -> 27.40ms (0.329ms, z: 2.29041)	40.20ms -> 39.90ms
cold loading multithread	49.84ms -> 48.97ms (0.875ms, z: 11.40648)	72.90ms -> 71.40ms
seeking multithread	36.17ms -> 28.17ms (8.001ms, z: 0.23911)	10.50ms -> 10.50ms
audio-track-reload multithread	27.54ms -> 27.33ms (0.211ms, z: 3.37909)	40.35ms -> 40.05ms
hot loading multithread	17.51ms -> 17.44ms (0.067ms, z: 1.93997)	25.40ms -> 25.20ms

If you want to skip performance checks for latter commits, add the skip-performance-checks label to this Pull Request.

peaBerberian · 2025-02-27T19:38:01Z

doc/api/Loading_a_Content.md

+  tracks are playable.
+
+- If neither the audio nor video tracks are playable, an error will be thrown regardless
+of this setting.


Prettier did not seem to include the end of this sentence in the same list element (it misses indentation) - I don't know how it will render once transformed into HTML

Also, maybe we should tell here the error code in question, so that application developers can know how to handle those without having to reproduce it.

Because this is inside a
I think the indentation does not matter (prettier does not want to add a tab)

I have added the error code

peaBerberian · 2025-02-27T19:38:35Z

doc/api/Loading_a_Content.md

+  tracks are playable.
+
+- If neither the audio nor video tracks are playable, an error will be thrown regardless
+of this setting.


peaBerberian · 2025-02-27T19:46:45Z

doc/api/Player_Events.md

+
+  - `id`: (`"string"`): The ID of the period.
+
+  - `start`: (`"number"`): The start time of the period.


First there's a line break between list elements then there's not :O, we must decide!

peaBerberian · 2025-02-27T19:48:32Z

doc/api/Player_Events.md

+
+  - `start`: (`"number"`): The start time of the period.
+  - `end`: (`"number" | undefined`): The end time of the period. (may be `undefined` if no
+    end time is set).


More generally, end is also undefined for live contents where there's no end for now, which is a frequent case.

peaBerberian · 2025-02-27T19:56:41Z

src/manifest/classes/period.ts

-          { tracks: undefined },
-        );
-      }
+      hasSupportedMedia[ttype] = hasSupportedAdaptations;


Is hasSupportedMedia used at all here?

peaBerberian · 2025-02-27T19:59:05Z

src/transports/dash/manifest_parser.ts

@@ -153,7 +153,7 @@ export default function generateManifestParser(
          return Promise.reject(cancelSignal.cancellationError);
        }
        const warnings: IPlayerError[] = [];


From what I understand, warnings isn't used anywhere anymore? We could remove it from the transports API

src/main_thread/tracks_store/tracks_store.ts

peaBerberian · 2025-02-27T20:03:48Z

src/main_thread/tracks_store/tracks_store.ts

+      // there is no track of this type to select.
+      log.debug(
+        `TS: The period does not have adaptation for ${ttype} there is no track to choose`,
+      );


I'm wondering if having this debug log for each Period without some text AdaptationSet at each update wouldn't be too much

peaBerberian · 2025-02-27T20:07:26Z

src/main_thread/tracks_store/tracks_store.ts

+    if (
+      adaptationsForType.every((adapt) => adapt.supportStatus.hasSupportedCodec === false)
+    ) {
+      const firstPlayableAdaptation = findFirstPlayableAdaptation(period, ttype);


I don't get it, how could firstPlayableAdaptation be set to anything other than undefined here, didn't we just check that no adaptation of that type is supported in that Period?

What possibility am I missing?

peaBerberian · 2025-02-27T20:09:00Z

src/main_thread/tracks_store/tracks_store.ts

+        { tracks: undefined },
+      );
+
+      if (


It seems that in this particular case, we do not have the noPlayableTrack event.

Is that wanted? From reading the API documentation, I would think that this case would lead to that event.

Though after doing more thinking, it seems that sending the noPlayableTrack event here might not work because we might not have advertised the Period yet through a newAvailablePeriods event, no?

peaBerberian · 2025-02-27T20:14:58Z

src/main_thread/tracks_store/tracks_store.ts

+        textTypeInfo.storedSettings = storedSettings;
+        textTypeInfo.dispatcher?.updateTrack(storedSettings);
+      }
+      return;


By returning early, we don't send the trackUpdate event

peaBerberian · 2025-02-27T20:18:22Z

src/main_thread/tracks_store/tracks_store.ts

+      if (this._isDisposed) {
+        return; // Someone disposed the `TracksStore` on the previous side-effect
+      }
+      this.trigger("error", noRepErr);


Note: I don't remember if we exchanged about this, but this means that we throw even if the application e.g. disable the video track on a noPlayableTrack event for video content - where it could theoretically play under the wanted settings.

I'm OK with this behavior, it's just to check whether we're on the same page on this.

peaBerberian · 2025-02-27T20:25:08Z

src/main_thread/tracks_store/tracks_store.ts

+    if (isNullOrUndefined(typeInfo) || typeInfo.storedSettings !== storedSettings) {
+      return;
+    }
+    typeInfo.dispatcher?.updateTrack(storedSettings);


Aren't we in the risk of setting null for the video track when there's already no audio track and vice-versa here? Shouldn't we also check for the NO_AUDIO_VIDEO_TRACKS error?

peaBerberian · 2025-02-27T20:26:32Z

src/main_thread/tracks_store/tracks_store.ts

-            };
+      const audioAdaptation = getSupportedAdaptations(period, "audio")[0] as
+        | IAdaptationMetadata
+        | undefined;


The as is just to indicate that the value may be undefiined?
There may be better ways no? The problem I see with as is possible bugs if we make code changes while trusting TypeScript a little too much to save our back

Yes it's just to indicate it could be undefined as typescript does not know that accessing through indexing can be undefined. I found a more clever way to handle it.
Note that using Array.at() correctly infer the type but this is ES2022 so I didn't use it there.

I'm wondering if TypeScript automatically translate at calls

peaBerberian · 2025-02-27T20:30:32Z

src/main_thread/tracks_store/tracks_store.ts

+      if (textTypeInfo !== undefined) {
+        const storedSettings = null;
+        textTypeInfo.storedSettings = storedSettings;
+        textTypeInfo.dispatcher?.updateTrack(storedSettings);


No need to do this if it was already null before no?

peaBerberian · 2025-02-27T20:37:19Z

src/main_thread/tracks_store/tracks_store.ts

+        | undefined;
+      if (audioAdaptation === undefined) {
+        trackStorePeriod.audio.storedSettings = null;
+        this.onNoPlayableRepresentation(period, "audio");


/!\ I'm under the impression that this method may lead to noPlayableTrack events which communicate about Periods not yet advertised through newAvailablePeriods events, so the application is generally not supposed to know about them in the order of events we had until now :/

Florent-Bouisset added the API Relative to the RxPlayer's API label Jan 6, 2025

Florent-Bouisset force-pushed the feat/play-without-audio-or-video branch from 724b4a3 to 5a073f9 Compare January 6, 2025 09:30

peaBerberian reviewed Jan 6, 2025

View reviewed changes

doc/api/Loading_a_Content.md Outdated Show resolved Hide resolved

Florent-Bouisset force-pushed the feat/play-without-audio-or-video branch from ef25c83 to 4b6acd7 Compare January 9, 2025 10:39

peaBerberian reviewed Jan 9, 2025

View reviewed changes

src/core/stream/period/period_stream.ts Show resolved Hide resolved

peaBerberian reviewed Jan 9, 2025

View reviewed changes

src/main_thread/tracks_store/tracks_store.ts Outdated Show resolved Hide resolved

peaBerberian reviewed Jan 9, 2025

View reviewed changes

src/main_thread/tracks_store/tracks_store.ts Outdated Show resolved Hide resolved

peaBerberian reviewed Jan 9, 2025

View reviewed changes

src/main_thread/tracks_store/tracks_store.ts Outdated Show resolved Hide resolved

Florent-Bouisset force-pushed the feat/play-without-audio-or-video branch 2 times, most recently from 5e29ebe to dc7bfe7 Compare January 31, 2025 09:37