Decoding artifacts when using ingress and WHIP

[slabajo]

We have experienced decoding artifacts like pixelation and even green screens on browsers receiving video tracks from an ingress that is using WHIP.
We verified that the video that is fed to ingress using whip was correctly encoded and in fact we could record it and see that the video at whip ingress was correct. However at the same time, the video received in all browsers subscribed to the track show decoding artifacts. We have been able to reproduce the problem using H264 and VP8. and if many browser subscribe to the tracks, all of the show the same problem at the same time

We have analysed the ingress and we think we have found possible causes for this problem: basically on ingress frames may be dropped to several causes, that may happen when media comes from WHIP. The problem seems to be that dropping samples in the middle of an encoded video stream casuses gaps in the stream that will make the video player impossible to correctly decode the stream until a new keyframe appears on the stream. The main problem is that after a sample is dropped subsequent samples are still pushed downstream even to the RemoteTrack that without sequence numbers has no way for livekit server or the browser to detect that there is a gap on the stream. Thus, when this issue happens, the decoder on the browser cannot detect the gap and just tries to decode frames causing decoding artifacts.

This happens in two places:

• The jitter buffer used at the reception of WHIP packets may drop late ones. It is true that in this case, jitter buffer dropping samples will also cause a PLI request that theoretically should generate a key frame at some point in the near future, but until that happens, subsequent frames that has lost their previous frame reference will be pushed downstream, making the decoder to cause decoding artifact that may last from just a flash to a few seconds depending on when the keyframe appears
• Also the output synchronizer may drop samples

  ingress/pkg/whip/sdk_media_sink.go

  Line 271 in 2204d43

  drop, err := t.sink.outputSync.WaitForMediaTime(ts)

  and in this case apart from pushing downstream subsequent frames, the point is that no PLI request is generated, so the publisher has no way to know that a keyframe is needed and the decoding artifact may last for many seconds, even minutes, in fact until for any other reason a keyframe is generated.

We have created a fix for this that basically whenever a sample is dropped, it just drops all subsequent frames until a keyframe appears. Also for the sdk_media_sink.go the fix includes that if a sample is dropped a PLI is requested upstream.

We have verified that with this fix the decoding artifacts completely dissapear, so we are pretty sure that the issue is as we described. However, I am afraid that the fix also impacts the output synchronizer causing freezings longer than what theoretically should appear. We haven't gone further on that analysis, but it seems that as when a sample is dropped a sequence of samples is dropped, this may be impacting the output synchronizer that handle sample durations thus having more difficult to correctly synchronize when not short gaps appear on the stream.

[biglittlebigben]

The jitter buffer and output synchronizer have been removed from the bypass_transcoding path. Are you still experiencing these issues with the current ingress server?

[slabajo]

I haven't tried, the feature was urgent to us, so we decided implementing our own ingress feature. If I find time to test I will let you know