Introducing AnimaTrip, an application for low-latency webcam video

[xnakos]

Hi, Jamulus users!

On https://github.com/xnakos/animatrip I have pushed an extended experiment of mine for the last couple of months, aiming to provide low-latency webcam video for multiple users in a simple way as a client-server application. I was greatly influenced by JackTrip, hence the name to pay... tripute! Of course I had also come across and experimented with Jamulus, the brilliance of which I had admired from the start. My target was something complementary to low-latency audio for online performance, from the webcam video aspect. For that I used the fantastic GStreamer library and tools. AnimaTrip functionality is currently limited to Linux-only. You also have to build it on your own, first installing the gstreamer and gio dependencies. Maybe the application proves useful to some of you or just raises questions about low-latency video. Discussions, issues, etc., are welcome.

Thanks and keep jamming!
Harry

[mcfnord]

I've been fishing around for a video option to add to any Jamulus server. I might recompile Jitsi with audio off. I do need a client-server design that embeds in a browser, so the custom client build wouldn't work. But there is a need for a lightweight video-only teleconference solution.

[xnakos]

Thank you for the comment. I have not used Jitsi and I would be interested in what latencies are achieved. In general I am missing what latencies are required and for what purposes. Like, for instance, less that 10ms are required for effortlessly playing while listening to yourself, less than 25-30ms are required for effortlessly playing with others, etc. As for the browser-embeddable client, that makes a lot of sense.

[ann0see]

Sounds interesting. If the latency is good enough to follow a conductor that would change quite a lot.

[xnakos]

Thank you for the insight. I did not have a conductor in mind and now wonder what an acceptable latency for that would be. I mostly had a drummer in mind. :)

[dcorson-ticino-com]

"Low Latency" is a very spongy concept depending on who you are talking too.
Do you have any measured times that you could share with us?
What image update rate are you working at?

[mcfnord]

I agree that conductor-level sync between audio and video would be a breakthrough, and I agree that missing this mark and just reaching "atmospheric fun" would still be very valuable.

I'd like to configure Jitsi for our particular needs: No sound, lowest latency, probably sacrificing pixel count and frame consistency. I believe Jitsi is also UDP and client-server, which suggests its performance could be similar to a Jamulus server. But Jitsi is a bigger server than Jamulus, with higher costs, so it won't be widely deployed for lowest latency the way Jamulus server is. (A no-sound recompile could change that.) Ideally, a video-only server could deploy with each Jamulus server, because the latency of the two channels would be very similar. Maybe video could even throttle itself when the Jamulus server reports performance degradation.

Some commercial competitors offer video, and most say it's a "nice-to-have" but a basic integration that prioritizes audio (of course) would be a breakthrough for us.

[ann0see]

@mcfnord do you know https://jamulus.io/kb/2021/04/18/Jamulus-Video.html

[chrisrimple]

It's not good for conducting a group or other real-time needs, but the WorldJam uses OBS to livestream Jamulus + Zoom to YouTube, Twitch, etc. with the audio stream delayed about 250ms to sync with the video stream.

[ann0see]

@chrisrimple it's probably worldjam.vip

But I think jitsi is sometimes ok ish if the conductor just gives dynamic or non time critical signs.

[chrisrimple]

Oops, yes, fixed. I was thinking of the Jamulus website, which ends in .io :)

[gene96817]

Thank you for all your work. This past year I only have had time to lurk. The speed of sound is about 3 ms per meter (1 ms per foot). Consider the distance many musicians are from a conductor. Many of us are easily 20-40ms away from the conductor. There is plenty of margin for the video to be useful. Gene ---------------------------- Eugene Chang Honolulu, Hawaii ***@***.***

…

On Aug 28, 2023, at 6:20 AM, John Dempsey ***@***.***> wrote: I agree that conductor-level sync between audio and video would be a breakthrough, and I agree that missing this mark and just reaching "atmospheric fun" would still be very valuable. I'd like to configure Jitsi for our particular needs: No sound, lowest latency, probably sacrificing pixel count and frame consistency. I believe Jitsi is also UDP and client-server, which suggests its performance could be similar to a Jamulus server. But Jitsi is a bigger server than Jamulus, with higher costs, so it won't be widely deployed for lowest latency the way Jamulus server is. (A no-sound recompile could change that.) Ideally, a video-only server could deploy with each Jamulus server, because the latency of the two channels would be very similar. Maybe video could even throttle itself when the Jamulus server reports performance degradation. Some commercial competitors offer video, and most say it's a "nice-to-have" but a basic integration that prioritizes audio (of course) would be a breakthrough for us. — Reply to this email directly, view it on GitHub <#3154 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ARAO53NKDNUISH2SYSHFKJ3XXTAM3ANCNFSM6AAAAAA36WQRH4>. You are receiving this because you are subscribed to this thread.