Handle audio issues like feedback (semi) automatically

[ann0see]

Continuing #797

In short:
How can we improve the following situation:

1. Inexperienced user connects to a Jamulus session with wrongly configured inputs/outputs —> feedback
2. Inexperienced users haven’t set new client level
3. How do we ensure that he knows about this issue and how to fix it?

I hope that’s right?

Possible solution: automatic feedback detection/suppression?

[ann0see]

It would be difficult for a simple algorithm to detect feedback. Jamulus already indicates (latches) a clipping condition.
We could detect when a channel is "too loud" (clipping for a "long" time) and suppress it so that it doesn't make the session unusable for the other musicians. Suppress could mean turn the volume down to something very low (5%?) so that someone could troubleshoot that channel with continuous sound. Suppress could be something more drastic (e.g. kill or disconnect) but then it will be harder to troubleshoot.

@gilgongo We can do more analysis on "abnormal" traffic if the packets are numbered. Then we can do things like (1) detect out of order packets, (2) detect erratic interpacket times, (3) detect a long delay and then see all the late packets causing xxxx, etc. This is interesting for understanding when bad Internet transport is affecting our sound. This could also be a rabbit-hole.

I think numbering the packets and collecting interarrival rates in a log could be really helpful with minimal processing. This still does not help in creating code to detect feedback.

Is it possible some people on this thread are using the term feedback to mean some other audio defect?

Originally posted by @gene96817 in #797 (comment)

[ann0see]

Detecting feedback would be great. But Jamulus should not automatically mute or lower the volume of this user (this could become really annoying in my opinion). We could add a red warning like the "you are muted" warning which says: "Your signal has been very loud for quite some time! Do you hear feedback? Try to mute yourself and check if you‘ve set up your in/outputs correctly!" [Don‘t show this message again] [Mute myself]

[ann0see]

Detecting feedback will take the same kind of code as eliminating feedback.
Let's clarify this thread to dealing with a user that is overwhelming the session with too much sound.
Then we can decide what conditions of "too much sound" are bad and need/deserve automatic suppression.
Some degree of automatic suppression will be needed, especially for a naive or inexperienced user that will respond slowly to muting themselves.

For fans of loud music, perhaps the threshold needs to be adjustable (i.e. triggering volume and duration or even disabled). Acoustic musicians probably want a lower threshold. People with tinnitus probably need a lower threshold too.

Originally posted by @gene96817 in #797 (comment)

[gene96817]

If we really want feedback detection, perhaps we should build echo cancellation into the client.
However, echo cancellation added delay. Alternatively, we could use the echo cancellation to detect echo but not insert that code in the data path. When the echo cancellation kicks in, we can use that condition to trigger the messages (or indicators) to the user.

What is good about this approach is the client computer should have plenty of compute cycles for this. All the telephony and video conferencing apps have built-in echo cancellation. Most if this is in the codec. Does the codec we use have echo cancellation?

[ann0see]

Echo cancellation would probably result in the typical audio quality issues of video conferencing tools.

If we can use it to detect problems that‘s good.

We should find ways to show the user what he‘s done wrong.

[gene96817]

I was referring to echo cancellation as a class of well-known algorithms that can automatically detect an echo.
The problem with video conferencing tools for singing has nothing to do with echo cancellation. It is their assumption that in a well-run meeting, only one person should be speaking at a time. Note, we do not want to run the audio to the server through the echo canceller because the echo canceller has a delay line as part of the algorithm to find the matching pattern (that is the echo). (The delay line stores the original signal for the pattern matching to find the echo.)

[ann0see]

The delay line stores the original signal for the pattern matching to find the echo.

Thanks for the clarification!

[gilgongo]

@ann0see

We can do more analysis on "abnormal" traffic if the packets are numbered.

Do we need to open a ticket for packet numbering or is that already being worked on?

This is interesting for understanding when bad Internet transport is affecting our sound.

By "interesting" do you mean that it would allow a user to be able to know whether (for example) to adjust their client's jitter buffer and not the server one? Or to reduce their audio quality setting rather than their buffer delay setting? Otherwise (as I keep asking), what's the point in having information if you can't reliably act on it? And more to the point - would such adjustments be able to be done automatically without user intervention? That would be cool if so...

[ann0see]

Do we need to open a ticket for packet numbering or is that already being worked on?

No. I don't think someone is working on it. Feel free to open one

[gilgongo]

@ann0see I think an engineer needs to open that ticket really as I'm not qualified to do so.

[gene96817]

Could someone show me what the payload of the UDP packets looks like?
I'd be glad to work with someone interested in coding a solution
.

[WolfganP]

Could someone show me what the payload of the UDP packets looks like?
I'd be glad to work with someone interested in coding a solution

See https://github.com/softins/jamulus-wireshark/

[ann0see]

Will convert this to a discussion since I think it's not yet ready.

[JohannesBrx]

What about the following idea, which should catch the majority of audio feedback cases: After the user has connected or unmuted his microphone during a connection, we monitor the microphone input level for about 3 seconds. If the level is too high during this time or even clipping occurs, we mute the microphone and show the user a short information dialog. When the user unmutes the channel again we would not check the level again because we know that it is intentional.

In my opinion the most cases of audio feedback occur when the user connects to the server and has wrong settings. On the other hand, it is quite unlikely that the user joins a server and directly starts with full microphone volume levels. Checking the microphone level (instead of output level) has the advantage that it does not cause a false alarm when a band is jamming at a high overall output level or the mixer settings are simply wrong. The monitoring lasts for only 3 seconds after connecting/unmuting because high microphone levels might also occur during jamming and we do not want to cause false alarms.

I think the implementation of this strategy would be rather straightforward and avoid many annoying situations in Jamulus.

[ann0see]

Yes. That's a good idea. In this case, it has to be very clear what the user must do to recover to a "working" state.

[JohannesBrx]

In that case I would suggest in addition to the mute myself that we lower the users fader so that they still hear that there is a problem they need to fix, but don't cause hearing damage.

When I pull down my channel slider to approx. 10%, feedback vanishes completely. In that case, it would almost have the same effect than muting the channel.

[JohannesBrx]

@JohannesBrx @ann0see (and anyone else) Do we think this is a viable solution? If so, we should create an Issue (or a PR) for the work, and reference this discussion.

@gilgongo As soon as I have the time for it, I could prepare a draft PR such that we can test this. I would reference this discussion. Maybe a live test will give us more valuable clues about the correct strategy.

I would start off with the following basic implementation:

• detect microphone level and check for clipping (or high values) for 3 seconds after beginning a new connection

In case of feedback detected:

• activate mute myself
• either mute the channel or pull down the fader to a low amount
• show some kind of visual feedback to the user

Later on we may decide about whether to restart feedback detection after unmuting and all the special cases discussed here.

[ann0see]

Great thanks!

show some kind of visual feedback to the user

I'd just use the existing red label which shows up once you're muted. Just change the text programatically (and maybe use another red tone.)

[gene96817]

Once we get this working, could we reexamine the need or a 3-second interval. Our software will hear things faster than a human. It should be possible to "hear" the condition and correct it in less than a second.

3 seconds is a long time for a new connection to disrupt a serious rehearsal in progress.

[ghost]

One would use convolution in the design an adaptive filter to suppress echo, or feedback, or background noise (some may call this deep-learning) ... Jamulus already accomplishes this without an idiot-proof method, as RTFM.
One could use AGC (automatic gain control) in the form of a compressor ... maybe.
Muting will lead to unmuting, which leads to muting, ...