idea: allow to send your mix to other users

[pieterbos]

We rehearse with a big band using Jamulus. Right now we have a sound-check where one person creates a mix, saves it, sends it using whatsapp, and the others open that. Or at least those who want it and understand how this works open it. Then they make any personal adjustments they may need for their own mix.

Of course, that's a bit complicated. Too complicated for a couple of people to do, or rather tricky to use for those of us on Raspberry Pis used only for Jamulus. Especially since there's a perfectly fine network protocol that could be used. So, my idea:

• add a menu item or button to send your own mix to all others on the server
• when doing so, the other people connected to the server receive a message asking if they want the mix, and they can choose to use that mix, or to ignore it.
• accepting this mix changes their personal mix only once. Everyone still stays in control of their own mix afterwards, and can fine tune it.

The message to indicate someone sent a mix could be a chat message with a button or link to apply it, or a separate dialog. I would say a chat message is less annoying and easier to ignore if you don't want the mix.
The option to send the mix could be a simple menu item, perhaps under the current save/load options in File. Since this is not often needed, hiding it away there could mean it doesn't get in the way of other users.

The mix could include levels, pan settings and groups.
A startup flag could be added to set it to auto-accept these mixes, for example to control headless clients remotely.

Of course, this isn't a bit problem, just a nice to have feature. If I have enough spare time the coming weeks I might try to work on implementing this.

But first: Opinions?

[hoffie]

Hi!
Thanks for bringing this up. This sounds like an alternative, possibly more flexible solution to having a single mix controlled by the first user (#599).

I also thought about having the opposite direction: Don't send a mix to another user, but let another user "steal" or copy any other user's mix.

Maybe we should have some overview of advantages/disadvantages of solving this problem (usability, flexibility, implementation).

cc @ann0see @cwerling

[pieterbos]

Thanks, nice additions!

The performance improvements and potential issues are described in detail in #599. Your implementation does not solve this potential performance issue, because you exclude personal monitoring from the mix, so all the mixes are unique. It looks like you should probably allow multithreading again as well, otherwise your solution will perform less well for large groups than regular Jamulus.
The solution to performance problems for very large groups in the original idea/fix was to send everyone the same mix that is calculated only once. If such a feature is required, it may not be entirely possible to separate it from the solution to this issue, because it relies on people getting the same mix, and probably needs to be solved in large sessions before enough people join that performance becomes a problem. The mixing part looks like it has O(n^2) complexity otherwise, but adding more processors will solve it up to a certain point.
Is this actually a problem that needs to be solved?

If there really is a requirement to disable local monitoring for peoples own mix remotely, that is a specific feature as well, that could to be considered in the design for each alternative approach. I would be highly annoyed to lose personal monitoring in a group - it is rather important for me to hear my own sound as others hear it, and it helps with timing. Is this is a way to solve problems for people with too high latencies, due to audio interface or network issues, in groups where timing is not as important?
The --mutemyown command line parameter could maybe help here?

I'm not sure I agree on what is the least or most complex UI addition. The sending the mix to others feature can be hidden away unless you need it - the option to activate it can be put in a menu, and only is one button. If done with chat messages, it does not add any complexity or even UI Elements unless it is used, and could be self-explanatory for users. The button hidden away in the menu would be mostly for one more advanced user per group, possibly with a hot-key to enable quick sends.

The stealing the mix feature would need a way to select which mix to copy. It could be done with a button per channel - that is adding many buttons, not just one, for users who don't need it. Or alternatively it could be hidden in a menu or a single button, but then it would require an additional dialog to select the user to copy the mix from. Also for very large groups, and some people use 60+ members in a group, it could be a challenge to find the correct user to copy the mix from. So I would say the option with the less complex and intrusive UI could be sending the mix, and the more complex UI could be stealing the mix.

[cwerling]

Let me throw in a visual idea for the stealing mix:

The dropdown's menu has these options: "None", "{Client 1}", "{Client 2}", ..., "{Client n}".

After selecting a not-None entry from the dropdown, our mixer board will flip to the respective client's mix and will be controlled by the respective client for as long as the client's item is selected in that dropdown.

Then, I see two variants:

Option 1: All controls become disabled (as seen on the right side in my PR) as long as a not-None dropdown item is selected. We are not able to change the mix unless we select "None" again in the dropdown.

Option 2: All controls keep being enabled. When we make the slightest change at any fader/button, the dropdown's choice jumps back to "None" and we're back in normal local mixing mode.

[pieterbos]

Looks like a good and simple idea!

A friend of mine suggested Option 3: when using someone elses mix, you can use the controls to deviate from that mix, but you keep following that user. For example to make people who play in the same section a bit louder, to change the volume of your own monitoring, or even to disable it. The sliders would store your personal deviation of the gain set by the person who controls the mix.
This could for example be used to make the section you play in a bit louder, or to change or even mute your own monitoring.
The sliders could show the mix you follow, or it could set all sliders to the middle, meaning follow the mix exactly (unity gain, so 1.0), and setting the sliders higher or lower could deviate from the middle.

I would personally prefer a way to just copy the mix from the other user and regain control directly after that, but I guess there are different users with different use cases. And all three options could allow for this use case.

[cwerling]

@DominikSchaller just brought up an important disadvantage of (my) single mix: If the mix master "solo"s sombody (e.g. to find the culprit for noise or musical mistakes), everybody will hear that solo. This is suboptimal. Unfortunately, I don't see any easy way on the protocol side for the server to filter out "solo"ing from normal muting/gain-to-0 changes.

@pieterbos Could you be so kind to update your initial overview with this?

[pieterbos]

I updated the overview.

It could also be a hybrid solution, so sending the mix where you can select to auto accept new changes from that person during the session. Or a system where you can steal a mix where the mixer can pause broadcasting his mix, in order to adjust it, then resume sending to others. These are a bit more complicated to implement, I guess. Keyboard hotkeys would probably help here :)

[pieterbos]

Ok, did some thinking, with all the previous input. How about the following design:

• A new menu item, under file probably, called 'Start broadcasting my mix'
• When clicking it, it toggles a server state that indicates this user is now broadcasting his/her mix. The menu item changes to 'Stop broadcasting my mix'
• Other clients now see a dropdown box as in idea: allow to send your mix to other users #1380 (reply in thread) . It only appears if someone is broadcasting their mix.
• Next to the dropdown box is a 'follow mix' button, in the same style as the mute, solo and group buttons
• when moving any of your sliders, you stop following that mix and are now back on your own.
• when unfollowing the mix, you keep the state from the mix you were following when pressing the 'unfollow mix' button
• when the broadcaster stops broadcasting, the UI elements disappear and everyone is on their own, keeping the last state of the broadcasted mix
• a startup flag can be used with a name to automatically follow the mix from for headless clients.

that way, there is absolutely no clutter in the UI unless a mix is being broadcast. If there is a mix being broadcast, it is very clear that this is the case. It should be easy to follow that mix, or to unfollow it, making this simple to use for people who just want to follow. People joining later can still follow this mix because of the server-side state. There is no privacy issue from others seeing your mix, preventing 'you muted me!' arguments. The headless client case is supported as well.

Drawbacks are that this feature is a bit hidden for the person wanting to broadcast their mix (also a benefit!), and that personal adjustments mean unfollowing the mix altogether. Another drawback is that checking for problems by muting everyone is not possible. that could be solved by starting a second jamulus instance with a second sound card (often possible because people use an external audio interface and have their internal sound card as well).

Possible nice to have additions:

• the person broadcasting gets a pause broadcast toggle button instead of the dropdown in their UI, to solve the 'first adjust the mix locally, then broadcast it later' problem
• the dropdown is great for a large number of people broadcasting their mix. It could be a simple list of names+buttons for smaller numbers, as this usually will be at most one person, and it could auto-switch to a dropdown with > 2 broadcasters, Or it could be limited to only one person broadcasting their mix at the same time.

[cwerling]

This seems like a thorough concept. But I must say that I am still working on bits and pieces on the plain Single mix variant (so far probably spent 5 full working days on it and counting).
I'm at the moment not able to rewrite it all according to this concept, but still willing and able to take on additional tweaks to my current PR.

[cwerling]

(But it obviously doesn't have to be implemented by me ;))

[pieterbos]

I may have some time, but not sure how much. It has been many years since I last wrote C++, and I have never done any QT work, but we'll see.

So far, I defined three new protocol messages:

- PROTMESSID_BROADCAST_MIXER_STATE: Set the mixer state for a single channel

    +-----------------------------------+
    | 1 byte broadcast enabled/disabled |
    +-----------------------------------+

- PROTMESSID_BROADCAST_MIXER_STATE_LIST: A list of channels that are broadcasting their mix

    for each channel that is broadcasting, will contain a 1 byte channel id
    if nobody is broadcasting, contains no data, n = 0

- PROTMESSID_FOLLOW_BROADCASTED_MIXER: Indicate which mixer to follow, or to unfollow

    +--------------------------------------------------------------------------+
    | 1 byte client wants to follow or unfollow | 1 byte channel id to follow  |
    +--------------------------------------------------------------------------+

• PROTMESSID_BROADCAST_MIXER_STATE to be sent from the client that wants to broadcast or stop broadcasting to the server
• PROTMESSID_BROADCAST_MIXER_STATE_LIST to indicate any changes in broadcasting state, from server to client
• PROTMESSID_FOLLOW_BROADCASTED_MIXERto be able to follow or unfollow

Upon follow, the server will:

• send a PROTMESSID_CHANNEL_GAIN+PROTMESSID_CHANNEL_PAN message to the client per channel
• will store that this client is following someone, probably a vector with a channel id per channel, or some special value to indicate not following
• upon any change in pan/gain of the followed client:
  • will update the gain of any following clients
  • will send a new PROTMESSID_CHANNEL_PAN or PROTMESSID_CHANNEL_GAIN to all following clients
• will set the

possible optimization for performance with many clients:

• when following, calculate only one mix + opus encoded data for any following clients, and send that.

Looks like when implemented this might already set the faders correctly

to decide:

• do we need a new message for sending a full list of channel gains+pans upon follow, or is many messages ok?
• how to indicate 'disable local monitoring'?
  although I do not understand the use case for disabling local monitoring - is this for choirs who do not care about timing with high latency audio hardware?
• does 'solo' just send many separate gain messages?

Any comments?

[pieterbos]

Started development in https://github.com/pieterbos/jamulus/tree/broadcast_mixer . So far the basics:

Next steps towards a basic implementation:

• implemented the messages to start and stop broadcasting a mixer, plus the necessary events in protocol, channel, client and server
• storing broadcasting state on the server and client, in CChannel
• sending a list of mixer broadcasters to all clients on every change of mixer broadcasting state
• created a broadcast or stop broadcasting menu item that toggles broadcasting state
• for now, as a test, make auto clients automatically follow the mixer broadcaster with the lowest channel id, if there is one
• storing follower state on server and client
• set the correct levels server-side - this is now a working single mix.
• implement auto-unfollow when someone stops broadcasting or a broadcaster disconnects, server and client side
• initial sending of all pan/gain states to a newly-following client
• broadcasting any pan/gain changes to the following clients
• set mute/solo state correctly in the UI when receiving gain/pan levels
• Send broadcasters list to client upon connecting to server. Should be: Request broadcasting state, which will auto trigger a send to all clients, which solves the next issue, which is:
• Send following and broadcasting state from client to server upon restart of server
• Implement setting faders + pans on change Pan and Gain messages on the client
• add UI to be able to follow/unfollow clients, replacing the current auto-follow the broadcaster with the lowest channel id.
• add command-line parameter to create auto-follow functionality
• implement auto-follow functionality when enabled using command-line parameter.
• proper consistent naming throughout the code

To optimize

• don't just set the gains from the other server, also send the exact same audio for faster processing.

To check:

• Thread safety: is the Mutex that is locked in server shared with the channels, so no problem with IsFollowingMixer and getting the mixer channel in channel.cpp ? Or is another mechanism needed?

[hoffie]

Thanks @pieterbos and @cwerling for working on this! It seems like there are multiple approaches to solve this issue and it's great to have a structured list of advantages/disadvantages.

Sadly, neither I nor any other @jamulussoftware/maindevelopers had time to dig into details until now. I think there is openness for having such a feature, but there is no clear tendency yet what approach would fit Jamulus' principles best.

Freel free to go ahead with your proof of concept implementations, I'll be sure to try them out. Please keep in mind though that it is highly likely that we will have to settle on one of the approaches at some point.

[pieterbos]

Thanks for your message!

Of course, this does not look like multiple implementations of this single feature are needed in the end. I do think Jamulus could benefit from an approach that covers enough use cases, and one that is simple enough to use without getting in the way of people who do not want to use this.

@cwerling my broadcast a mixer to others proof of concept implementation now has enough working code to send a single mix to all currently connected clients, if enabled through the menu of the mixing client. Your feature does not cover our use case, but I think this one could cover both ours and yours. Are you sure you don't want to join efforts, and split the implementation work?

[cwerling]

Thanks @hoffie for the heads up! This is an interesting and worthwile journey.

My own approach is close to be done and it's very much tailored to my own choir's needs, i.e.

• ability for everybody to control monitoring individually
• ability for mix master to 'secretly' solo people
• ability for non-mix masters to constantly follow the master mix

I understand these are quite specific to our environment (semi-professional choir with low-latency hardware and Jitsi session for conducting).

I'm generally open to a collaboration, @pieterbos (developing together is always more fun!). I will let you know later this week if I find time and motivation to contribute to another implementation.

[dakhubgit]

Ok, here is my idea for that kind of functionality: namely a button for following (rather than stealing) someone else's mix. That reduces the numbers of mixes that the server has to create. For large ensembles where a useful mix for getting all the important cues is important, everyone in a section can follow the section leader. Following someone's mix means that you cannot change your own mix anymore unless there is some circularity established also following you (yes, this becomes a lot of graph theory handwaving in larger groups and needs to find a more formal and dependable as well as practically useful definition so that it becomes reasonably straightforward to arrive at a setting with a single mix that everyone can adjust, either per section or for all).

[pieterbos]

Thanks. That is close to what I described in the latest form of this idea, in detail, and is what I am implementing, including the term 'follow'. If you check out the code in the fork mentioned above, it already partially works, except clients now auto-follow because I have not build a follow button in the UI yet.
Except that in mine you need to explicitly broadcast before someone can follow, so it is easier to find the correct mix to follow. Plus you cannot broadcast and follow at the same time, avoiding the following another follower case, which can lead to cycles that could cause problems, which would probably mean relatively complex code to handle.

[pieterbos]

And if you want something adjusted, just stop following by adjusting, and broadcast the adjusted mix.

[dakhubgit]

The difference here is that I am not sold on the "broadcast" thing and that the avoidance of cycles also means that you cannot work with a common mix that everyone in a set can adjust.

[pieterbos]

That sounds like an interesting experiment, a cooperative mix . I wonder if it would work, or would cause annoyances and arguments.

Could perhaps also be done by broadcasting and indicating that others can edit the mix? Then the cycle-detection code would not be needed, and arriving at a single mix would require much less coordination.

[pieterbos]

Perhaps in that case, broadcast is the wrong term, and it could be a second option with a name such as 'create a cooperative mix'?
That others then could join, or perhaps participate in, instead of the more passive sounding to follow?

That is not hard to implement, apart from to decide what to do if two people grab the same fader at the same time. Could you perhaps explain why you would want such a feature, and what it would solve?

[dakhubgit]

Pieter Bos ***@***.***> writes:

Perhaps in that case, broadcast is the wrong term, and it could be a second option with a name such as 'create a cooperative mix'? That others then could join, or perhaps participate in, instead of the more passive sounding to follow? That is not hard to implement, apart from to decide what to do if two people grab the same fader at the same time. Could you perhaps explain why you would want such a feature, and what it would solve?

"Following" basically means not taking responsibility for what you get to hear. A lot of people in a group setting, particularly a large group, will not be eager to bother with something as technical as creating an individual mix. Others will enjoy putting together something particular. Currently the main course for not having to mess with controls is to tell someone when they are too loud or not loud enough so that just keeping every control at the same level works reasonably. However, in a large group setting the server would be taxed with creating, say, 32 separate mixes and a single mix would likely not work well for both sopranos and basses, so there is some need for something in between. A main technical incentive for giving up an individual mix is that any shared mix reduces server workload. I thought that the setup for following would be somewhat reliable in allowing for several different but significantly reduced in number mixes. When followers enter a circle, that would create a lockup where nobody can change the mix anymore unless they stop following for a moment. The idea was that when detecting such a situation, to use it for just considering the circle as a single synchronised group and unlock it for change by everyone instead of change by noone. I am not sure that this resolution of what would otherwise be a lockup of mutually released responsibility is a sensible interface, to be honest. When doing it spontaneously, it would likely result in some random group of masters (either synchronised or independet) and followers without control, with the overall outcome hard to plan for. So it might be too cute to be useful.

…

-- David Kastrup

[pieterbos]

So, the first working proof of concept is finished enough to show how this would work. The code is at
pieterbos#1

It is demo/prototype quality, intended to show how it works and to discuss if this is the way to go, both functionally and technically. The linked pull request contains a description of the status and all the things that are unfinished. The basics should work well though. You need both the server and client of this branch to run this.

Opinions?

[cwerling]

Sounds like a solid start, will test this asap!

[ann0see]

Any updates on this topic?

We tested the master mix #1381 once and I'm still not fully convinced. I do see a huge benefit if actually only one mix is needed; unfortunately, I don't think that's the case in reality (Sopranos might not want to hear Tenors too loud, while Tenors would not like to hear Sopranos too loud; a server might be used by two bands/...). The major disadvantages I see are:

1. It doesn't reduce server load
2. You need to rely on one good mix

Nevertheless, it seems to be almost fully backwards compatible which is a huge benefit.

I think the most flexible approach would be the broadcast mix functionality and probably people would ask for it in future?

[pieterbos]

Well I made a proof of concept implementation, to be able to try this. see the post earlier in the discussion for the link to the fork if you want to try yourself. I decided to wait a bit to see if there was more interest. So far, you are the first to show interest after I posted the proof of concept.

The status of my implementation is just enough to demonstrate the concept fully, but should be good enough as a basis for a full implementation . I haven't had much spare time recently, but I could probably make some time.

This approach can work with older jamulus clients. the broadcast button needs to be disabled for older servers, and older clients will be unable to follow the broadcasted mix. I
Haven't had much time recently, but I could probably make some time to spend on this. A check if my code is in the right direction would be helpful in that case. I could open a draft pull request?

[ann0see]

Thank you for the quick reply. I'll try what I can test.

and older clients will be unable to follow the broadcasted mix.

That's a small disadvantage unfortunately (at least for my choir) we've just updated and I doubt my choir members would like to go through the hassle of updating again. I think some would do it, but surely not all. Nevertheless, I might be able to test it (but I can't guarantee anything)

I haven't had much spare time recently

Same for me.

I could open a draft pull request?

Sure. We have a lot of open PRs at the moment and I hope it will get visibility.

[pieterbos]

I would say this is not ready to test with a full choir yet in its current state - unless they are ok with things not working in quite some edge cases :)

[ann0see]

Ok. Just compiled it and yes, probably it still needs some work. No rush needed ;-). Also we haven't agreed on a way of implementing this, so probably it still needs some time ;-). No need to rush anything. I wouldn't suggest a Draft PR then.

[dakhubgit]

ann0see ***@***.***> writes:

Thank you for the quick reply. I'll try what I can test. > and older clients will be unable to follow the broadcasted mix. That's a small disadvantage unfortunately (at least for my choir) we've just updated and I doubt my choir members would like to go through the hassle of updating again. I think some would do it, but surely not all. Nevertheless, I might be able to test it (but I can't guarantee anything) > I haven't had much spare time recently Same for me. > I could open a draft pull request? Sure. We have a lot of open PRs at the moment and I hope it will get visibility.

With regard to user interface: I wonder whether it makes sense to combine this with the "mixer group" menus since in some manner you don't lock to your own sliders then but to someone else's? Not sure this makes for a convincing concept but thought I'd mention that handwaving idea anyway.

…

-- David Kastrup

[pieterbos]

I'm not sure I understand what you mean. Do you mean to add the follow-button in the slider menu?

Indeed, you lock to someone else's sliders, but as soon as you move one, you do go back to your own mix - this is not implemented in the proof of concept yet.

[ann0see]

Just tested it and there were some warnings:

src/server.h:144:47: warning: unused parameter ‘bIsBroadcastingMixer’ [-Wunused-parameter]

and GUI issues (the follow mix checkbox remained even after I disconnected from the server). Probably that's ok for a PoC?

[pieterbos]

Yes, that parameter is actually needed for something that is not yet implemented, that will fix one of the edge cases :)
and that is indeed the state of the proof of concept - demo-ready, and to discuss whether this is the way forward.