What's wrong with: "AAA JaZZcape" server on "Any Genre 1"

[dingodoppelt]

Hi!
On serverlist "Any Genre 1" there is a server called "AAA JaZZcape" that has been showing up full with 25 people for the last few days but is actually down.
Can this be a bug in the central server code?
cheers, nils

[ann0see]

Hi Nils!

Is this server always filled by the same people?

Tagging @jamulussoftware/maindevelopers here.

For me it seems to be hosted on a home connection. Strange...

[softins]

I don't think it can be a bug in the Central Server. The Explorer back-end first receives the list of servers from the directory (central) server, and then talks to each server in the list, first doing a ping (twice), and then requesting the Version and OS, followed by the client list. Since Explorer is displaying all that info from AAA JaZZcape, the server is up and responding to those requests and the data is fresh. The fact that the server is full would explain an inability to connect to it.

[ann0see]

Something seems to be quite wrong with it (the server). Why would a jam take that long?

Tagging @passing here (we might want to E-Mail with him concerning the E-Mail we received a few weeks ago).

I doubt we can contact the server admin ;-(

[softins]

The only thing I could think of is that the server is sane enough to respond to messages, but that the timer event that detects and removes clients with no traffic has stopped firing and timing them out. I haven't studied that area of code.

[softins]

Looks like they are a big band, who could be contacted via either https://www.facebook.com/jazzcape or https://www.jazzcape.nl/

[softins]

I have messaged them on Facebook to enquire.

[softins]

I had a chat with the guy, and he has reset the server. It is running on a RPi. I said I thought 25 clients was a bit ambitious for a server on a Pi, and I see he has now set it to 15 instead :)

[gene96817]

a max of 15 is a good choice. My RPi4 changes from usable at 15 to overall delay >500 at 16 -17 users.

[passing]

I just tried to reproduce this by running a jamulus server in a docker container with a hard CPU limit and then connected Clients to it to reach that limit. But did not manage to get it to a situation like described

[hoffie]

I can also confirm this observation. An non-scientific explanation attempt:

• Jumps in resource usage may be explained by the algorithmic behaviour of the mixing algorithm. Mixing should have quadratic behaviour. On a very high level: When going from 4 clients to 5, mixing operations increase from 16 to 25 (+9). Going from 24 clients to 25 will already increase opreations from 576 to 625 (+49).
• If Jamulus fails behind once (i.e. is unable to mix in realtime), it will possibly never be able to catch up again which would explain why "infinite" ping is reached so quickly.

[dcorson-ticino-com]

Pretty obvious once you point it out.

[gene96817]

Mixing should have quadratic behaviour.

Sounds like the computing load for mixing grows by n^2? Can someone confirm this when going from 20 users to 100 users?

If Jamulus fails behind once (i.e. is unable to mix in realtime), it will possibly never be able to catch up again which would explain why "infinite" ping is reached so quickly.

What does it mean to "catch up?" Do you think "falling behind" and "catching up" are affected by late packets?

[passing]

Interesting! how comparable is your docker container CPU limit with the RPi4?

My setup was just local, and I set the CPU limit so low that more than 1 client was "too much". Then the server got slow (showed ">500ms overall delay"), but returned to normal when I disconnected the clients. So it sounds resonable that "high network latency plays a role."

The CPU usage for the channel mixing should grow by n^2 - but the CPU for OPUS decoding and encoding just by n, I guess?
And then the question is, which of these operations is more costly / how much is the total CPU usage linear and how much quadratic in relation to the number of clients?

[gene96817]

I like your test environment. It promises to be useful to confirm CPU requirements for different numbers of users.

It is (almost) impossible to have an algorithm where computing grows by n. An efficient algorithm usually grows by nlogn. An algorithm that grows by n^2 suggest scaling problems. I am really curious what is the real CPU demand for n= 10, 20, 50, 100, 150.

[dingodoppelt]

Hi everybody,
on server list "Rock" there is another server with this issue:

29 | Thomakaju | Kamen | Germany | 16 / 16 | 37 | X.X.X.X | 22124 | 3.7.0 | Linux

shows up full with the same people for about a week now. (horror movie material if you ask me ;)

[softins]

Hi everybody, on server list "Rock" there is another server with this issue:

29 | Thomakaju | Kamen | Germany | 16 / 16 | 37 | X.X.X.X | 22124 | 3.7.0 | Linux

shows up full with the same people for about a week now. (horror movie material if you ask me ;)

Yes, we've been aware of that one for a while, but have no idea how to find the owner.

We think what is likely is that the audio timer has either stopped or is locked, so the channels are not timing out. If the server is running with multi-threading, it might be an issue with the futures. The multi-threading implementation changed between 3.7.0 and 3.8.0. But without being able to contact the server owner, we can't find out anything or get it reset.

The rest of the server is still running, so it responds to protocol messages and returns its client list. But as it's full, we can't even fetch the welcome message, if there is one. @dingodoppelt do you recognise any of the participants in Germany?