Meaning of htop metrics for a Jamulus server?

[ve3meo]

I could use some help in understanding what the Linux htop resource monitor is telling me about my server after having searched and browsed for something relevant in the discussions. At the time I took the screenshot, there were 11 clients and no recording on a virtual machine described as "Available" (the lowest monthly fee), 1 vCPU (2667MHz), 1GB RAM, 20GB SSD running 3.6.2 on Ubuntu 18.04. Also runs Apache server for Jamulus-Recording-Remote.

For reference, the host's Dashboard graph showed:
CPU 47% avg max
Network 4.8Mbps avg max

1. Why are there multiple Jamulus PIDs? With 0 clients, 7; in this session, 8.
2. PID 1706 is always highest CPU%.
3. PID 1706 ranged from 25% to 100% during session but the CPU meter never went over 65-70%. Are the differences among Dashboard graph (ca 47% max), CPU meter (ca 70% max) and PID 1706 (ca 100% max) simply due to the duration of the average?
4. I did not hear any dramatic change nor did latency jump when PID 1706 maxed out to 100%.
5. What does "Load average" mean? How does it relate to CPU%? It ranged from 0.2 to >2
6. How close were we to server overload, skyrocketing ping? I've noticed that the Dashboard graph shows a rise in avg CPU of ~1/3 when I turn on recording.
7. Does anything jump off the screenshot that I should be concerned about?

Thanks for any guidance you may offer.
Tom

[ann0see]

Hi! Thank you for your post. Did you already ask in the multi threading discussion? I‘ll search for it

[ann0see]

Linking #1052

[gilgongo]

That screenshot doesn't seem to be showing anything bad if we assume your machine has more than one CPU core. "Load average" is a general indication of how hard the machine in working. It takes account of things like CPU and I/O latency. If you have a machine with a single core CPU and your load average (the three numbers are the averages over the past 1, 5, and 15 minutes) is 1 then you are using 100% of your "capacity" (mostly the CPU). If you have 4 cores, then 100% capacity is indicated by a load average of 4 and so on.

So - how many cores do you have?

Other than that I see you aren't swapping out which is good (so you have enough RAM) and there aren't any other potentially difficult processes listed like databases etc. which might try to grab resources. As to the issue of multiple Jamulus PIDs, that's just the way Jamulus rolls I think.

Unless you intend to host more than about 30 or more players at one time, it seems to me that server hardware capacity in most cases will not be the source of problems. Much more likely to be errant clients, their hardware and networks.

[gilgongo]

If you type cat /proc/cpuinfo | grep CPU how many lines appear? There will be one line for each core.

EDIT: I think if you're on a VM then you may need to do cat /proc/cpuinfo | grep cpu instead and look at the "cpu cores" value.

If you have just one, then you do have a load problem, since a load average above 1 means that the CPU has to "queue" requests for its time (like traffic over a bridge, say). I'd expect a load average of > 1 for more than a few minutes at a time to cause problems with Jamulus, certainly. As to the utilisation of Jamulus PIDs then that too would be a problem if they can only call on one core (otherwise it's fine to be using 100% of a core for a while if there are other cores available).

Noisy neighbours are hard to detect properly - and practically speaking, what would you do if you found out that somebody's else's CPU usage was affecting yours?

[ve3meo]

"CPU" as filter gives:
model name : Intel(R) Xeon(R) Platinum 8270 CPU @ 2.70GHz
"cpu co" as filter gives:
cpu cores : 1

So the Load average in excess of 1 when 1 Jamulus PID is at 100% CPU and others are non-zero and there is no apparent symptom of overload does not suggest that some processes are stealing cycles from some other CPU or core on a VM?

[gilgongo]

Without knowing how the machine you are using is set up exactly it's hard to say what an acceptable load average would be. But if you only have one core available (Platinum 8270 has 26 I think), and your load average is 1 or above then that means you're running at or above 100% of capacity. Perhaps ask your ISP to confirm that.

Fundamentally though: are you experiencing some form of degradation that can't be traced to clients? If so, then it's either the server or the network(s) the server or the clients are on. If it's the server, and regardless of whether something is stealing cycles from your CPU or not, then if you are seeing a load average of > 1 then I'd say you need to upgrade :-)

I can't answer your question about how the individual Jamulus processes work, I'm afraid though.

[ve3meo]

Trying to relate the metrics and Jamulus performance is still proving elusive to me. On that session I monitored with htop, I cannot say I saw/heard any difference in degradation between a Server Load average of 0.25 and 2 or between a CPU% for the main Jamulus PID of 100.0% and some lower value such as 32% in the screenshot. Ping and Overall Delay remained much the same. With nothing new started or stopped on the client side, it's a mystery why the Load and CPU% should soar and descend.

I've yet to experience a limiting behaviour that would persuade me to upgrade the server. Having seen so many advocations for dedicated hardware, have I just been extraordinarily lucky that our utilisation at up to 12 clients, sometimes with recording, has sailed under a time-varying limit that has so far been very accommodating on a shared-resource VM?

I'd like to set the user limit for Jamulus to a reasonable number to preclude overload when the server is operating without a noisy neighbour, recognising that the number differs with the recording option and, likely, the clients' channel and quality settings.

"The Intel Xeon Platinum 8270 is a server/workstation processor with 26 cores, launched in December 2018. It is part of the Xeon Platinum lineup, using the Cascade Lake-SP architecture with Socket 3647. Thanks to Intel Hyper-Threading the core-count is effectively doubled, to 52 threads."

[gilgongo]

For all I know, your ISP is using some method of allocating CPU that means the htop readings aren't accurate. Best ask them really. But if I wasn't seeing any degradation, I wouldn't worry about it.

This is why (again, incidentally) "sizing" servers for Jamulus is pretty tricky unless you're comparing like for like I think. Guestimation and experimentation is probably all you can really do. And unless you are talking about big (> 20) numbers, it's almost always the case that the server is not the source of problems - at least in my experience.

[gene96817]

@gilgongo These server metrics are great to explain problems at the server. Once we know the server does not have any bottleneck causeing audio problems, the question then moves to which stream (and user node) is having problems. Since it is not usually a Jamulus client problem, there is a need to know what transport parameter is the source of the audio problems. (I am an odd duck it that, I believe it is possible to fix some transport issues.)