Servers on custom central server not visible for some clients

[DominikSchaller]

Describe the bug
In the last two weeks we have some clients which cannot see any server on our custom central server.

To Reproduce
Use this central server: jamulus.herrenbesuch.net
Open Connection dialog and select Custom list.

Expected behavior
Every client should be able to see the servers.

Version of Jamulus
The server version is a patched 3.6.0 server with some local changes.
It seems like all problematic clients are on 3.7.0. But it works for some clients with 3.7.0.
I wasn't able to get more information about the problematic clients yet. It works for me, but I'm still on the 3.6.2 client. Maybe you guys can try to connect to the server and see if there is any problem?

[hoffie]

Works for me with both 3.6.2 and current (post-3.7.0) master.

Is there any evidence that it works with 3.6.2 reliably and stops working with 3.7.0?

Traffic captures would always help, but could be hard to get from non-technical people. Maybe you could get at least one from the server side?

[DominikSchaller]

Thanks, I will ask them for more details. It happened last week on another server I manage, and today it happened in our rehearsal but I didn't want to spend too much rehearsal time investigating the problem. I just gave them the IP:port and had them type it in directly.

It worked reliable since two weeks ago, updating the clients from 3.6.2 to 3.7.0 is the only thing we changed.

[hoffie]

They may also hit what is described here under "Not seeing a list of servers at all?": https://jamulus.io/wiki/Client-Troubleshooting

[DominikSchaller]

@hoffie What do you mean by traffic captures? Can I get these after the fact (log files)? Or can you tell me what I should prepare for the next test with these guys?

[hoffie]

I would be interested in tcpdump/Wireshark captures (.pcap). It should be sufficient to limit this to the protocol message stuff, which should keep them small.

There are some examples here: https://github.com/softins/jamulus-wireshark/blob/593bf494119df13a0d77512ffb238c90b32b41f5/tcpdump/capture-jamulus-proto.sh

Sadly, I don't think there would be anything helpful in the logs.

With a server-side capture we can tell if the server list request reaches the server at all (I'd think so) and if and how the server replies.
With an (additional?) client-side capture we can tell if and what the client side receives. I suspect that the reply may go missing due to fragmentation, intrusion prevention systems or similar things.

[dcorson-ticino-com]

I have seen this also with the standard directory servers.
I have given all our people the server IP just in case.
I think this has something to do with the vagarities of the Internet.
If you leave the dialog open long enough (at times more than 10 minutes) the servers do show up.
I have actually thought that things are better with 3.7.0 as I haven't seen this happening for the last couple of weeks. Difficult to debug as from one minute to another it can work or not.

[DominikSchaller]

How is it possible that Jamulus can send UDP packages to these clients, but it is not possible to make sure that the servers are visible? Is a different protocol used? And if yes, can we change it? It's weird that this is an issue for a lot of users ... today I saw this on facebook: https://www.facebook.com/groups/619274602254947/permalink/888323272016744/

[dcorson-ticino-com]

It works like this:
When a client opens the connect window the selected directory server is sent a message that the client requests the list of servers the directory manages. If the client does not receive a response within a certain time it requests again until it does. When the client receives the server list it pings all the servers on the list to get their ping times. etc.

This is all done with Jamulus' UDP protocol. If for some reason the directory server is overloaded or the Internet is slowing or disallowing UDP packets between you and the directory you are out of luck. As the communication is between client and directory server it is different for each user. As we all know there is nothing guaranteed on the Internet.

I don't think that there is anything particular happening, it is just that the directory servers are positioned critically for us and there is no redundancy, either on computing locations or routing.
For the moment I think the best thing to do to try to guarantee connections is to have everyone put the IP of your server in their Server Address field. It will stay there and be available in case of need. Not really great, but a viable workaround.

[gene96817]

One option is to make the directory server message to clients more tolerant of UDP message loss. E.g. use only short UDP messages and enumarate the messages for selective retransmission. This change will also reduce UDP traffic.