Strange behaviour trying to connect to a server

[pljones]

Background

• I'm running the Rock directory server on my LAN - LAN1 - on machine 1A on port 22424.
• Someone else is running a Jamulus server on their LAN2 on machine 2Z on port 22124.
• They are also running the Jamulus client on their LAN2 on machine 2Z. (Actually two client instances.)
• A third person, on neither LAN1 nor LAN2 can happily see the server on machine 2Z listed in the directory and connect.
• Jamulus Explorer can also see everyone connected to the server.

Problem

• I'm also running the client on a second machine on my LAN1, machine 1B, default port 22124.
• I cannot see the server on machine 2Z from machine 1B in the Rock directory.
• I can see other servers from machine 1B in the Rock directory and can connect to them.
• I cannot connect to machine 2Z by IP address listed in Jamulus Explorer.
• ping gives me

--- XX.6.184.XX ping statistics ---
9 packets transmitted, 9 received, 0% packet loss, time 8007ms

so there isn't a ICMP-layer network routing problem.

Anyone any idea why one server would act differently from the others?

[gene96817]

8007ms (8.007 seconds) is. a long time. Is this the ping from 1B to 2Z?
Q1: What is the ping time from 2Z to 1A? I nice problem would show symmetric ping times.
Q2: what ping times do you see from 1B to the other servers? I hope most of the ping times are <100ms.
The farthest servers from me are <400ms (it could momentarily be <410ms).
Q3: what does tracert show? It would be highly suggestive if there is a hop that causes most. of the unusual delay.
At 8007ms, I would expect some timeouts to be happening.

If the 8007 ms is a typo, then I would start with confirming the (bad) behavior is symmetric between two machines.
If the. 8007 ms a real measure, I hope tracert would show some interesting measurements.

[pljones]

That's odd...

64 bytes from XX.6.184.XXX: icmp_seq=9 ttl=51 time=22.8 ms
^C
--- XX.6.184.XXX ping statistics ---
9 packets transmitted, 9 received, 0% packet loss, time 8008ms
rtt min/avg/max/mdev = 21.407/22.211/24.220/0.820 ms

No idea what the time 8008ms means. But meaningless, apparently. (If it includes the sleep between pings, it's really meaningless! Running with time gives real 0m8.256s, so it looks like that 8008ms is, indeed, wallclock...)

[gene96817]

ah... sorry I didn't read it carefully.
Your ping times are between 21.4 and 24.2 ms with a mean value of 22.2.
That seems a perfectly normal range. Usually ping will pause between each ping sample.

If your problem is in the network, it is something else. I suggest running tracert from 2Z-to-A1 and A1-to-2Z.
My prediction: if the results are symmetrical, the problem has disappeared.

[pljones]

Further diagnosis isn't possible currently as the server isn't online (according to Jamulus Explorer).

[softins]

Does your Genre Rock server use --serverpublicip? Do you know if the server on Machine 2Z uses --serverpublicip?
Presumably both the LANs mentioned are behind NAT. What kind of router do you have?
(just trying to gather complete information at this stage).

[pljones]

Genre Rock doesn't (it shouldn't need to). (The only machine I have that uses --serverpublicip is my "wip" (work in progress) server, that registers with jamulusdirectory8.drealm.info:22824, which is on the same public IP and same internal IP.) All the configs are here.

Our PlusNet One router is fairly dumb and the server port allocations are straight through externalIP:portNN <-> internalIP:portNN, always using the same portNN external and internal. The ports in use are 22X24 ports 1 to 8 (so no clash with client, then), plus 5XXX4 ports for various XXX values (again, well out of harm's way).

I can't imagine Machine 2Z uses --serverpublicip -- it wouldn't have a reason to. I don't know what the Virgin network routers are like. Machine 2Z is running MacOS, if it makes any difference. (And my box is Linux.)

[softins]

OK, will need to think it through.

[softins]

• I'm also running the client on a second machine on my LAN1, machine 1B, default port 22124.

A client doesn't default to 22124, but rather to 22134 and then trying again with successive ports until it succeeds in binding.

[pljones]

🤷‍♂️ True...

[softins]

• I cannot see the server on machine 2Z from machine 1B in the Rock directory.

That server will of course be in the list that the Rock directory sent to machine 1B. But without using --showallservers, the client will not display it in the list until it has successfully pinged it using Jamulus protocol. It suggests that either the ping or the response is not getting through as expected.

It might be informative to try running Wireshark on machine 1B to observe the Jamulus traffic.

[softins]

Disclaimer: I did not look at the Jamulus code for that which I'm about to talk, because I'm not certain where to look. However, I think the code for punching through firewalls may work something like this:

• remote public server connects to remote' directory server.
• client connects to remote' directory server and uses info to ping remote public server.
• ping fails or succeeds.

The solution may be for the client to send a ping request message to the remote' directory server that is passed to the remote public server. Thus, the client will wait for a ping from the remote public server and will use the new ipaddr:port for communicating (maybe different NAT:port). The client or servers may issue a warning.

There is more to it than that. What actually happens is this:

• Remote server (maybe behind a NAT or a firewall) registers with a Directory Server. It repeats its registration every so often to keep the session open to the Directory Server, so that the latter can send messages back to it.
• A client (also maybe behind a NAT or a firewall) sends a server list request to the Directory Server.
• The Directory Server sends back the list to the client.
• At the same time, the Directory Server itself sends a request to each server in the list, asking it to send an empty Jamulus message to the IP and port of the above client. It does not matter whether the empty message reaches the client or not. Its purpose is to establish a session from each server to the client in the server’s NAT or firewall. That will enable it to receive a direct Ping message from the client.
• The client sends a Jamulus ping to each server in the list, and should receive a reply to this ping, from which it can calculate the ping time.
• Once the client has received a ping response, it adds the server and its ping time to the connect window.
• The user can then choose to connect to one of the listed servers.
• If the client was started with --showallservers, all the servers returned in the list are displayed immediately, even if they have not responded to the ping. Trying to connect to one without a ping time will likely fail.

[ghost]

To spare myself some thinking, I'll ask for your opinion on:

• At the same time, the Directory Server itself sends a request to each server in the list ...

Maybe each server in the list should send the 'empty request' several times to guarantee that it doesn't get dropped for any reason (including multiple firewalls).

[pljones]

At the same time, the Directory Server itself sends a request to each server in the list, asking it to send an empty Jamulus message to the IP and port of the above client.

This is the "special case" here -- I've only one external IP address, the same for my client and the directory server.

It does not matter whether the empty message reaches the client or not. Its purpose is to establish a session from each server to the client in the server’s NAT or firewall. That will enable it to receive a direct Ping message from the client.

And that's not working here, for some reason - it's not arriving and no session can be established. Presumably the request to send the empty message arrives at the server but somewhere the second UDP excursion to the same IP -- but on a different port -- doesn't register.

Given that it's appearing to be server OS related - see #1952 (comment) - I don't think it can be something at my end.

What I'm wondering now, of course, is what can be done - either during server install on MacOS and Windows, or when they get a new connection from the same IP as the directory with which they're registered. I don't know either OS, unfortunately.

[softins]

To spare myself some thinking, I'll ask for your opinion on:

• At the same time, the Directory Server itself sends a request to each server in the list ...

Maybe each server in the list should send the 'empty request' several times to guarantee that it doesn't get dropped for any reason (including multiple firewalls).

That isn’t needed. It makes no difference whether this empty message reaches the client or not; the client is not waiting for it, and many clients that are behind NAT or a firewall will never see it. The only purpose of the empty message is to open a NAT/firewall session at the server end, ready to allow the pings that will come from the client. A single empty message is sufficient to do that.

[softins]

What I'm wondering now, of course, is what can be done - either during server install on MacOS and Windows, or when they get a new connection from the same IP as the directory with which they're registered. I don't know either OS, unfortunately.

It’s been a busy few days, and I haven’t yet had time to map out the possible scenario(s). It ought to be possible to come up with an understanding of why it is happening. (He says...)

[pljones]

OK, I was wrong. It's not just this one.

It seems to be prevelantly MacOS and Windows machines. As in all MacOS and Windows machine, plus a few more.

What I think may be happening is that the OS firewall isn't happy with two "intrusions" from the same IP (i.e. my external IP) on different source ports - one the directory, one my client. I don't know why, though.

Sort of ties in with what David is saying.

[pljones]

That sounds like a different situation from what I was seeing, if what you're supposing above is correct, with the port ransomisation being the problem, that is. On that basis, I'd expect L always to have the problem - if it starts with -p 58124, the router will establish L to D as using 55124 and then mean any other outbound session from that server would get a different port, preventing Jamulus access.

In the case I was seeing, #1952 (comment), changing the port fixed the problem. So the server - say S -- when on -p 22124 showed with split ports on Jamulus Explorer and couldn't be seen but worked on -58124. (We tried restarting and switching directory servers.)

However, it was sometimes visible on this port to some clients (just never to me, where C and D had the same IP address). We'll have to see if changing the port to -p 58124 is a permanent fix once there's more than two clients on.

Fundamentally, it relies on non-standardised operations within the router to get the session. It sounds like there are potentially multiple different ways for it not to work. That then pushes even more for recommending manually configuring the router for the desired port.

[softins]

That sounds like a different situation from what I was seeing

Oh yes, very likely. My explanation was more aimed at clarifying to @gene96817 the general operation and how it would apply to the Lola example.

[gene96817]

@softins @pljones Thank you for all the details. There are some surprises to me.

From 3 days ago,

In the Lalo example, the Directory server does not know about the 1034, so is unable to tell the requesting client about it. All it can tell the client is the 22124 from which Lalo registered to the Directory.

Yet the Directory server table shows a contradiction to step 3

Lalo | 209 | 186.107.55.5 | (1034) 22124

4. At the same time D sends a request to each server in the list, asking it to send an empty message to the IP address and port from which C's request to the server came.

What is the purpose for D to ask each listed server to send an empty message to C? Is this the step that populates the ping times in C's directory table? In general conversation, it is usually implied that C will ping the servers in the table received from D to populate the ping times.

6. L sends an empty message from source port 22124, to C's IP address and port that was set up in 2 above....

At this step, it implies all the registered servers at D will do this step. After all at this step, L has no idea that C is interested in a connection. Between step 4 and 6, it seems there are a lot of unnecessary messages.

8. C now starts to ping each server in the list. ...

This step seems to confirm step 4 is unnecessary as well as to confirm step 7 is a functional NOP.

Step 9 seems to be the crucial step with the lack of clarity of port 1034 in this example or in general why the Directory server would have a port listing of the form (nnn)mmm

10. ... If it finds multiple servers on one IP address (very rare in this situation), it cannot tell which one has had its port mapped.
    ...

Seems to me there are two issues:
(a) Server S at xxx:p1 went down and restarted. Depending on the changes, S could now be using xxx:p1 or xxx:p2. This is really a coordination betyween S and the local NAT function
(b) Does anyone operate two Jamulus servers from a private network through a router? Seems the only way to make this work reliably is for all the ports to be assigned. This also requires D to associate each S to IP:port pairs and not just IP addresses.

@pljones I am not able to interpret completely what you are describing, especially "port randomization". I am more familiar with services that assign each new session to an incremented port number. Everything you describe loosely relates to a server or client on a private LAN getting out of sync with the router/NAT port mappings. This is especially true with

However, it was sometimes visible on this port to some clients (just never to me, where C and D had the same IP address). We'll have to see if changing the port to -p 58124 is a permanent fix once there's more than two clients on.

In this case, C and D have the same public IP address and different private IP addresses. Since I have only one server on my private network, I put a fixed route in my server that maps the public IP to the server's private IP. In that way, when my client sends a Jamulus message to the server's public IP address, the router delivers it to the server's private IP address. These are all IPv4 NAT problems.

Some day, when we all move to IPv6, these problems will be moot.

[softins]

@softins @pljones Thank you for all the details. There are some surprises to me.

From 3 days ago,

In the Lalo example, the Directory server does not know about the 1034, so is unable to tell the requesting client about it. All it can tell the client is the 22124 from which Lalo registered to the Directory.

Yet the Directory server table shows a contradiction to step 3

Lalo | 209 | 186.107.55.5 | (1034) 22124

No, in that data, the 1034 did not come from the Directory Server's list. It was deduced by the Jamulus Explorer back-end observing that the empty message came from an unexpected port.

1. At the same time D sends a request to each server in the list, asking it to send an empty message to the IP address and port from which C's request to the server came.

What is the purpose for D to ask each listed server to send an empty message to C? Is this the step that populates the ping times in C's directory table? In general conversation, it is usually implied that C will ping the servers in the table received from D to populate the ping times.

No, the sole purpose of this step is to create a session in each server's firewall/router (e.g. L's), ready to accept the subsequent ping messages from the client C. It doesn't matter that the empty message is unlikely to make it all the way to C, so long as it has the desired effect in L's firewall/router. The client at C isn't even looking out for this message; Jamulus Explorer is unusual in that it does look out for it, for diagnostic purposes.

1. L sends an empty message from source port 22124, to C's IP address and port that was set up in 2 above....

At this step, it implies all the registered servers at D will do this step. After all at this step, L has no idea that C is interested in a connection. Between step 4 and 6, it seems there are a lot of unnecessary messages.

Yes, all the servers will do this step, as it may be the only way their firewalls will be prepared to accept the client's ping messages. The Connect dialog in Jamulus does ping all the servers in the list, so that it can display the ping times to the user.

1. C now starts to ping each server in the list. ...

This step seems to confirm step 4 is unnecessary as well as to confirm step 7 is a functional NOP.

Nope, without steps 4 and 7, many server's firewalls would not allow the pings through, if they did not have explicit port forwarding set up. A lot of Jamulus users running a server on their own PC may well not know how to do that. That is why the technique was added to Jamulus (long before my involvement!).

Step 9 seems to be the crucial step with the lack of clarity of port 1034 in this example or in general why the Directory server would have a port listing of the form (nnn)mmm

The (nnn) part is deduced by the Explorer back-end (which I wrote), not supplied by the directory server.

(b) Does anyone operate two Jamulus servers from a private network through a router?

@pljones does, but with explicit port forwarding, not relying on the magic I described above.

Some day, when we all move to IPv6, these problems will be moot.

Not necessarily. Even without NAT, systems will still have firewalls with dynamic sessions to allow only packets that are part of established sessions.

[gene96817]

@softins OMG, Jamulus communications is a massively clever hack. The scheme works well.
I suggest our documentation provide explicit advice on
(a) the routing table entry that is needed when a client and server are in the same private network, and
(b) how to handle more than one server in the same private network.

Thank you for all your explanations.

[pljones]

So I spent some time today with the server operator.

Initially he started the server up on Any Genre 3. I noticed it came up with "split ports" -- (nn) MM, which I immediately felt shouldn't happen.

The server is relying on "magic hole punch" (as I call it) for convincing the router to allow inbound traffic. I didn't see this (or, at least, don't remember seeing it), previously. The server was being run with -p 22124 explicitly stated. Jamulus Explorer had that as the primary port (MM) with a low numbered secondary port (nn). I checked and the server operator hadn't set anything on their router for forwarding, as expected.

My thinking was that "22124" might be getting "clogged" at the router - i.e. something other than the server had already claimed it.

So I suggested trying a high numbered port, like 58124. This worked -- Jamulus Explorer showed just the specified value and I could then connect to the server - still on Any Genre 3.

Then we tried on one of my directory servers - and that worked, too.

So it looks like "magic hole punch" has some issues around port allocation at the router. (It's not something I've ever been keen on... so my first reaction is just to drop it...) (When the router has been configured manually with an allocated port assigned to Jamulus Server there won't be an issue, I expect.)

[gene96817]

What I observed (some months ago when I was having problems) that when the server starts up it will get the requested port (e.g. 22124) if it is available. If the firewall and server gets out of sync, on restarting the Jamulus server, 22124 will be busy and the firewall will give the server another port. This is when I see "split ports". If you clear the port assignment on the firewall and then restart Jamulus, the problem will go away. The brute force simple way is shutdown your Jamulus server (the OS too) then reboot your router/firewall to clear all the port assignments. (The OS firewall has a role too but I don't know how different OS manage the port assignments.) Then a clean restart of the OS and Jamulus will get the port assignments the way you expect.

[pljones]

Yes, I'd imagined the split was something like that. It's a complex issue -- which is why I feel the "simple" approach is to say "set your router correctly" rather than trying to do the magic hole punch like this. It's easier to say "Find your LAN IP address, pick a LAN port number, tell your router to forward all requests it gets on that port number to that internal IP". That way, the router will not allocate that port number to other things.

[gene96817]

I agree. "magic anything" is hope and pray process. My recommendation is (a) if your router/firewall supports a DMZ, put your jamulus server in the DMZ, alternatively (b) assign the ports needed by Jamulus in your NAT (the same ones we recommend for the Linux OS). Then the port usage will be predictable. (Scenario (b) is what @pljones prescribes.) If you have more than one physical server on your private LAN, you will be managing a lot of port mapping.

[ghost]

I use the DMZ for experimenting and have always found a non-DMZ solution in the end. Jamulus should not need to be in the DMZ, if it does then I would start blaming everything that gets in my way.

[gene96817]

The point is if you only have one server that needs to receive connections, a DMZ is an easy way to map inbound traffic to your server. You have more control over the firewall/NAT to assign individual ports. I usually chose the option that is less work. (Ironically, my new router doesn't support a DMZ.)