UDP QUIC Hole Punching Fails due to Port Reuse #3165
Comments
|
Thank you for taking the time to file this and include nice diagrams. A couple of questions:
|
Peer A: Peer B:
Unfortunately it does not appear so
A few reasons:
Yes each Peer in this case gets scheduled to a Kubernetes Node (EC2 instance) that is a part of the cluster.
No but some peers in the cluster might be scheduled behind the same NAT as the NAT is per-EC2 instance which many peers could be scheduled onto.
I did confirm that the hole punching logic is called based on the logging Added, it should be noted that I have a filter function which forwards only the addresses that will work for hole punching. I've added the relevant logs below Peer A: Peer B: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We're leveraging LibP2Ps hole punching functionality to mesh together several private networks. The network layout looks something like this
We have multiple apps running in Kubernetes clusters, each cluster has:
Below visualizes how this looks in actuality
flowchart TD subgraph Cluster1 R1["Relay Node<br>Public: 443 open<br>Private: 443 open"] P1["Peer Node 1<br>Public: no ports<br>Private: 443"] P2["Peer Node 2<br>Public: no ports<br>Private: 443"] P1 -- "register via Private:443" --> R1 P2 -- "register via Private:443" --> R1 end subgraph Cluster2 R2["Relay Node<br>Public: 443 open<br>Private: 443 open"] P3["Peer Node 1<br>Public: no ports<br>Private: 443"] P4["Peer Node 2<br>Public: no ports<br>Private: 443"] P3 -- "register via Private:443" --> R2 P4 -- "register via Private:443" --> R2 end subgraph Cluster3 R3["Relay Node<br>Public: 443 open<br>Private: 443 open"] P5["Peer Node 1<br>Public: no ports<br>Private: 443"] P6["Peer Node 2<br>Public: no ports<br>Private: 443"] P5 -- "register via Private:443" --> R3 P6 -- "register via Private:443" --> R3 endThe connection flow looks something like below:
graph TD subgraph "Initial State & Local Discovery" subgraph "Cluster A" PA1[Peer A1] PA2[Peer A2] PA1 --- PA2 end subgraph "Cluster B" PB1[Peer B1] PB2[Peer B2] PB1 --- PB2 end endgraph TD subgraph "Relay Connection Phase" subgraph "Cluster A" PA1[Peer A1] PA2[Peer A2] RA[Relay A] end subgraph "Cluster B" PB1[Peer B1] PB2[Peer B2] RB[Relay B] end PA1 --> RA PA2 --> RA RA --> RB RB --> PB1 RB --> PB2 endgraph TD subgraph "Final Direct P2P State" subgraph "Cluster A" PA1[Peer A1] PA2[Peer A2] PA1 --- PA2 end subgraph "Cluster B" PB1[Peer B1] PB2[Peer B2] PB1 --- PB2 end PA1 --- PB1 PA1 --- PB2 PA2 --- PB1 PA2 --- PB2 endThe problem we're experiencing is that during the hole punch from A1 to A2 we require the HolePunch to happen over a set range of ports since 443 does not work for hole punching.
We've ensured that LibP2P is listening on these ports and added a filter function to ensure that both peers agree on the same port range (40000-40050) however when observing the traffic leaving both peer pods it originates from port
443.graph TD subgraph "Cluster A" A1["Peer A1<br>Outbound: \"443\""] A2["Peer A2<br>Outbound: \"443\""] end subgraph "Cluster B" B1["Peer B1<br>Inbound: \"40000-40050\""] B2["Peer B2<br>Inbound: \"40000-40050\""] end A1 -- "443 -> 40000-40050" --> B1 A1 -- "443 -> 40000-40050" --> B2 A2 -- "443 -> 40000-40050" --> B1 A2 -- "443 -> 40000-40050" --> B2This means that the hole punch fails since both peers outbound ports do not match their expected inbound port.
We were able to resolve the bug by removing this particular line
go-libp2p/p2p/transport/webtransport/transport.go
Line 210 in 0fde4dc
@MarcoPolo mentioned in this issue it's worth filing a proper bug for to track as this PR seems to be the root cause
#2936
Version Information
The text was updated successfully, but these errors were encountered: