Most communication on the internet happens between an end-client and a server with an IP address accessible from the internet. Even applications with the explicit purpose of providing direct communication between clients (e.g. Slack, IRC, and the most recent Skype protocol) use a client-server architecture where the server proxies (and stores) all messages.
Two end-clients talking directly to each other (peer to peer, P2P) over the internet is not straight-forward. Due to Network Address Translation (NAT) it is not possible to simply start sending messages to each other. This describes a proof of concept implementation (p2p_broker, p2p_client) written in Rust that allows a direct connection to be set up between two clients, each behind their own NAT.
Note this is just a proof of concept and more robust solutions exist. Probably the most popular is ICE which uses both STUN and TURN1 to set up a direct connection between clients behind NAT. This proof of concept is most similar to ICE in combination with STUN.
2.1 NAT basics
Most business and residential routers will perform NAT. This means that IP addresses they assign to their LAN clients are not usable from outside of the LAN. The clients effectively share one IP and every packet going out from the router to the internet will use the same source IP2. The most common traffic happens over TCP and UDP and in those cases the router (ab)uses the source port fields3. It rewrites the source port of the packet and stores this in a NAT table:
|Original src port||New src port||LAN IP|
When the router receives a reply back it will be addressed to it's own IP, so it looks up the destination port in the NAT table. For our above example if it receives a packet addressed to port 19263 it will:
- set the destination IP address to 10.0.0.3
- set the destination port to 2456
and route the packet over the right interface.
From this it is clear that it's impossible to directly address a client from outside the LAN using only an IP address. After the router receives the packet, there is no way it can figure out who to forward the packet to.
You might think that if we could figure out the appropriate NAT table entry in the router of our destination we would be able to send packets. This is almost true, however there are some additional things that we have to take into account.
2.2 NAT variations
There are different variations of NAT4. Each type restricts who exactly can send packets that will be routed by it.
2.2.1 Source port restricted
This is the simplest type and it works as described above. When the router receives a packet it only looks at the destination port to do the reverse NAT.
2.2.2 Source port + destination address restricted
Here an additional column exists in the NAT table specifying the destination IP:
|Original src port||New src port||LAN IP||Dest IP|
Whenever the router receives a packet it will not only use
port, but also
Dest IP. Essentially it means that a client can only
receive a packet from a host if it previously sent a packet to that
2.2.3 Source port + destination address + destination port restricted
This is identical to the previous one, there's just an additional destination port column:
|Original src port||New src port||LAN IP||Dest IP||Dest port|
In this case the router will match 3 variables to do the routing:
Dest IP and
Dest port. It means that a client can only
receive a packet from a host if it previously sent a packet to that
host to the port it's now replying from.
Symmetric NAT is only used for very large networks and is thus not very common. It works differently from the other types because multiple external IPs could be used. Because of this it's sometimes impossible to traverse this NAT. Usually with this type of NAT a server not behind NAT is used to relay all traffic between clients. Depending on the exact implementation it's possible that the below solution doesn't work for this type.
Clients first register with the broker. The broker holds a list of registered clients containing their:
- user name
- source IP
- source port
This information will allow us to traverse all above described NAT types except Symmetric in some cases.
In order to allow both of them to talk to each other the following steps are executed.
3.1 Individual steps
3.1.1 Bob registers
Bob sends a
REGISTER message to the broker. This is possible because
the broker is not behind NAT. The broker will remember Bob's username,
external IP and port.
After this, Bob sends a
LIST command to see who else
registered. Unfortunately it doesn't return anything and Bob realizes
he's all alone.
3.1.2 Alice registers
Alice registers with the broker:
Just like before, the message goes through Alice's NAT. This means the broker can now reply back to Alice.
Alice sends a
LIST command to the broker and gets back Bob's
username, external IP and port.
Alice wants to set up a connection to Bob. The first thing she does is send a message to Bob using the external IP and port returned by the broker.
Doing this establishes an entry in Alice's NAT that allows incoming packets coming from Bob's external IP and port.
Note that Bob will not receive the message. When it arrives at Bob's NAT the router will drop it, as it doesn't know where to route it.
3.1.4 Ask broker to ask Bob to send a message to Alice
The last thing missing is an entry in Bob's NAT that allows incoming
packets from Alice. For this Alice will use an
ASK message to tell
the broker to ask Bob to send a message to her. Remember that the
broker can send messages directly to Bob.
3.1.5 Bob talks to Alice
When Bob receives the request from the broker to talk to Alice he sends her a message. He uses her external IP and port that were registered with the broker. The message causes an entry to be created in Bob's NAT that will allow incoming packets from Alice. The message will also successfully reach Alice, since an entry allowing packets from Bob was added before.
At this point the connection is set up and Bob and Alice can continue sending messages directly to each other.
3.2 Full flow
For completeness here's a (quite confusing) flowchart of messages, edges with the same number contain the same message.
The proposed solution successfully traverses NATs to set up a connection between two clients. It also works when clients are behind multiple NATs, as long as none of them are symmetric. The broker will use the IP and port of the closest NAT, but all messages described above would flow through all NATs. So even if there are multiple a similar type of entry is added to each one of them and the solution still works.
5 Possible improvements
For simplicity the implementations of p2p_client and p2p_broker use UDP. This way connections don't have to be managed. However for reliability reasons it would probably be better to use TCP instead. Mostly because it would avoid having to deal with situations where packets are lost or arrive out of order. The actual communication between the clients that happens afterwards could still remain UDP depending on the application.
5.2 NAT persistence
How long an entry remains in a NAT table is implementation specific. To ensure clients who registered remain reachable by the broker it could send a keep-alive message at regular intervals. This should prevent the NAT entry from being removed by the router.
Currently it's not possible for a client to unregister. This could be handled in conjunction with the above. When a broker sends a keep-alive message it could wait for a response from the client. If the client doesn't answer the broker could unregister it.
When two clients are behind the same NAT they are not able to
connect. When a packet is sent to the external IP of a router it
doesn't seem to route it back in using the normal NAT table. One way
this could be solved is by including the internal IP in the
message. When a client wants to connect to another and sees that their
external IPs match it could instead directly connect to the internal
Although TURN is not peer-to-peer, it uses a server that relays all traffic. It's usually only used when STUN doesn't work.
This is sometimes referred to as IP masquerading.
Router manufacturers have to be creative for protocols without a consistent port mapping, e.g. ICMP.