Right, so Kevin (one of my staff) had the savvy to take a few tcpdump traces on both the client and the server side of a failed PPtP VPN connection over the weekend. The result? It seems the great firewall of Vodacom has yet again taken another victem.
I’m not sure whether this is a result of too little testing, total ignorance or just incompetence. Either way, it would seem it’s a bit of a race condition, and hits something similar to what we in the office refer to as the “connection tracking bit bucket”. Basically it seem most connection tracking implementations (when combined with a state full firewall such as that used by Vodacom – as per their own admission in their last letter to me) results in certain flows being prematurely marked as “invalid”. In particular in the example that Kevin has captured for me the server ends up being the first entity to send a GRE packet, this then gets (or got, seeing that it’s fixed again) intercepted by the firewall, perceived as an inbound connection to the client and the uni-directional flow gets marked as invalid. When the client now sends GRE traffic to the server this gets allowed, but the return traffic still bites the “invalid” mark. I can only speculate as to the exact state (seeing that Vodacom doesn’t reveal exactly what software they are using – probably proprietary anyway) of things, making it difficult. This I will attempt to speculate as objectively as possible (not always easy).
Seeing that there are two entities involved in this dump, and I want to do a side-by side comparison, some ASCII art is in order. Essentially three columns being used, the sending agent will indicate what is being sent, and if it was received by the destination I’ll mark that column with an ACK. I’ll also add (R) to retransmits on the sending side. The ISN mods still applies to the TCP connections, however, the data itself isn’t being tampered with in this case. Note that in the GRE traffic case there are still ACK packets being sent by the server, these ACK packets however goes lost (as indicated in the packet sequence).
Client Direction Server
SYN -> ACK
ACK <- SYN
PPTP (Start Req) -> ACK
ACK <- PPTP (Start Resp)
<- GRE (PPP-LCP Conf Req)
GRE (PPP-LCP Conf Req) (R) -> ACK (goes lost)
GRE (PPP-LCP Conf Req) (R) -> ACK (goes lost)
... a few more of these ...
GRE (PPP-LCP Conf Req) (R) -> ACK (goes lost)
PPTP (Call Clear Req) -> ACKOnce the Call Clear Req is received TCP/IP teardowns happens, surprisingly without the flurry of injected RST packets I’ve growned accustomed to, just a single out-of-order delivery between one ACK and FIN/ACK packet.
What I would want (not sure what resolution they picked) is for them to either perform a routine inspection of the PPTP control traffic (specifically the Start Request and Start Reply packets) to determine the GRE traffic parameters (based on what I can see, just mark the fact that GRE is to be expected between the two given end points) and allow that traffic, or, stop this firewalling nonsense. It’s only the Cellular “ISPs” performing actions such as these. The arguments for providing this protection is sound. But then it needs to be done sanely. For the most part I’ll have to admit that the firewall works and doesn’t cause too many problems.
Seeing that the problem has been resolved by Vodacom, I’ll let it rest, for now.
Is there any decent TCP testers? It will make debugging (mostly firewall caused) TCP-issues a lot easier… (Cisco firewalls also mess around with TCP flags, which breaks some applications… They also mess around with the ISN by default)
It should be able to test most TCP features (transfers, sequence numbers, urgent flag, common extensions) and give you an idea of what the network allows and what not…
Back to 3G: I wonder if their “internet conenctions” actually work properly for non-TCP / UDP-based protocols… (such as SCTP-based protocols…)
I always just use tcpdump. The number of these problems I have are far and few between, and vary so much in nature that I believe it’ll be difficult to write a tool to reliably test for all possible things that can go wrong. In terms of probing you can probably use nmap to see what’s open and what not.
The more interesting part as you suggest is the question surrounding non-TCP and non-UDP. I know that GRE works in combination with PPTP (as per above, when they don’t break it). But other protocols may not work properly, this remains to be seen and I can’t really comment. But brings me back to elementary networking … the iso networking layer model. You have been given an IP right? Which is routable? So why does the ISP feel it’s required to break the iso layer by looking at stuff anything higher than layer 3? Why do they feel it’s required to look at anything inside the IP packets when that is all they actually need to look at in order to deliver traffic?