I recently wrote a post regarding some rather trouble-some TCP RSTs. Well, it turns out that this was exactly what was required to get a reasonable response from Vodacom. Ok, well, not quite, but thankfully a not quite so strategic post on the mybroadband forums got someone at Vodacom to pick up on it and get me some reasonable answers.
Specifically, this is the response I got from Vodacom (my own comments are inlined):
Good Day and thank you for your detailed query
The purpose of this response is to provide you with a status as to where we are with our investigations and how we intend moving forward towards a resolution to your problems.
Firstly we would like to allay any concerns you have around data tampering and IP address spoofing through our network. To do this we will need to sketch out a few aspects regarding how the 3G service is delivered and billed for. I will keep things on a reasonably technical level.
Your IP session becomes active on a device called a GGSN (Gateway GPRS Support Node – pretty much analogous to a NAS or RAS in old dialup terms) and passes through various elements that are used for load balancing, session setup and billing (as well as adult content filtering, but this should not impact SSH connections) thereafter it breaks out to firewalls before going out our internet pipes, where we do a certain amount of transparent caching for all web based traffic.
As well they should be. In fact, I believe that an ISP must keep a certain amount of logs by law. Not sure about what the exact requirements, musts and may nots are, but I do know that I need to keep a rather insane amount of mail logs on my mail servers.
In terms of IP spoofing, yes we do perform IP spoofing of our subscriber traffic, in a couple of places for a couple of reasons:
Firstly through a device after the GGSN called the CSG (Charging Services Gateway) your initial 3 way TCP handshake is in fact half proxy’d with your SYN spoofed by the CSG onto the destination server. This is done to maintain session awareness for billing purposes.
I personally fail to see the purpose of this, and as can be seen later on in their response, this is NOT where the ISN (Initial Sequence Number, part of the TCP protocol) modifications are taking place. I would have thought that their accounting happens in a very similar fashion to that of the ppp/radius combination, but seeing as there is no authentication actually going on (except for the fact that you need the PIN to unlock the SIM) there has to be some other mechanism. I never really did give this much thought simply because I don’t really care about this, as long as it gets done. I would still have reckoned that there is some kind of “session” between the modem and the tower though and that accounting happens on that session (again, very similar to ppp with radius).
After this handshake the remainder of the conversation is left undisturbed and passes through transparently until session shutdown whereby an RST is spoofed by the CSG back to subscribe and server.
Again I fail to see the purpose of this spoofed RST. TCP has stuff in place to terminate connections cleanly.
Secondly, spoofing occurs on our transparent cache servers on the Internet APN. Our ISP facing routers run a protocol (WCCP) with web cache infrastructure and all (apart form some exceptions) port 80 traffic is diverted to these cache servers. Firstly content is checked for previous caching and then delivered back to the subscriber or when un-cached, the cache server fetches the content on the subscriber’s behalf and delivers it to the subscriber. In order to maintain consistency and for legal requirements, the subscriber IP address is spoofed when fetching content. Also if we don’t do this all web accesses look like they are coming from one IP address (from a web server perspective) and some of them (the websites) close up shop as they see this as some sort of DDOS.
So transparent HTTP caching only happens on port 80, with exceptions – not sure which these would be. This makes sense. And their argument about the whole spoofed IP here is fine, in fact, the presence and IPs of these proxies can sometimes be obtained by sending some funky requests. One such funkyness can be to telnet to port 80 on an IP you KNOW doesn’t have port 80 open, if the connection gets established, you’ve got a transparent proxy. Getting it’s location/IP can take a bit of work (especially if it’s well written).
In terms of ISN number altering, yes we do this on our firewalls. Before breaking out to the internet subscriber traffic goes through a firewall. In short terms this is here to provide our subscribers a degree of protection. No internet side, unsolicited connection attempts are allowed though as well as proper state based monitoring of packets to pick up any out of spec behavior that could constitute a crafted attack of sorts. Various inspection algorithms are also present to ‘fix up’ the behavior of certain common protocols (such us an established FTP control session [tcp/21] requiring an incoming (internet side) session allowed for data [tcp/20]) In addition to this, there are various enhancements to prevent subscribers and servers from various forms of attack. One of these is ISN randomization:
I’ll address the ISN issue three paragraphs down. I’d like to firstly address the ftp protocol fixup issue. What is being referred to here is protocol tracking, from what I can tell. ftp specifically has two modes of operation, passive and active. In the one mode (passive iirc) the server NEVER establishes connections back to the client. This mode will function even without so called “protocol fix-ups”, active mode however requires that the ftp client requests the server to connect to a specific IP/port in order for data transfers to happen. In this case it’s required for a NAT firewall to modify the requested IP/port combination in order to not actually break the ftp protocol. So calling it a fix-up is in my opinion a tad bogus and should really be called a workaround for a brain dead protocol specification that was never designed with NAT in mind. Also, port 20 has long since been abandoned and I haven’t seen ftp-data connections actually coming from port 20 for a long, long time now (This does not mean it does not happen, it just means I haven’t used any ftp servers that does this, or I simply haven’t paid attention due to the fact that it works and I don’t have to figure out why it’s broken). On a standard (non-NAT) firewall the only reason to “inspect” the ftp control data is to “punch a hole” in the firewall for the returning data connection (create an “expect” in netfilter, or a “related” connection).
SIP is another protocol that does not take well to NAT.
Each TCP connection has two ISNs: one generated by the client and one generated by the server. The firewall randomizes the ISN of the TCP SYN passing in both the inbound and outbound directions.
Randomizing the ISN of the protected host prevents an attacker from predicting the next ISN for a new connection and potentially hijacking the new session.
More detail can be viewed on : http://tools.ietf.org/html/rfc1948
Whilst the article above clearly (albeit with a lot of technical jargon) explains what Vodacom is trying to protect against I fail to see how this measure is effective. The RFC itself already indicates that only a specific subsection of attacks is being blocked, and in this case it’s host-based authentication specifically that is being abused. Whilst certain blocks are still host-based (DNS based email blocklists anyone?) the attack described in the RFC is a blind host-based attack where the attacked needs to guess ISN information that the server is sending to the client, not the other way round. Thus the ISN randomization that Vodacom is performing with respect to the client’s connection has absolutely no bearing on the referenced RFC.
Quite the contrary actually, by making use of state-full firewalls they are actually exposing servers that are vulnerable to this kind of attacks by supplying them with a whole subnet of “gagged” (but probably/hopefully untrusted) hosts 
. I don’t care about that though – any server that is vulnerable to this DESERVES to be hacked to shreds.
This historically comes down to certain hosts having easy to guess ISN prediction functions or algorithms. It is a recommended best practice, so it is not something that we would willingly turn off.
And not something I would waste resources on.
Finally we get to your traces and the operation of SACK within this context. We have indeed found an issue with the processing of SACK packets through our firewall infrastructure. This has been raised with our vendor and we are to implement a fix/workaround in the new week. We hope this will resolve your issues with problematic TCP sessions. We will let you know once the fix has been implemented and request that you monitor your performance and provide us with feedback as to how things are working. If the problem persists we may need further traces from you and we will also look into doing further forensic traces within the internet breakout site that you use.
This should probably read “with the lack of processing of SACK options in TCP headers”.
Thanks and regards
IP Data Networks
Vodacom SA
Thanks to you too for at least clarifying the problem. May the next time I need your assistance be a lot less painful for both of us.
