Denial of Service attacks is a very vague term for a group of attacks that causes a service to be denied to legitimate users. Obviously when you host your website publicly you want it to be available – so having it taken down by some hacker is highly annoying. And I’m not referring to only websites, there are many other types of applications and services out there that can be “taken down”, or to which “service can be denied”.
First up, please note that whilst I do consider myself to have a reasonable knowledge of the concepts described here I by no means consider myself to be an expert in this particular field. I explain here how I see things, which is my opinion, and may well differ with the opinions of others. Most of this discussion also revolves around experience, and not around a whole lot of reading up on what others say.
The term “Denial of Service” (DoS) refers to any kind of attack with the aim of making a service unavailable to legitimate users. In my experience there are numerous ways to achieve this, but generally they all aim to exhaust some kind of resource on which we service relies. A few examples:
- Network bandwidth
- TCP socket resources
- RAM
- Disk
Disk and RAM is difficult because we need to attack those indirectly, we can’t trivially just drop loads and loads of files onto the victims drives, and we can’t magically fill up his system RAM. The obvious way of filling up the disk is to cause the application to generate massive amounts of logs – which is easier said than done in most cases. RAM is even trickier as we need to keep that memory consumption resident.
The obvious targets then remain: network resources.
Both TCP resource exhaustion and sheer bandwidth exhaustion is perfection viable targets. Consider a 5 line highway with a number of off-ramps. Off-ramps are generally much smaller than the highway and cannot handle the same amount of traffic. The Internet backbone and the servers attached to that function in a similar manner. The far majority of ISPs has more “back bone” bandwidth available than the switches and servers linking onto that. In an extreme example an ISP could have one or more 10Gbit peerings, with 10G capacity internally on it’s network, with server farms running on 1G links into that, or as in a recent case I’ve seen, the last layer of switches being 10/100Mbps switches.
A 10/100Mbps switch will not deal with 8Gbit worth of traffic being sent to one of the servers attached to it. You will see around 98.8% packet loss, depending on packet sizes and other factors. It’s a simple fact, if you have five lanes of cars suddenly wanting (needing) to get off onto a single-lane off-ramp they are going to get stuck. The Internet’s way of dealing with that is to discard packets. For this it doesn’t matter whether the traffic is legitimate or not, what matters is how much data we can send at the victim – the more the better.
The other option is to flood the server’s TCP stack. Consider a switchboard with 4 incoming PRI channels (which carries 30 voice channels each). It’s fine if there is up to 120 incoming calls, but what happens with call number 121? It gets busy. So all that an attacker needs to do is consume those 120 calls … and keep consuming them – thus preventing other users from phoning in.
In a similar fashion servers have finite resources. Some times it’s application specific (eg, apache has a setting called MaxClients), in other cases it’s dependent on the operating system. Unlike the telephone example where all 120 channels can ring simultaneously servers also has a limited number of calls (for example 10) that can be in the “ringing” state, any additional calls coming in after that is just assumed that it won’t make it and is simply discarded. From a technical perspective http://veithen.github.io/2014/01/01/how-tcp-backlog-works-in-linux.html explains it quite well, but it’s generally safe to say that we can have 128 “ringing” connections, and usually no more than a few thousand “active” (established) connections. So if the attacker can either fill up the ringers, or the established (or both) then access to the application is also denied.
To defend against these actually turns out to be rather difficult. And technical. The former is easier to explain, the latter gets quite technical.
In the former case (sheer amount of traffic) there is a few things we need:
- More bandwidth than the attacker, as far out from our servers as possible. So in the ISP example, if the attacker has 100G of bandwidth available and it comes at the 10G link, after which the ISP can pick it up – it’s too late. We need more bandwidth than the attacker.
- We need the ability to filter legit traffic from traffic we don’t want, as far as possible away from the server. As it turns out this problem is difficult to solve. Essentially the simplest mechanism is to implement stateless firewalls as far out to the edge as possible. For example, if we don’t care about UDP traffic, have the ISP drop it as soon as it receives the traffic at it’s peering. The problem with this is it gets very expensive very quickly in terms of resource requirements. It’s not unusual for ISPs to have a few million IP addresses on their networks, and to replicate firewall settings all over is not only a managerial nightmare, at each point you also need some very, very expensive hardware that’s capable of applying your (very, very big) set of firewall rules at rather insane speeds. Keep in mind we want to keep latency very low, and throughput at a maximum. We sacrifice on both of these when a firewall comes into play. Not to even mention the sheer resource requirements in terms of RAM and CPU to perform the matching.
Even when that’s all done, what if I send traffic that seems to be legit? For example, to get a page from a website I need to connect to port 80 … in a stateless firewall I can’t keep state as to things like sequence numbers etc … so if the attacker just sends random packets at port 80 we still can’t filter everything out. It’s probably possible to make some guesses in the TCP case with respect to sender sequence numbers (ie, allow a range), and normally a connection starts with a SYN frame, but we don’t *know* that in the Internet case (two drivers with the same origin and destination could travel along different routes – the Internet is no different).
So even if we have edge protection in place, we are still not safe. We will still need to apply some form of shaping as the bandwidth goes down to the server. We simply cannot get 10Gbps of bandwidth over a 1Gbps (not to mention 100Mbps) link. Performing this filtering is expensive.
The other scenario where we exhaust the ringer queue is something which the operating system can defend against, assuming that the SYN packets (the initial packet that one computer sends to another to let it know that it wants to talk) actually reach the server. Firstly, we can rate-limit these SYN packets based on the source-IP (if we pair up the source and destination IPs we can already rate limit further out, doesn’t have to be at the server) since it’s very unlikely that we’ll get hundreds and hundreds of connections from the same IP in a short time frame. Based on absolutely no investigation, I’ve guessed at a rate of 1 SYN packet/second, with a max burst of for example 10. So far this seems to work quite well, and I haven’t seen a case under “normal” operation where we’ve exceeded that yet (only been doing this for a few hours at this stage).
That in itself is not good enough since the source address can be spoofed and the server still needs to add an entry into the accept queue (refer to the link above). So by spoofing hundreds of SYN frames from different IPs we need to somehow distinguish which ones are valid and which ones not.
The first defence against this spoofing is for ISPs to enforce RPF on their access links. At a minimum this forces the attacker to reveal the source of his attack (well, at least the IPs of the machines performing the actual traffic generation), at best it drops spoofed packets resulting in us not having to deal with them at all.
The other, in the case of TCP is something known SYN cookies. In simple terms, normally when we receive a SYN packet, we store some data locally, and respond with a SYN/ACK, then wait for the ACK (process is well explained in the first link above). What SYN cookies does is when that queue fills up we respond with a specially crafted initial sequence number in the SYN/ACK, as a result of which we do not need to store any data locally. In other words, there is no longer a queue to overflow. In order to exhaust the TCP stack now we have to establish the connection fully, taking it to the established state. This is extremely difficult unless you reveal the source addresses, and with various timeouts etc the duration of each “attack” is very limited, and since we know the source of the attack we can switch to filtering it (and if we do this intelligently we can filter it automatically, for example, if we log http connections that simply opened and then timed out we can based on heuristics decide to shorten the time out based on the speed of those incoming connections, or even block them completely without even going through the TCP handshake).
What about operating systems without SYN cookie protection? I don’t know about operating systems other than Linux, but the question has been raised to me. And the idea that came forth was that it should be possible to place a device in front of that OS, that splices into the connections, similar to a transparent http proxy, except that this one proxies raw tcp connections. Doing this in userspace is quite easy (I’ve already implemented transparent https proxies that performs logging of domain names involved in the ssl connections based on the certificates involved, without modifying the data stream at all – source port numbers are affected). The problem is that we add latency, and degrade service. So the idea would be to:
- Rate-limit SYN packets to 1/second, burst 10 on a per SourceIP+DestinationIP basis (easily do-able using iptables hashlimit).
- Rate-limit simple forwarding of SYN packet to for example 50/second with a burst of accept queue-size, and to redirect to the tcp proxy when the syn rate exceeds this.
That way we provide the device with some base protection to begin with, and when we start seeing those kinds of crazy rates of incoming SYN packets, probably being indicative of a SYN flood denial of service, in which case I’d rather degrade service that provide no service. As a friend of mine always say: Some service is better than no service.
We could also implement something similar for sheer connection counts, but that will take some more thinking.
And those are my thoughts on Denial of Service attacks currently.