jkroon » Networking

Cell C following in the footsteps of Vodacom?

Jaco Kroon — Sun, 04 Jul 2010 18:52:48 +0000

Most people that know me well will know that I really don’t like the way Vodacom runs their firewalls for their 3G consumers. In fact, they’ve managed to make it onto my blog no less than 3 times now – and not once for anything they’ve done right. And now Cell C have decided to join the crowd of braindead arseholes who can’t run firewalls. I present to you the man-in-the-middle TCP connection reset. As it stands right now I can’t ssh. I can’t connect to my jabber server. I can’t even browse. At least, not using my Cell C internet connection.

UPDATE: Please note that Cell C has already contacted me regarding this. See comment #1 below for more details.

Unfortunately it’s insanely hard to prove conclusively where the TCP resets are coming from, again the only evidence I’ve got that it has to be Cell C is the fact that it works flawlessly from everywhere else (SAIX ADSL, Mweb ADSL and Vodacom 3G). So the first things I started noticing yesterday was ssh connections going something down these lines (serenity is my local machine, linux.delter.co.za a relatively big mail server from one of my clients):

jkroon@serenity ~ $ ssh root@linux.delter.co.za 
ssh_exchange_identification: read: Connection reset by peer

Now my employees knows, if I can’t ssh and it’s your fault, you’re going to get it. Firstly I will hunt you down, then I will do things which cannot be considered polite, and if you’re name is bigger than mine and my client believes that because your name is bigger than mine that implies you’re right and I’m wrong I will ensure that I prove them wrong and make very sure that they understand that I don’t take these things lightly. Well, not when it affects my work anyway, but I do understand if things breaks periodically, but at the moment I can’t even browse and in excess of 95 % of the connections I’m pushing out over my Cell C SIM is outright being reset.

So after seeing the above for for approximately 3 out of 5 connections this morning whilst sitting in a data center in johannesburg I just ran a tcpdump in a different shell on serenity to see what happens:

08:53:56.835503 IP 196.35.70.139.ssh > 10.213.51.133.47019:
    Flags [R.], seq 2902460094, ack 3806794395, win 199,
    options [nop,nop,TS val 26312228 ecr 4876698], length 0

Now I KNOW the way I set up my servers. And if my server is in fact generating that RST then there is something severely broken. But I’m getting this from different servers. So, having had one of the craziest weekends for a while going on I decided to push this to the back of my mind and concentrate on more urgent matters. It’s only about 30 minutes back that I wanted to quickly check mail, browse a bit and just unwind a little that I couldn’t actually browse, ssh to my servers for a quick checkup after the weekend’s events and write an official complaint to a certain hosting company that I decided enough is enough. Got a jump box and ssh’ed via another route to linux.delter.co.za (and no surprises) it worked flawlessly. Fire up pppd and add a route for linux.delter.co.za over that, fire up tcpdump on both ends and I get this, first on serenity (sorry for the horizontal scrolling, and also note that the time on my laptop is out by ~30 minutes due to ntp failing and the CMOS on this Lenovo being of the ultra crappy kind):

18:46:14.706636 IP 10.212.100.200.46247 > 196.35.70.139.ssh: Flags [S], seq 8197450, win 5840, options [mss 1460,sackOK,TS val 187563 ecr 0,nop,wscale 7], length 0
18:46:15.326350 IP 196.35.70.139.ssh > 10.212.100.200.46247: Flags [S.], seq 1497501202, ack 8197451, win 5792, options [mss 1460,sackOK,TS val 2837150 ecr 187563,nop,wscale 6], length 0
18:46:15.326445 IP 10.212.100.200.46247 > 196.35.70.139.ssh: Flags [.], ack 1, win 46, options [nop,nop,TS val 187626 ecr 2837150], length 0
18:46:15.659155 IP 196.35.70.139.ssh > 10.212.100.200.46247: Flags [P.], seq 1:21, ack 1, win 91, options [nop,nop,TS val 2837190 ecr 187626], length 20
18:46:15.659267 IP 10.212.100.200.46247 > 196.35.70.139.ssh: Flags [.], ack 21, win 46, options [nop,nop,TS val 187659 ecr 2837190], length 0
18:46:15.659458 IP 10.212.100.200.46247 > 196.35.70.139.ssh: Flags [P.], seq 1:22, ack 21, win 46, options [nop,nop,TS val 187659 ecr 2837190], length 21
18:46:15.969153 IP 196.35.70.139.ssh > 10.212.100.200.46247: Flags [.], ack 22, win 91, options [nop,nop,TS val 2837221 ecr 187659], length 0
18:46:15.969221 IP 10.212.100.200.46247 > 196.35.70.139.ssh: Flags [P.], seq 22:814, ack 21, win 46, options [nop,nop,TS val 187690 ecr 2837221], length 792
18:46:16.349157 IP 196.35.70.139.ssh > 10.212.100.200.46247: Flags [P.], seq 21:805, ack 22, win 91, options [nop,nop,TS val 2837221 ecr 187659], length 784
18:46:16.386434 IP 10.212.100.200.46247 > 196.35.70.139.ssh: Flags [.], ack 805, win 58, options [nop,nop,TS val 187732 ecr 2837221], length 0
18:46:16.599149 IP 196.35.70.139.ssh > 10.212.100.200.46247: Flags [.], ack 814, win 116, options [nop,nop,TS val 2837284 ecr 187690], length 0
18:46:16.599227 IP 10.212.100.200.46247 > 196.35.70.139.ssh: Flags [P.], seq 814:838, ack 805, win 58, options [nop,nop,TS val 187753 ecr 2837284], length 24
18:46:16.926374 IP 196.35.70.139.ssh > 10.212.100.200.46247: Flags [.], ack 838, win 116, options [nop,nop,TS val 2837315 ecr 187753], length 0
18:46:16.986396 IP 196.35.70.139.ssh > 10.212.100.200.46247: Flags [P.], seq 805:957, ack 838, win 116, options [nop,nop,TS val 2837315 ecr 187753], length 152
18:46:16.986458 IP 10.212.100.200.46247 > 196.35.70.139.ssh: Flags [.], ack 957, win 71, options [nop,nop,TS val 187792 ecr 2837315], length 0
18:46:16.988362 IP 10.212.100.200.46247 > 196.35.70.139.ssh: Flags [P.], seq 838:982, ack 957, win 71, options [nop,nop,TS val 187792 ecr 2837315], length 144
18:46:17.477898 IP 196.35.70.139.ssh > 10.212.100.200.46247: Flags [.], ack 982, win 140, options [nop,nop,TS val 2837372 ecr 187792], length 0
18:46:17.798919 IP 196.35.70.139.ssh > 10.212.100.200.46247: Flags [P.], seq 957:1677, ack 982, win 140, options [nop,nop,TS val 2837372 ecr 187792], length 720
18:46:17.801802 IP 10.212.100.200.46247 > 196.35.70.139.ssh: Flags [P.], seq 982:998, ack 1677, win 83, options [nop,nop,TS val 187873 ecr 2837372], length 16
18:46:18.048892 IP 196.35.70.139.ssh > 10.212.100.200.46247: Flags [R.], seq 1677, ack 982, win 0, length 0
18:46:18.088924 IP 196.35.70.139.ssh > 10.212.100.200.46247: Flags [R.], seq 1677, ack 998, win 0, length 0

And on linux.delter.co.za:

20:17:52.448213 IP 41.157.80.24.57790 > linux.delter.co.za.ssh: S 8197450:8197450(0) win 5840 
20:17:52.448247 IP linux.delter.co.za.ssh > 41.157.80.24.57790: S 1497501202:1497501202(0) ack 8197451 win 5792 
20:17:52.840000 IP 41.157.80.24.57790 > linux.delter.co.za.ssh: . ack 1 win 46 
20:17:52.846789 IP linux.delter.co.za.ssh > 41.157.80.24.57790: P 1:21(20) ack 1 win 91 
20:17:53.079622 IP 41.157.80.24.57790 > linux.delter.co.za.ssh: . ack 21 win 46 
20:17:53.161323 IP 41.157.80.24.57790 > linux.delter.co.za.ssh: P 1:22(21) ack 21 win 46 
20:17:53.161345 IP linux.delter.co.za.ssh > 41.157.80.24.57790: . ack 22 win 91 
20:17:53.162056 IP linux.delter.co.za.ssh > 41.157.80.24.57790: P 21:805(784) ack 22 win 91 
20:17:53.750503 IP 41.157.80.24.57790 > linux.delter.co.za.ssh: P 22:814(792) ack 21 win 46 
20:17:53.784763 IP linux.delter.co.za.ssh > 41.157.80.24.57790: . ack 814 win 116 
20:17:53.839959 IP 41.157.80.24.57790 > linux.delter.co.za.ssh: . ack 805 win 58 
20:17:54.100058 IP 41.157.80.24.57790 > linux.delter.co.za.ssh: P 814:838(24) ack 805 win 58 
20:17:54.100072 IP linux.delter.co.za.ssh > 41.157.80.24.57790: . ack 838 win 116 
20:17:54.102989 IP linux.delter.co.za.ssh > 41.157.80.24.57790: P 805:957(152) ack 838 win 116 
20:17:54.479858 IP 41.157.80.24.57790 > linux.delter.co.za.ssh: . ack 957 win 71 
20:17:54.630280 IP 41.157.80.24.57790 > linux.delter.co.za.ssh: P 838:982(144) ack 957 win 71 
20:17:54.664784 IP linux.delter.co.za.ssh > 41.157.80.24.57790: . ack 982 win 140 
20:17:54.673005 IP linux.delter.co.za.ssh > 41.157.80.24.57790: P 957:1677(720) ack 982 win 140 
20:17:55.236939 IP 41.157.80.24.57790 > linux.delter.co.za.ssh: R 982:982(0) ack 1677 win 0

Upon initial inspection I have to say, I don’t see the tell-tale signs of tcp splicing as I did with Vodacom. There doesn’t appear to be any sequence number adjustments. There is some NAT going on which isn’t desirable (and Vodacom moved away from using NAT once they’re user base started getting beyond a certain point because “it didn’t scale” according to one of their lead technicians).

When I say I can’t find signs of tampering I really mean it. Looking at the above you’ll see there is 19 packets on linux.delter.co.za and 21 on serenity. The first 18 of both these traces ARE IDENTICAL (other than the NAT’ed IP). After this 18th packet the server side receives an RST packet directly after it sent the data for 957:1677, along with a correct ACK for 1677. The client side receives this data, and no surprisingly doesn’t actually respond with an RST but instead with an ACK. Directly after sending this ACK it receives two identical RST packets, which again, has not been sent by the server.

So I ask this – who is generating these RST packets? Who can I have beaten with a blunt object? I want to unwind – it’s been a bad weekend with this little cherry on top.

Share on Facebook

Capped, Uncapped and Unmetered

Jaco Kroon — Sun, 04 Jul 2010 10:13:27 +0000

Recently we’ve seen an explosion of uncapped accounts entering the market. We’ve also seen that they are typically horribly slow in comparison to capped accounts – and if one goes and reads most of the acceptable use policies it becomes clear that they are in fact not uncapped, but rather, severely shaped capped accounts.

Shaping implies that certain traffic classes will get priority over others, so for example http traffic and smtp to their local services will be OK’ish and mostly everything else will be rather bad. This, dear providers, is NOT broadband. This is abuse of the word uncapped and you should clearly mark your products for what they are. If you have a “soft cap” of 30GB – that is still a cap. If you have a (like IS’s newly released business class uncapped accounts) floating CAP – state it (thanks IS, at least you don’t try to mislead here, 20 GB over 10-days floating CAP with the top 20 % of users throttled sounds reasonable).

Now, the price tag on the IS uncapped account mentioned above is slightly in excess of R2000 ex VAT. For my office I’ll need two of those accounts (we run through approximately 100GB worth of traffic each month and I’m NOT willing to sacrifice on quality of the service). For those doing the math already, R4400 vs R5000 … yea. If those were the complete facts I might consider switching and doing some careful load balancing over the two accounts to try and stay out of the top 20 % as well as to remain under the floating cap. However, the price difference isn’t severe enough for me to honestly consider it, and I’ve got another trick up my sleave: split routing.

I’ve approximately three years ago figured out how to separate local and international bandwidth at the client premises into two separate accounts. This allows me to utilize cheap local-only ADSL accounts for local bandwidth, and normal blended ADSL accounts for my international traffic. Seeing that our split is about 60 % local 40 % international this means that my cost ends up being around R2500 to R3000 per month, ex VAT for my bandwidth every month. And I get this at the same quality that you’ve come to expect from SAIX’s ADSL accounts. No frills, no fuss, good international latencies (usually at around 250 to 300 ms) and excellent local latencies of as low as 10ms (compared to around 25 to 30 on IS accounts).

Even three years ago this was beneficial, and I thought that this concept was going to be killed when the uncapped accounts started entering the market … yet the opposite has become true – I’ve now got even more inquiries asking WHY uncapped is so bad, and what alternatives are there. And this is only from a “consumer dsl” perspective (ie, sme and home market).
When one starts looking at data centres the costs associated with bandwidth starts looking even worse. No more el-cheapo ADSL (yes, trust me when I tell you ADSL bandwidth is EXTREMELY cheap). Now you have to start getting things like metro-ethernet. You need to start buying transit. If you’re hosting you’re most likely paying per GB over a certain thresshold (eg, first 3GB for your server is included with monthly and after that you’re paying per GB). If you’re the hosting environment you’re most likely buying bandwidth in per mega-bit chunks.
No matter in which of these arenas you’re playing there is NO SUCH THING as uncapped bandwidth. Either you’re being limited by an artificial cap such as you can use 3GB at any rate you please (which is also a lie as the upstream BW has a limit in terms of bits per second) or you’re being limited by the bits per second.

What uncapped really means is unmetered. In other words: We will allow you to consume bandwidth at an average of X bits per second, and we won’t actually (for billing purposes) measure how many bytes you push over the link. This means immediately that you pay for capacity instead of per byte. It’s also possibly to buy such unmetered solutions in an oversubscribed manner, for example, you can buy “gold” or “silver” transit from SAIX, “gold” means that you will have a contention ratio of 1:1 (meaning you will always be able to use your full capacity), or with silver you can get a contention ratio of 3:1 – which means that permitting that the other people aren’t consuming bandwidth you can burst up to your pipe size, but you’re only guaranteed of a third of it. Either way – 4Mbps of this is likely to make you understand why uncapped ADSL is a bad idea.

Share on Facebook

The Great Wall of Vodacom – FAIL

Jaco Kroon — Tue, 25 May 2010 09:35:05 +0000

Right, so Kevin (one of my staff) had the savvy to take a few tcpdump traces on both the client and the server side of a failed PPtP VPN connection over the weekend. The result? It seems the great firewall of Vodacom has yet again taken another victem.

I’m not sure whether this is a result of too little testing, total ignorance or just incompetence. Either way, it would seem it’s a bit of a race condition, and hits something similar to what we in the office refer to as the “connection tracking bit bucket”. Basically it seem most connection tracking implementations (when combined with a state full firewall such as that used by Vodacom – as per their own admission in their last letter to me) results in certain flows being prematurely marked as “invalid”. In particular in the example that Kevin has captured for me the server ends up being the first entity to send a GRE packet, this then gets (or got, seeing that it’s fixed again) intercepted by the firewall, perceived as an inbound connection to the client and the uni-directional flow gets marked as invalid. When the client now sends GRE traffic to the server this gets allowed, but the return traffic still bites the “invalid” mark. I can only speculate as to the exact state (seeing that Vodacom doesn’t reveal exactly what software they are using – probably proprietary anyway) of things, making it difficult. This I will attempt to speculate as objectively as possible (not always easy).

Seeing that there are two entities involved in this dump, and I want to do a side-by side comparison, some ASCII art is in order. Essentially three columns being used, the sending agent will indicate what is being sent, and if it was received by the destination I’ll mark that column with an ACK. I’ll also add (R) to retransmits on the sending side. The ISN mods still applies to the TCP connections, however, the data itself isn’t being tampered with in this case. Note that in the GRE traffic case there are still ACK packets being sent by the server, these ACK packets however goes lost (as indicated in the packet sequence).

Client                Direction  Server
SYN                        ->    ACK
ACK                        <-    SYN
PPTP (Start Req)           ->    ACK
ACK                        <-    PPTP (Start Resp)
                           <-    GRE (PPP-LCP Conf Req)
GRE (PPP-LCP Conf Req) (R) ->    ACK (goes lost)
GRE (PPP-LCP Conf Req) (R) ->    ACK (goes lost)
... a few more of these ...
GRE (PPP-LCP Conf Req) (R) ->    ACK (goes lost)
PPTP (Call Clear Req)      ->    ACK

Once the Call Clear Req is received TCP/IP teardowns happens, surprisingly without the flurry of injected RST packets I’ve growned accustomed to, just a single out-of-order delivery between one ACK and FIN/ACK packet.

What I would want (not sure what resolution they picked) is for them to either perform a routine inspection of the PPTP control traffic (specifically the Start Request and Start Reply packets) to determine the GRE traffic parameters (based on what I can see, just mark the fact that GRE is to be expected between the two given end points) and allow that traffic, or, stop this firewalling nonsense. It’s only the Cellular “ISPs” performing actions such as these. The arguments for providing this protection is sound. But then it needs to be done sanely. For the most part I’ll have to admit that the firewall works and doesn’t cause too many problems.

Seeing that the problem has been resolved by Vodacom, I’ll let it rest, for now.

Share on Facebook

FQDN – ala Fully Qualified Domain Name

Jaco Kroon — Tue, 09 Feb 2010 12:43:31 +0000

Right, so I conceptually know what this is, but how is it determined? What’s the impact?

As far as I can tell most applications couldn’t really care about the fqdn, exceptions would be email, and apache likes having one. The FQDN is basically the full name that a remote machine will use to recognize your machine with. Eg, my VoIP server sits at 196.35.70.140 but users use voip.uls.co.za to resolve it. If you do a reverse resolution of the IP then the name is r2d2.uls.co.za – and it’s this part that is important for us, the FQDN for that machine is r2d2.uls.co.za, it’s hostname is r2d2 with it’s dns domain name uls.co.za.

So how does Linux determine this? Dead simple actually.

It does a forward lookup of it’s hostname (r2d2 in this case), including any domain and search domains configured in /etc/resolv.conf, if nsswitch actually reaches DNS lookups.
It does a reverse lookup of the obtained IP and this result is used as the FQDN.

Complex? Not really, until you start looking at what all needs to work in order for this to be correct. A common thing done by many a sysadmin is to in /etc/hosts modify this line:

127.0.0.1   localhost

To:

127.0.0.1   localhost hostname

Well, this will result in hostname -f giving you “localhost” as the FQDN. That’s broken. The correct line above (for r2d2) would be:

127.0.0.1 r2d2.uls.co.za r2d2 localhost localhost.localdomain

That is overly complete and localhost.localdomain can probably be left out. There is another way though, because let’s face it, 127.0.0.1 isn’t our public IP. So here we go, with the “correct” way:

Set the domain/search lines in /etc/resolv.conf correctly. Basically I’d recommend leaving “search” alone unless you want to search multiple domains for a “short” name, and only set up “domain”.
Make sure that there is a properly configured PTR record for you public IP.
Make sure that hostname.${domain} resolves to your, and only your, public IP.

Eg, in my /etc/resolv.conf I’ve got “domain uls.co.za” along with my nameservers (ok, 127.0.0.1 which points to a djb dnscache). Then my hostname is set to “r2d2″. With this configuration when I resolve r2d2 I get an IP of 196.35.70.140, then when I reverse-lookup that IP then I get r2d2.uls.co.za which is my FQDN. Forward looking up r2d2.uls.co.za again resolves to that IP which is exactly what I want.

Share on Facebook

Linux, 3G, bluetooth and DUN

Jaco Kroon — Wed, 20 Jan 2010 21:37:43 +0000

So I got this working on my older Nokia 3120, then got an E52 and then had to go through all of this again. So I decided that’s that, I need a reference. Posted here because it’s least likely to go away, and easiest place to find it again. Also, other people may find this useful.

First things first, you obviously need the Linux bluetooth stack. Look for a package containing “bluez”, on Gentoo Linux there is two sets, the first is bluez-utils and a bunch of other dependencies, and “bluez” (which from what I can tell is the newer stuff). Once you’ve got this installed you should probably try and start the bluetooth daemon to see if it starts, also whether the drivers are loaded etc …

Finding the Bluetooth drivers for your chip is not something I’m going to discuss, lspci and lshw is your friends in this respect, and most distro’s will auto-load kernel modules based on available hardware. If you can type “ip ad sh” and see a pan0 device you’re probably good to go.

At this point I suggest looking at /etc/bluetooth and editing as you see fit (starting with main.conf, probably only want to edit Name = … and even that’s a maybe).

To find out whether all actually works as expected, you should be able to run “hcitool scan”, this should output something like:

# hcitool scan
Scanning ...
        34:7E:39:62:7B:65       Reboot
#

Reboot is my new Nokia phone. The MAC-addr like number on the left is called a bdaddr. Get that string in your cut buffer – you’re going to need it a lot.

Now you need to “pair” your phone with your laptop. This is the hardest part of the whole exercise and depends on which of the tools you’re using. I had quite a lot of trouble doing this from command-line only and eventually just installed gnome-bluetooth and used the wizard to pair, then on my phone set my laptop to authorized to allow it to connect without having to go through the whole pairing process every time. On the laptop side I didn’t actually change anything or had to set things explicitly.

The next step is to locate the bluetooth “endpoint” on the phone to use for dial-up networking. This is done using sdptool:

# sdptool search --bdaddr 34:7E:39:62:7B:65 DUN
Searching for DUN on 34:7E:39:62:7B:65 ...
Service Name: Dial-Up Networking
Service RecHandle: 0x10047
Service Class ID List:
  "Dialup Networking" (0x1103)
Protocol Descriptor List:
  "L2CAP" (0x0100)
  "RFCOMM" (0x0003)
    Channel: 5
Language Base Attr List:
  code_ISO639: 0x454e
  encoding:    0x6a
  base_offset: 0x100
Profile Descriptor List:
  "Dialup Networking" (0x1103)
    Version: 0x0100
 
#

The important bit there is Channel: 5. Now edit /etc/bluetooth/rfcomm.conf, in particular we want to create a serial device called rfcomm0, that always binds to our phone, on the correct channel. The config file ends up looking like this:

rfcomm0 {
    bind yes;
    device 34:7E:39:62:7B:65;
    channel 5;
    comment "DUN on Reboot";
}

Now restart bluetooth again and you should now have a rfcomm0 device node in /dev/. If you do, try opening two terminals, in the one, issue cat /dev/rfcomm0, and in the other issue “echo ATZ > /dev/rfcomm0″, the cat side should echo back your ATZ and hopefully not spit too much gunk at you. From here on it’s pretty simple. I use my chat perl script from my 3G, PINs, prompts and pppd entry, along with this file in /etc/ppp/peers/:

/dev/rfcomm0
linkname 3g-rfcomm0
defaultmetric 5000
 
# Don't require the peer to authenticate (this _seems_ to get the connection going in a shorter time)
noauth
 
# If you want more detailed logs, enable this (it doesn't generate that much
# noise, but assists a lot in finding problems):
#debug
 
# Make this one second since the modem probably has to be re-initialized at least
# once.
holdoff 1
 
# modem initialization
connect /usr/local/sbin/uls_3g_connect.pl
 
# We probably want to use the DNS as advertized by the peer
usepeerdns
 
# Use this link as the default gateway
defaultroute
 
# Inform out ip-up script about what this is:
ipparam "type=intl dnsroutes=yes routemetric=10"
 
# 3G doesn't like all kinds of compression ... which they did - it's slow
# enough.
noccp
nobsdcomp
novj
 
# Make the connection persistent, and not terminate if/when errors occur.
persist
maxfail 0

There is no real difference between this peers file and the one in 3g from the blog mentioned above. This one just adds a linkname and explicit reference to the rfcomm0 device. I saved this file as bluetooth. Now you should be able to run “pon bluetooth”, or more directly “pppd call bluetooth” or even “pppd call bluetooth debug nodetach” if you prefer.

Share on Facebook

ibdriver and the 2.6.31 kernel (iBurst Linux drivers)

Jaco Kroon — Sat, 31 Oct 2009 08:23:03 +0000

So in the 2.6.31 kernel the older (deprecated) network API finally got removed – biting quite a number of people rather badly. I can think of at two projects that’s problematic due to this:

1. The ibdriver package – used for the iBurst usb and pcmcia devices.
2. The dahdi 2.0.x drivers used for telephony in Asterisk.

The latter isn’t that serious a problem as I really need to move to dahdi-2.2.x anyway, the ibdriver however caused me some embarrasment as I plugged in the usb device, downloaded the drivers and … it didn’t compile. Oops. So I decided it’s time to return to some of my older roots and just make the driver work – and that’s exactly what I did.

This morning, after reading a blog post yesterday about it taking a guy two hours (without posting patches) I set aside three, and figured, at a minimum I’ll remember why I don’t do kernel hacking for a living.Â That happened, but it also only took me about 30 minutes to build a patch for running the iburst drivers against the 2.6.31 kernel.Â The main changes are that you’re now to use netdev_priv(dev) for accessing private data instead of dev->priv and a bunch of the device operations which has previously been part of the net_device struct is now in net_device_ops, with a few slight name changes.Â The only other thing really is that one struct changed name in the pcmcia subsystem (config_info_t -> socket_state_t).

With no further ado, you can find the patch here.

Please note that whilst I’m busy running the code with this patch I can’t guarantee that there are no mistakes in it.Â I just took the kernel headers, and updated the ibdriver code according to what I could tell from these headers, use at your own risk.

Share on Facebook

ARP Spoofing – a lost art? Maybe not.

Jaco Kroon — Sun, 10 May 2009 10:09:21 +0000

Just over a month back we had an incident where the default gateway on our servers would just sporadically stop responding, we first observed this as our servers sporadically just stopping to respond and only once we realized we could log on to other servers and during these “outages” we could still communicate with our servers via our other servers (ie, we could access them from the local LAN but not anywhere else) did we start pointing fingers at the gateway.

At the time we figured this was a misconfigured or faulty gateway (Seeing as Internet Solutions is using HSVRP we figured these outages were just the “fail over” time).Â Quintin however was told that our gateways are misconfigured and they told him he should be using x.y.z.253 and NOT x.y.z.1 as we were told during the installation of these servers.Â Since then the Windows server which he was working on was not affected again and seeing as most of my Linux servers just keep running I didn’t think too much about this again until last weekend when I was actively working on one of them.Â I re-reported the issue, making it clear that the explanations I were given previously was inaddequate and not acceptable (I were given answers such as someone accidentally bumped a cable etc …).

After running mtr traces from my dsl into the DC to three of the servers (web/mail, voip and windows) since reporting it at 15:54 (first response at 16:48 – reasonable response time for ONCE, thanks IS) I realized two things:

~ 30% round-trip packet loss on the final hop, and around 0.4 – 0.7 % loss on the hops before that (meaning, probably the DSL side of things).
The Windows server was not affected at all!Â This made no sense whatsoever to me until I realized it was using a different gateway.

Either way, after sending these traces off to IS @ 17:25 – showing even higher loss on the last hop than the above 30 % (67 % and 57 % respectively), and later (18:40) passing on an update showing that I could still get responses from my servers internally inbetween each other in spite of not being able to get access to the outside world.Â At 7:38 the Sunday I looked at the traces again and since the loss decreased to 5 % and 6 % I figured they must’ve restored sanity and fixed the problem.Â Â I was wrong, very very wrong.

One of my clients phoned me around 8:45ish and said I should look at my email – they think I might want to look at it, it looks like my admin site has been compromised.Â Of course I immediately loaded the site, just to see a funny bar at the top-left and all my fonts looking rather funky.Â A few page source showed a