jkroon

Ok, so I just spent 3 days hunting after a problem with my Linux configuration that turns out to be due to some “wireless expert” that configured a “wireless bridge” between three sites at one of my clients.  They became my client about a week after the quote was accepted and no longer makes any network topology change without consulting with me first (at least, not as far as I know).  Either way … if you want a prime example of why it doesn’t work, read on …

The configuration looks (rather simplified), something like this:

No, firstly, there is actually THREE wireless units on the “high site” (shown in the middle), these each provide an “AP”, where the wireless interface is bridged with the LAN interface (where all three in turn is plugged into a switch).  All of this is being run off some rather big sealed batteries that is supposed to be charged by a solar panel.  At each of the three sites the wireless interface is in station mode, again, bridged with the LAN interface.

The end result of all this is that I’ve got about 20 odd networks (yes, behind each of the linux routers there is some other /24 networks, fooling with a few hundred IPs in total, mostly segmented for logical reasons).  None of these networks are being NATed, however, since the particular wireless APs in question does bridging, and wasn’t configured for any kind of routing, and on a subnet which makes absolutely no sense in the rest of the configuration (and which for political reasons I’m not allowed to change, and even if I could I wouldn’t as these things can’t do anything but static routing).  Well, that’s bygones.  Now, let’s consider two of the Linux routers, on the LAN side the one (Let’s call it A) has 192.168.3.1/24 and on the wireless side 192.168.1.70/24, the other router (B) has 192.168.8.1/24 and 192.168.1.69.  Routes are set up between them:

A # ip ro ad 192.168.8.0/24 via 192.168.1.69 \
    src 192.168.3.1

and

B # ip ro ad 192.168.3.0/24 via 192.168.1.70 \
    src 192.168.8.1

Yes, the src is actually required as the 192.168.1.0/24 network isn’t known anywhere else on the network, nor do we want it known.

So, it’s actually possible to ping 192.168.8.1 from A and it works.  From B I can ping 192.168.3.1 and it works.

My problem?  Well, randomly some IPs doesn’t work.  I spent a good few hours doing tcpdump and trouble shooting from A into B all the packets (according to tcpdump) are arriving on the wireless side, but Linux discards them going into the network. Sounds funny?  Well, yes.  I checked the firewall, I checked (and rechecked) the routing tables, and flippen everything I could think of.

Then (on a hunch) I decided to look at the ARP tables on either end of the link:

A # arp -n
192.168.1.69  00:06:c7:01:47:9b
192.168.1.20  00:06:c7:01:47:9b
A # ip link sh dev ethwl
2: ethwl:  mtu 1500 qdisc noqueue
     link/ether 00:30:05:ea:fc:7f brd ff:ff:ff:ff:ff:ff

Ok, that looks sane (keep in mind that wireless stations that claims to do briding actually does proxyarp.  And on B:

B # arp -n
192.168.1.70  00:30:05:ea:fc:7f
B # ip link sh dev ethwl
3: ethwl:  mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:0c:42:1a:0b:7d brd ff:ff:ff:ff:ff:ff

If you’re aware of what’s going on on networks I’m hoping you’re guessing what’s coming next.

Ethernet header for _both_ working and non-working packets leaving from A looks like this (spaces used instead of colons to group dst mac, src mac and content type):

00:06:c7:01:47:9b 00:30:05:ea:fc:7f 08:00

This looks perfectly sane based on the arp tables, absolutely stunning.

The working ethernet header that arrives at B:

00:0c:42:1a:0b:7d 00:30:05:ea:fc:7f 08:00

That looks fine, MAC got de-masqueraded, absolutely perfect, packet gets accepted into the Linux kernel, and routed.  The non-working packet looks like this:

00:06:c7:01:47:9b 00:30:05:ea:fc:7f 08:00

Now if you’re not screaming that that looks identical to the packet that left at A then you need to look again.  Here is what’s happening, the wireless _station_ just in front of B sees the MAC, it’s it’s MAC, but it also looks at the IP, notices that it’s not it’s own IP and now “broadcasts” the packet onto all it’s interfaces except the ingress “port” (wireless link).  Now if there was a switch the switch would have sent the packet right back at the wireless AP, but since in this particular case I have a cable the network card sees the packet, gives the destination one look and discards the packet without ever disturbing the Linux kernel.  The _only_ reason why I can see that packet at all with tcpdump is because it puts the card in promiscuous mode.

Possible fixes?  This is a tough one.  I’ve spent an hour or so thinking about this now, and I’ve come up with the following solutions (In order of preference):

1. Do the right thing, throw out the junk that was put in and put in the right stuff (Wireless routers capable of communicating with the Linux servers via OSPF so that I can get link status on the Linux servers and fallback to IPSec over DSL/ISDN/whatever).
2. Make insane use of NAT on the wireless interfaces, and again NAT when traffic comes in off the wireless.  For auditability reasons this really isn’t an option.
3. Encapsulate the traffic inside some other protocol.  Also a pain since this lowers the MTU of the link.  IPSec _should_ be able to get this done, but it’s also “costly” in terms of CPU resources.

I really can’t think of anything else, making use of ebtables isn’t an option as the kernel doesn’t even get woken up for these packets.  proxyarp won’t help either.  In short, it’s one glorious mess.

Time to go chill.

This entry was posted on Wednesday, January 28th, 2009 at 10:20 pm and is filed under Networking, Wireless. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.