Firstly, I think people need to understand how many emails comes in which are duplicates of existing issues (often logged by the same person). This can be caused by many things, the best thing to note here is that most ticketing systems use a “subject tag” to identify the ticket it’s using. RT for example puts a [uls.co.za #???] string in our tickets. Now consider the following scenarios:

• Subject Cleanup: The act of cleansing the subject. Most users guilty of this don’t realize that email can be processed by automated systems, and that those automated systems may be relying on the subject to remain in tact for some reason.
• New Emails with subject referencing the other ticket: Often I’ll see subjects like “re ticket 1234″, this isn’t good enough for RT to link the tickets automatically, causing a new ticket to be logged.
• New emails with no reference: Often we’ll get emails that are obvious replies, but doesn’t contain message context. For example, one we’ve recently recently was an email with subject “problem” and body “yes” and I also recall some tickets where the subject is simply “thanks” with an empty body.
• Replying to an existing email for a new issue: This also has the potential for causing great chaos. Often one reads emails in the light of the context, and keeping the subject in the back of your head …
• Useless subjects: For examples “phones” or “email” or something. This is NOT a descriptive subject. When under load support staff tends to scan subjects of logged tickets for important things that needs to be dealt with first. Subjects such as “phones” or something equally cryptic is NOT likely to get your issue on the top of that list.

These things are actually general email etiquette, and comes from writing proper letters. I know I used to be hammered on how to write a proper letter, both in Afrikaans and English. I also recall thinking how useless that was at the time seeing that everything is moving towards email. The art of writing letters however teaches you how to think structured. In other words: How do you pick a subject, how to structure paragraphs and how to convey information in a concise manner (an art I suck at – I tend to write essays and then have to trim them back down).

It’s difficult to give advice on how to best approach an issue. However, I can say this, often just phoning isn’t going to get your problem sorted out. The problem here is that unless the technician has the savvy to open a new ticket with your email address as requestor, and actually entering all the information you’re giving him/her over the phone chances are another call is going to come in as soon as you put down the phone totally obliterating the technicians thought process, and if you’re lucky he might context-switch back to your issue. More likely than not (especially for the scatter-brained like myself) your issue is completely forgotten by now. Thus I personally believe that if at all possible, take the time to write an email with the information you deem required and email that. If the technician requires more info he’ll contact you. The other risk is providing too much information, requiring the technician to first sift through relevant and irrelevant information. Not always an easy task.

I think the most crucial thing when logging a ticket is picking a good subject. For example, instead of just saying “phones” one could say “switchboard phone @ company xyz is down” or “switchboard phone @ company xyz is unable to transfer”. This is concise and I can almost promise you you don’t even need to write something in the body. This appears immediately in the list of tickets, it’s easy for technicians to pick out their tickets and they can almost commence work on your issue without even having to open your email.

Picking subjects like “urgent” is absolutely useless, it doesn’t tell us what the issue is about, and we’ve learned from experience that all issues are urgent. However, if you email me a subject with “urgent” stating that you’re unable to receive incoming email you’re very likely to be ignored the next time round you do have an urgent query. Yes, your email may be urgent, but let’s say you’re the technician and next to that you have a ticket that states: “PABX system down, unable to make/receive calls”, or “web server down” or “database corruption on server xyz database abc” – which do you think is more urgent? Your email problem (which is most likely caused by a temporary delay anyway) or the fact that a database that goes corrupt tends to cascade down the corruptness ladder once it gets going?

Other than subjects my other pet peeves are not using the reply and forward buttons correctly. There are two extremes for the reply here – on the one hand there are people that will NEVER use the reply button – they always create a new email, usually with a subject somewhat similar to the email they’re replying on, without copying any of the context they’re replying on. On the other hand you get folks that will always go and dig out an old email and hit reply, and updating the subject line is optional. At this point I’m going to curse at Microsoft. Outlook doesn’t do any kind of half-decent threading display last I checked. Almost every other mail client do. Guess what the usual suspects tends to use. The importance of the reply button is critical, almost all mail clients honor the Message-Id and In-Reply-To email headers (not visible by most mail clients, but used in order to figure out threading). If you don’t reply when you should our threaded displays will start a new thread (which it’s not). If you reply instead of creating a new email your email goes into an existing thread, again incorrectly so.

The rule is really very simple: Always use the reply-all button and not just reply (there is a reason people was added as a Cc on a mail in the first place – they should probably also receive the reply). Use reply if you’re answering or responding to an existing email. Do NOT use reply if you’re starting a new “discussion”. It’s not difficult. Common sense should be more than sufficient to grok this.

Now, recently I’ve started seeing a new trend (or possibly accidental usage). And that is to click forward instead of reply, and then manually re-entering all the email addresses. Not overly serious, just annoying. This mostly just messes with the subject line.

When forwarding messages, please do bother checking the subject. A prime example here is if you receive a bounce and simply forward it – the subject itself is most likely useless, in fact, I’m likely to treat it as a bounce myself and it’s likely going to end up somewhere trashy. In the case of a bounce, it’s all good and well, but please do think a bit, a subject of “mail undeliverable” compared to “unable to send email to abc@def.co.za” is likely to be better received, and in the body you can simply state “when trying to send email to abc@def.co.za I receive the following bounce back: …”. The same goes if you forward something you’ve received from another person.

Keep enough context, but do clean it up. This is related to the whole top-posting vs bottom-posting vs inline posting. I tend to use a combination. I usually have a greeting at the top, then I respond inline in the email, sometimes add a new conclusion paragraph and then my salutation and signature at the bottom. The reason I do this is so that someone that sees my email (without having received the original) can read the newly formed “document” top to bottom like a sane human being. It’s extremely important to clean up the email though. Outlook users tend to spam us with email footers that is approximately ten times longer than the actual content of their email (this goes to the cut buffer never to be returned).

On the topic of footers – yes, they can look nice, but more often than not they only serve to irritate the recipient, especially if you top-post (as most outlook users do). When logging issues with issue trackers – strip them out. They waste space, and seeing that issue loggers often prefer reading the plaintext version it looks like garbage when compared to the html variant. The reason why these footers are irritating when top-posting is because it makes it difficult for the recipient to locate the original email. Why is that important? Well, you may only send one or two emails a day and can remember exactly what you said in each of them – there are some of us that send anywhere upwards of 30 to 50 emails a day. Personally I don’t remember what I said in an email I wrote a minute back, thus when I receive an email I first scan what I wrote in order to joggle the memory before I read your reply. If three quarters or even more of the email is headers and not actual content this becomes difficult.

Now, when everyone follows these extremely basic rules it saves technicians a LOT of time. To give you an idea – I personally spend probably around 45 to 60 minutes a day just sorting through the above few things. This wastes time that could have been spent way better towards actually solving real problems. In normal day-to-day email it’s not as painful because I just tend to read my email top-to-bottom usually but even here it’s preferred to follow the rules. Especially the aesthetics such as maintaining good subjects, cleaning up emails etc. For ticketing systems especially it’s crucial to follow the rules regarding when to reply and when no to reply. Never clean out the subject. Ever. There is most likely a reference tag in the reply you received and removing this will annoy the technicians trying to help you.

Share on Facebook

Unfortunately it’s insanely hard to prove conclusively where the TCP resets are coming from, again the only evidence I’ve got that it has to be Cell C is the fact that it works flawlessly from everywhere else (SAIX ADSL, Mweb ADSL and Vodacom 3G). So the first things I started noticing yesterday was ssh connections going something down these lines (serenity is my local machine, linux.delter.co.za a relatively big mail server from one of my clients):

jkroon@serenity ~ $ ssh root@linux.delter.co.za 
ssh_exchange_identification: read: Connection reset by peer

Now my employees knows, if I can’t ssh and it’s your fault, you’re going to get it. Firstly I will hunt you down, then I will do things which cannot be considered polite, and if you’re name is bigger than mine and my client believes that because your name is bigger than mine that implies you’re right and I’m wrong I will ensure that I prove them wrong and make very sure that they understand that I don’t take these things lightly. Well, not when it affects my work anyway, but I do understand if things breaks periodically, but at the moment I can’t even browse and in excess of 95 % of the connections I’m pushing out over my Cell C SIM is outright being reset.

So after seeing the above for for approximately 3 out of 5 connections this morning whilst sitting in a data center in johannesburg I just ran a tcpdump in a different shell on serenity to see what happens:

08:53:56.835503 IP 196.35.70.139.ssh > 10.213.51.133.47019:
    Flags [R.], seq 2902460094, ack 3806794395, win 199,
    options [nop,nop,TS val 26312228 ecr 4876698], length 0

Now I KNOW the way I set up my servers. And if my server is in fact generating that RST then there is something severely broken. But I’m getting this from different servers. So, having had one of the craziest weekends for a while going on I decided to push this to the back of my mind and concentrate on more urgent matters. It’s only about 30 minutes back that I wanted to quickly check mail, browse a bit and just unwind a little that I couldn’t actually browse, ssh to my servers for a quick checkup after the weekend’s events and write an official complaint to a certain hosting company that I decided enough is enough. Got a jump box and ssh’ed via another route to linux.delter.co.za (and no surprises) it worked flawlessly. Fire up pppd and add a route for linux.delter.co.za over that, fire up tcpdump on both ends and I get this, first on serenity (sorry for the horizontal scrolling, and also note that the time on my laptop is out by ~30 minutes due to ntp failing and the CMOS on this Lenovo being of the ultra crappy kind):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

18:46:14.706636 IP 10.212.100.200.46247 > 196.35.70.139.ssh: Flags [S], seq 8197450, win 5840, options [mss 1460,sackOK,TS val 187563 ecr 0,nop,wscale 7], length 0
18:46:15.326350 IP 196.35.70.139.ssh > 10.212.100.200.46247: Flags [S.], seq 1497501202, ack 8197451, win 5792, options [mss 1460,sackOK,TS val 2837150 ecr 187563,nop,wscale 6], length 0
18:46:15.326445 IP 10.212.100.200.46247 > 196.35.70.139.ssh: Flags [.], ack 1, win 46, options [nop,nop,TS val 187626 ecr 2837150], length 0
18:46:15.659155 IP 196.35.70.139.ssh > 10.212.100.200.46247: Flags [P.], seq 1:21, ack 1, win 91, options [nop,nop,TS val 2837190 ecr 187626], length 20
18:46:15.659267 IP 10.212.100.200.46247 > 196.35.70.139.ssh: Flags [.], ack 21, win 46, options [nop,nop,TS val 187659 ecr 2837190], length 0
18:46:15.659458 IP 10.212.100.200.46247 > 196.35.70.139.ssh: Flags [P.], seq 1:22, ack 21, win 46, options [nop,nop,TS val 187659 ecr 2837190], length 21
18:46:15.969153 IP 196.35.70.139.ssh > 10.212.100.200.46247: Flags [.], ack 22, win 91, options [nop,nop,TS val 2837221 ecr 187659], length 0
18:46:15.969221 IP 10.212.100.200.46247 > 196.35.70.139.ssh: Flags [P.], seq 22:814, ack 21, win 46, options [nop,nop,TS val 187690 ecr 2837221], length 792
18:46:16.349157 IP 196.35.70.139.ssh > 10.212.100.200.46247: Flags [P.], seq 21:805, ack 22, win 91, options [nop,nop,TS val 2837221 ecr 187659], length 784
18:46:16.386434 IP 10.212.100.200.46247 > 196.35.70.139.ssh: Flags [.], ack 805, win 58, options [nop,nop,TS val 187732 ecr 2837221], length 0
18:46:16.599149 IP 196.35.70.139.ssh > 10.212.100.200.46247: Flags [.], ack 814, win 116, options [nop,nop,TS val 2837284 ecr 187690], length 0
18:46:16.599227 IP 10.212.100.200.46247 > 196.35.70.139.ssh: Flags [P.], seq 814:838, ack 805, win 58, options [nop,nop,TS val 187753 ecr 2837284], length 24
18:46:16.926374 IP 196.35.70.139.ssh > 10.212.100.200.46247: Flags [.], ack 838, win 116, options [nop,nop,TS val 2837315 ecr 187753], length 0
18:46:16.986396 IP 196.35.70.139.ssh > 10.212.100.200.46247: Flags [P.], seq 805:957, ack 838, win 116, options [nop,nop,TS val 2837315 ecr 187753], length 152
18:46:16.986458 IP 10.212.100.200.46247 > 196.35.70.139.ssh: Flags [.], ack 957, win 71, options [nop,nop,TS val 187792 ecr 2837315], length 0
18:46:16.988362 IP 10.212.100.200.46247 > 196.35.70.139.ssh: Flags [P.], seq 838:982, ack 957, win 71, options [nop,nop,TS val 187792 ecr 2837315], length 144
18:46:17.477898 IP 196.35.70.139.ssh > 10.212.100.200.46247: Flags [.], ack 982, win 140, options [nop,nop,TS val 2837372 ecr 187792], length 0
18:46:17.798919 IP 196.35.70.139.ssh > 10.212.100.200.46247: Flags [P.], seq 957:1677, ack 982, win 140, options [nop,nop,TS val 2837372 ecr 187792], length 720
18:46:17.801802 IP 10.212.100.200.46247 > 196.35.70.139.ssh: Flags [P.], seq 982:998, ack 1677, win 83, options [nop,nop,TS val 187873 ecr 2837372], length 16
18:46:18.048892 IP 196.35.70.139.ssh > 10.212.100.200.46247: Flags [R.], seq 1677, ack 982, win 0, length 0
18:46:18.088924 IP 196.35.70.139.ssh > 10.212.100.200.46247: Flags [R.], seq 1677, ack 998, win 0, length 0

And on linux.delter.co.za:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

20:17:52.448213 IP 41.157.80.24.57790 > linux.delter.co.za.ssh: S 8197450:8197450(0) win 5840 <mss 1460,sackOK,timestamp 187563[|tcp]>
20:17:52.448247 IP linux.delter.co.za.ssh > 41.157.80.24.57790: S 1497501202:1497501202(0) ack 8197451 win 5792 <mss 1460,sackOK,timestamp 2837150[|tcp]>
20:17:52.840000 IP 41.157.80.24.57790 > linux.delter.co.za.ssh: . ack 1 win 46 <nop,nop,timestamp 187626 2837150>
20:17:52.846789 IP linux.delter.co.za.ssh > 41.157.80.24.57790: P 1:21(20) ack 1 win 91 <nop,nop,timestamp 2837190 187626>
20:17:53.079622 IP 41.157.80.24.57790 > linux.delter.co.za.ssh: . ack 21 win 46 <nop,nop,timestamp 187659 2837190>
20:17:53.161323 IP 41.157.80.24.57790 > linux.delter.co.za.ssh: P 1:22(21) ack 21 win 46 <nop,nop,timestamp 187659 2837190>
20:17:53.161345 IP linux.delter.co.za.ssh > 41.157.80.24.57790: . ack 22 win 91 <nop,nop,timestamp 2837221 187659>
20:17:53.162056 IP linux.delter.co.za.ssh > 41.157.80.24.57790: P 21:805(784) ack 22 win 91 <nop,nop,timestamp 2837221 187659>
20:17:53.750503 IP 41.157.80.24.57790 > linux.delter.co.za.ssh: P 22:814(792) ack 21 win 46 <nop,nop,timestamp 187690 2837221>
20:17:53.784763 IP linux.delter.co.za.ssh > 41.157.80.24.57790: . ack 814 win 116 <nop,nop,timestamp 2837284 187690>
20:17:53.839959 IP 41.157.80.24.57790 > linux.delter.co.za.ssh: . ack 805 win 58 <nop,nop,timestamp 187732 2837221>
20:17:54.100058 IP 41.157.80.24.57790 > linux.delter.co.za.ssh: P 814:838(24) ack 805 win 58 <nop,nop,timestamp 187753 2837284>
20:17:54.100072 IP linux.delter.co.za.ssh > 41.157.80.24.57790: . ack 838 win 116 <nop,nop,timestamp 2837315 187753>
20:17:54.102989 IP linux.delter.co.za.ssh > 41.157.80.24.57790: P 805:957(152) ack 838 win 116 <nop,nop,timestamp 2837315 187753>
20:17:54.479858 IP 41.157.80.24.57790 > linux.delter.co.za.ssh: . ack 957 win 71 <nop,nop,timestamp 187792 2837315>
20:17:54.630280 IP 41.157.80.24.57790 > linux.delter.co.za.ssh: P 838:982(144) ack 957 win 71 <nop,nop,timestamp 187792 2837315>
20:17:54.664784 IP linux.delter.co.za.ssh > 41.157.80.24.57790: . ack 982 win 140 <nop,nop,timestamp 2837372 187792>
20:17:54.673005 IP linux.delter.co.za.ssh > 41.157.80.24.57790: P 957:1677(720) ack 982 win 140 <nop,nop,timestamp 2837372 187792>
20:17:55.236939 IP 41.157.80.24.57790 > linux.delter.co.za.ssh: R 982:982(0) ack 1677 win 0

Upon initial inspection I have to say, I don’t see the tell-tale signs of tcp splicing as I did with Vodacom. There doesn’t appear to be any sequence number adjustments. There is some NAT going on which isn’t desirable (and Vodacom moved away from using NAT once they’re user base started getting beyond a certain point because “it didn’t scale” according to one of their lead technicians).

When I say I can’t find signs of tampering I really mean it. Looking at the above you’ll see there is 19 packets on linux.delter.co.za and 21 on serenity. The first 18 of both these traces ARE IDENTICAL (other than the NAT’ed IP). After this 18th packet the server side receives an RST packet directly after it sent the data for 957:1677, along with a correct ACK for 1677. The client side receives this data, and no surprisingly doesn’t actually respond with an RST but instead with an ACK. Directly after sending this ACK it receives two identical RST packets, which again, has not been sent by the server.

So I ask this – who is generating these RST packets? Who can I have beaten with a blunt object? I want to unwind – it’s been a bad weekend with this little cherry on top.

Share on Facebook

Now, the price tag on the IS uncapped account mentioned above is slightly in excess of R2000 ex VAT. For my office I’ll need two of those accounts (we run through approximately 100GB worth of traffic each month and I’m NOT willing to sacrifice on quality of the service). For those doing the math already, R4400 vs R5000 … yea. If those were the complete facts I might consider switching and doing some careful load balancing over the two accounts to try and stay out of the top 20 % as well as to remain under the floating cap. However, the price difference isn’t severe enough for me to honestly consider it, and I’ve got another trick up my sleave: split routing.

I’ve approximately three years ago figured out how to separate local and international bandwidth at the client premises into two separate accounts. This allows me to utilize cheap local-only ADSL accounts for local bandwidth, and normal blended ADSL accounts for my international traffic. Seeing that our split is about 60 % local 40 % international this means that my cost ends up being around R2500 to R3000 per month, ex VAT for my bandwidth every month. And I get this at the same quality that you’ve come to expect from SAIX’s ADSL accounts. No frills, no fuss, good international latencies (usually at around 250 to 300 ms) and excellent local latencies of as low as 10ms (compared to around 25 to 30 on IS accounts).

Even three years ago this was beneficial, and I thought that this concept was going to be killed when the uncapped accounts started entering the market … yet the opposite has become true – I’ve now got even more inquiries asking WHY uncapped is so bad, and what alternatives are there. And this is only from a “consumer dsl” perspective (ie, sme and home market).
When one starts looking at data centres the costs associated with bandwidth starts looking even worse. No more el-cheapo ADSL (yes, trust me when I tell you ADSL bandwidth is EXTREMELY cheap). Now you have to start getting things like metro-ethernet. You need to start buying transit. If you’re hosting you’re most likely paying per GB over a certain thresshold (eg, first 3GB for your server is included with monthly and after that you’re paying per GB). If you’re the hosting environment you’re most likely buying bandwidth in per mega-bit chunks.
No matter in which of these arenas you’re playing there is NO SUCH THING as uncapped bandwidth. Either you’re being limited by an artificial cap such as you can use 3GB at any rate you please (which is also a lie as the upstream BW has a limit in terms of bits per second) or you’re being limited by the bits per second.

What uncapped really means is unmetered. In other words: We will allow you to consume bandwidth at an average of X bits per second, and we won’t actually (for billing purposes) measure how many bytes you push over the link. This means immediately that you pay for capacity instead of per byte. It’s also possibly to buy such unmetered solutions in an oversubscribed manner, for example, you can buy “gold” or “silver” transit from SAIX, “gold” means that you will have a contention ratio of 1:1 (meaning you will always be able to use your full capacity), or with silver you can get a contention ratio of 3:1 – which means that permitting that the other people aren’t consuming bandwidth you can burst up to your pipe size, but you’re only guaranteed of a third of it. Either way – 4Mbps of this is likely to make you understand why uncapped ADSL is a bad idea.

Share on Facebook

I realized that had I had an “upstream” git repository from which the changes came, I would simply have been able to “merge” this into my existing reports. And then came the idea … and then the simplification

The list of companies on the Quasar server is stored in /opt/quasar/data/companies, for each company there exists a .xml file. An extract of this file looks as follows:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE QuasarCompany>
<QuasarCompany>
 <name>Ultimate Linux Solutions CC</name>
 <uuid>...removed...</uuid>
 <version>2008-06-01</version>
 <dbtype>PostgreSQL</dbtype>
 <database>...removed...</database>
 <dataDir></dataDir>
 <status>valid</status>
</QuasarCompany>

Piet Ackermann pointed out the dataDir tag for me. This allows us to have a separate datadir for each company (well, I would probably create a separate datadir for each company and just symlink most of the folders straight back into the original data folder. With one exception, and that’s the reports folder. Now, for the above example, let’s assume we want to change the dataDir to /opt/quasar/ULS instead of the default /opt/quasar/data, so off we go:

cd /opt/quasar
mkdir ULS
cd ULS
for i in ../data/*; do ln -s $i .; done
rm reports

Note that I rm the reports symlink again. So now to get our “clean” copies into git (this assumes you haven’t already made modifications to reports, if you have, make backup copies and restore the clean, unmodified reports):

cd /opt/quasar/data/reports
git init
git add *.xml
git commit -m "prestine reports from 2.1.5"

Note that 2.1.5 is an example, it can be any older version, also, this is a log message, it can be anything. Once these are in git, we can proceed to clone them for modification purposes (can be done for as many companies as you desire):

cd /opt/quasar/ULS
git clone ../data/reports reports

This will create a reports folder inside /opt/quasar/ULS and copy everything from the “upstream” (/opt/quasar/data/reports) into it, along with some meta information stored in a hidden .git folder.

You can now edit your reports here. (or in the case of old reports, copy them in here). There are a few very basic operations now.

When updating reports, you should add them into the “slave” or “downstream” repository, and commit them. For example:

vi quote_print.xml
git add quote_print.xml
git commit -m "Added my logo"

This has the advantage of keeping a history. You can view the history using “git log”, it’s quite detailed.

After upgrading quasar new reports will overwrite those in /opt/quasar/data/reports, this isn’t a problem, in fact, it’s what we want, we now want to pull this into our little git system:

cd /opt/quasar/data/reports
git add .
git commit -m "Upgrade to version 2.1.6"

That’s it on the upstream side, and now to update the ULS reports:

cd /opt/quasar/ULS/reports
git pull

This will (usually) perform a clean import of the changes Linux Canada has made to the upstream reports into yours. If it doesn’t, git will inform you that a conflict has occurred. So let’s say for example we changed the same line that Linux Canada changed, then git will complain that it can’t auto-merge quote_print.xml. This is known as a conflict. Simply open the conflicting file in a text editor, look for the >>>>>> ====== <<<<<< conflict markers, edit as you see fit (you need to leave the file without any markers, as you would have wanted it to look after the merge). Now you need to "git add" the conflicting file, eg "git add quote_print.xml" and then you can commit the merge with "git commit".

A controved example follows, let's say in the original version Linux Canada had a file (called file) with one line it "this file has one line in it". Now this was committed in the repository at /opt/quasar/data/reports, at this point we cloned it, and changed the line to "customize". We also did a "git add file" and "git commit -m 'customization'". Now Linux Canada releases a new version of quasar, you go through the process above, and when you execute the "git pull" git spits this at you:

remote: Counting objects: 5, done.
remote: Total 3 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.
From /home/jkroon/tmp/a
   b6e12ff..457d983  master     -> origin/master
Auto-merging file
CONFLICT (content): Merge conflict in file
Automatic merge failed; fix conflicts and then commit the result.

As you can see, it says big and loud CONFLICT. It also says Merge config in file (which shows I’ve chosen my filename really badly). Opening the file in a text editor shows this:

<<<<<<< HEAD
customize
=======
this file only has one line in it
>>>>>>> 457d983d0405dac4617a6ca5f42299e491ce8a4c

We decide we want to keep our version, so we delete the <<<< marker, as well as ==== through to the >>>> lines. Now we can either use the git shorthand for “git add .” and “git commit” and simply do “git commit -a”. This will bring up a text editor with Merge branch ‘master’ of /opt/quasar/data/reports, indicating the Conflicts. You should probably add some text indicating what the cause of the conflicts was and how it was resolved. When you save and quit git will do what’s required.

It’s also possible to do this whole thing using branches, and it’s a shorter process, but there are more pitfalls. So I’ll run over it very quickly.

Do not bother making copies. What I’ve done just now is this. I’ve copied everything in /opt/quasar/data/reports to ~/quasar/reports/ in order to ensure that I’ve got backup copies. I then re-installed the same version of quasar to get fresh copies of the Linux Canada reports. Now comes the tricky part:

cd /opt/quasar/data/reports
git init
git add .
git commit -m "reports from 2.1.4"

So far so good. Now we “branch” this and re-import the backed up reports:

git checkout -b uls
cp ~/quasar/reports/* .
git add .
git commit -m "existing customizations"

All done. Now comes the careful bit though (and there are ways around this, so if you forget, know that git is ultra powerful and can save you). Before you update you should switch back to the master branch:

git checkout master

Once you’re done updating quasar, you can go back to your own branch:

git checkout uls

And now you can merge the updated reports:

git merge master

Resolve conflicts as per above.

Whichever way you choose, it is a good idea to familiarize yourself with git. I can highly recommend the book at progit.org, it’s very informative, but deals with git from a programmers perspective. The main points from my side:

* Always make sure you don’t leave uncommitted changes around, especially not before an update (git status).
* Use sensible changelog entries. Trust me.

Other advantages of this for me is that I get a revision history of what I changed and when (and depending on the details of your changelog, why). I can imagine that consultants like Piet will find this extremely useful for managing reports for multiple “hosted” clients.

Share on Facebook

#!/usr/bin/perl
 
$|=1;
while (<>) {
    if ($_ =~ /http:\/\/www\.google\.([^\/]*)\/? /i) {
        print "http://www.vuvuzela-time.co.uk/www.google.$1/webhp\n";
    }
#   elsif ($_ =~ /http:\/\/www\.google\.[^\/]+\/intl\/en_com\/images\/logo_plain.png/i) {
#       print "http://tauri.local.uls.co.za/bb.png\n";
#   }
#   elsif ($_ =~ /http:\/\/www\.google\.[^\/]+\/images\/nav_logo7.png/i) {
#       print "http://tauri.local.uls.co.za/nav_logo7.png\n";
#   }
    else
    {
        print $_;
    }
}

Then just add the redirect stuff in /etc/squid/squid.conf back:

url_rewrite_program /etc/squid/redirect
url_rewrite_children 10

And now you can hear whenever anybody is in need of assistance from google :p.

Share on Facebook

This begged a few questions (to which some of the answers is already present above). So a few bash notes (mostly obtained from the advanced bash scripting guide), firstly the objective, then two lines, firstly the semantics, then an example line:

Open a file for reading:

exec fd</path/to/file
exec 3</tmp/infile.txt

Open a file for writing:

exec fd>/path/to/file
exec 3>/tmp/outfile.txt

Closing a file descriptor:

exec fd<&-
exec 0<&-  # will close stdin

Writing to a file descriptor:

command >&fd
echo fneh >&3

Reading from a file descriptor:

command <&fd
read VARNAME <&4

Renumbering file descriptors (ala dup2(2)), depends on whether it’s for reading/writing, but basically you just open the new fd, and instead of specifying a filename you give it an existing file descriptor. So to “copy” stdin from fd=0 (standard) to fd=6 for whatever reason, and fd=2 (stderr) to fd=7 you can do this:

exec 6<&0
exec 7>&2

You can perform multiple exec actions in a single line, for example:

exec 3</tmp/infile.txt 4>/tmp/outfile 6<&0 7>&2

This is interpreted left to right. And yes, order does matter when for example doing this (saving stderr to fd 7, and redirecting the scripts stderr to /tmp/errout.txt:

exec 7>&2 2>/tmp/errout.txt

What I can’t seem to find is a bash way of doing pipe2(2) without using mkfifo (which has name prediction attacks on it). mktemp can’t create pipes for us, so we’re stuck with generating a random name, attempting to create it (remember –mode=0600). This still suffers from problems similar to mktemp in that the content can be high-jacked by users with read/write permissions to the pipe.

I’m still working on a sane way to enforce the lock though, and it’s looking more and more like I will need to create two fifo’s, and pass the filename of the lockfile as a parameter to the program, so something like (ignoring path name collisions and other failures on the fifo’s for the moment):

1
2
3
4
5
6
7
8
9
10
11
12
13
14

mkfifo /tmp/lockf-{in,out}.$$.fifo
trap "rm /tmp/lockf-{in,out}.$$.fifo" EXIT
# Note that exec will block waiting for the bash process
# preparing the lockf command to open the fifo's first.
# Order of redirects is crucial to prevent deadlocks.
lockf /var/run/lockfile.lock </tmp/lockf-in.$$.fifo \
    >/tmp/lockf-out.$$.fifo &
exec 4>/tmp/lockf-in.$$.fifo 3</tmp/lockf-out.$$.fifo
read LOCKED<&3
exec 3<&-
[ -z "$LOCKED" ] && echo "Error obtaining lock" && exit 1
... rest of code here ...
... to explicitly unlock:
exec 4<&-

This assumes the semantics from lockf is as follows:

* Takes a single argument, filename of the lockfile.
* Will open this file for writing (creating it if required).
* Issue lockf() on the file (entire file … probably empty anyway).
* Write a single line to stdout before closing stdout.
* If unable to obtain a lock, simply close stdout (probably by terminating).
* Block on a read from stdin, when stdin is closed (receives EOF) terminate (letting go of the lock).

The code for lockf below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
 
int main(int argc, char ** argv)
{
    int fd;
    char bfr[128];
 
    if (argc < 2) {
        fprintf(stderr, "USAGE: %s filename\n", *argv);
        exit(1);
    }
 
    /* parent process should correctly set umask
       (0066 is a good one) */
    fd = open(argv[1], O_WRONLY | O_CREAT, 0660);
    if (fd < 0) {
        perror(argv[1]);
        exit(1);
    }
 
    if (lockf(fd, F_LOCK, 0) < 0) {
        perror(argv[1]);
        exit(1);
    }
 
    fprintf(stdout, "locked\n");
    fclose(stdout);
 
    while (fgets(bfr, sizeof(bfr), stdin)) { };
    exit(0);
}

Initial (rudementary) testing shows that this does actually work. Surprise surprise.

I initially had two unlink(2)s in the code, which I realized introduces other races. If not in the invoking process then potentially in other processes. These locations was if we opened the file and then failed to obtain the lock (note that I don’t set alarms but I don’t block signals either, so there are other reasons the system call may get interrupted). The other was just before the final exit, with the lock still held. This opens race conditions as follows, for the failure case (line structure is P:action, where P is process indicator, a number, 1 is us, 2 + 3 is others, and action is a logical action):

1:create lockfile
2:open lockfile
2:lock lockfile
1:lock fails
1:unlink lockfile
3:create new lockfile
3:lock new lockfile

In this case both processes two and three will think it has the lockfile. In the success case:

1:create lockfile
1:lock succeeds
2:opens lockfile
2:blocks waiting for lock
1:unlinks lockfile
1:closes lockfile
2:lock succeeds on already open fd
3:creates new lockfile
3:lock succeeds

In this case again we have both 2 and 3 with the “same” lock.

In the case of portage this probably doesn’t matter too much seeing that “1″ should complete in downloading the file, and the lock file is mostly for in-process stuff, for it’s background fetching. There is (to the best of my knowledge) no concurrent compiling locks implemented, albeit, there probably should be.

As it stands, strictly speaking this is a bug in portage. The moral of the story is that a lock file should never, ever, ever, be removed. Ever.

Share on Facebook

So off I go, opening the portage code I looked at six years ago to just refresh my memory on how portage does the lock. It basically creates the file by opening it for read/write, then issuing the lockf function in the fcntl module. Essentially it’s a fresh open so the file position is 0, and the len isn’t passed (probably defaults to zero, indicating infinity, ie, the whole file). It attempts to take an exclusive lock, albeit an advisory (ie, the kernel will keep the data for us but it won’t actually enforce it, that’s what mandatory locks are for).

In the kernel there are two locking functions, fcntl (with operation F_SETLK) and flock. The man pages presents some cryptic information ((2)flock, (2)fcntl, (3)lockf). So I go off and read the Documentations/filesystems/locks.txt and mandatory-locks.txt files in the kernel sources. This reveals that flock was once upon a time (kernels older than 1.3.x from the looks of it) implemented flock on top of fcntl. In 1.3.x the flock emulation code was swapped out for proper flock implementation (compatible with BSD). The problem now is that the two ignore each other, completely. But that, from the looks of it, seems to be the intended behaviour.

Now, looking at (1)flock you will note that bash gives us flock semantics (util-linux-ng @ sftp://ftp.kernel.org/pub/linux/utils/util-linux-ng/ trictly speaking). Which means a snippet like:

exec 4>/usr/portage/distfiles/.fneh.tar.gz.portage_lockfile
trap "rm '/usr/portage/distfiles/.fneh.tar.gz.portage_lockfile'" EXIT
flock -x 4

Will not cause portage to block until we terminate. Quite the contrary, portage will still just proceed to grab the lockf version and continue on it’s merry.

Some (ok, not just some, a lot actually) googling finds us http://rpm.pbone.net/index.php3/stat/4/idpl/5182919/dir//com/bash-builtin-lockf-0.2-alt1.i586.rpm.html – which based on the name looks really, really promising. Until you download the source RPM and after examination it reveals this actualy looks like the current bash built-in flock version 0.1-beta1. Except the packages is all wrong from the looks of it. This essentially turns flock into a bash builtin.

This leaves me with some possible solutions:

1. Learn some more python, hook into the portage package and use their code to base torpage on top of.
2. Take the referenced code above as a basis for writing a lockf variant (shouldn’t be too hard).
3. Take whatever partition houses /usr/portage/distfiles/ and enable mandatory locking.
4. Alter flock(1) to add an option for using lockf instead of flock.

Option three looks like the simplest solution at first glance, however, it also won’t work any better than what I currently have. Basically once mandatory locking is enabled I thought of just trying to write one byte to the file – this will in the portage-has-the-lock case cause torpage to block, however, I still won’t be able to cause portage to block. Option 1 is probably the right thing to do but also the least attractive imho (almost complete rewrite of torpage). Or possibly invoke emerge/ebuild to fetch the files for the package. This however re-introduces a very old bug I had in portage. The files that needs to be fetched varies based on architecture and USE flags. This is problematic. In other words, I’m not even sure whether option 1 is viable.

In essence it seems option 2 and/or 4 is the most feasible in short time. The question between 2 and 4 is whether or not a single option to the existing flock is better, or whether one should rather write a separate utility. One should note that fcntl G_SETLK doesn’t differentiate between exclusive (write) and shared (read) locks. For this reason a separate utility may be better, on the other hand, a flag -f (for fcntl) may well be a simpler solution, and may well be something that could possibly be accepted upstream (given some motivation, which I don’t think will be too difficult considering the context of this discussion). On the other hand, cloning the utility and modifying allows me to package it straight into torpage (until upstream merges and releases) as an immediate solution. The advantage of a simple patch is that all the other functionality (-o, -c etc …) comes for free even though I personally only really care about the file-descriptor based case.

Based on the code it looks like flock() on an fd applies to the group of processes sharing the file description. To be exact (ignoring errors):

flock(fd, LOCK_EX)

is equivalent to:

pid_t p = fork();
 
if (p == 0) {
     flock(fd, LOCK_EX);
     exit(0);
}
 
waitpid(p, NULL, 0);

Albeit crap slowly.

This needs to be confirmed by testing, and then it needs to be confirmed that the fcntl based version has the same semantics. This is simpler than one would initially imagine. One can use bash to perform the initial testing against flock. Basically when bash executes a command it implicitly does the fork for us, so what we need is a command that has similar semantics to flock. In this case a program taking a single parameter, blocking for the lock, returning 0 if, and only if the lock was successfully obtained:

#include <sys/file.h>
#include <stdio.h>
#include <stdlib.h>
 
int main(int argc, char **argv) {
    long int fd;
    char *eon;
 
    if (argc < 2) {
        fprintf(stderr, "USAGE: %s fd\n", *argv);
        return 2;
    }
 
    fd = strtol(argv[1], &eon, 0);
    if (*eon) {
        fprintf(stderr, "USAGE: %s fd\n", *argv);
        return 2;
    }
 
    if (flock(fd, LOCK_EX) == 0)
        return 0;
    return 1;
}

This just needs to be compiled using something like “gcc -o takelock takelock.c” (assuming the content is stored in takelock.c) and placed on a partition mounted with exec perms. Now, open up two terminals and in both execute the following command:

exec 4>/tmp/lockfile

This will open /tmp/lockfile for writing in both terminals, attached to file descriptor number 4. In the one terminal now execute “flock -x 4″, which should terminate with exit code 0 (can be confirmed with echo $?). In the other you now need to execute our takelock executable (./takelock 4) which should block until you type “flock -u 4″ in the other terminal. To release the lock in the takelock terminal you can also use flock -u 4.

Updating the the code above to use lockf leaves us with this snippet:

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
 
int main(int argc, char **argv) {
    long int fd;
    char *eon;
 
    if (argc < 2) {
        fprintf(stderr, "USAGE: %s fd\n", *argv);
        return 2;
    }
 
    fd = strtol(argv[1], &eon, 0);
    if (*eon) {
        fprintf(stderr, "USAGE: %s fd\n", *argv);
        return 2;
    }
 
    if (lockf(fd, F_LOCK, 0) == 0)
        return 0;
    return 1;
}

Running this first in the one terminal and then in the other reveals that the semantics differ and doesn’t work. We’ll be able to enforce the -c semantics from the flock command, but not the fd semantics. Which is a shame really, but there is probably good reasons for it behaving this way. Again, this can be confirmed by adding a sleep(10); to the lockf if statement. Something like:

    if (lockf(fd, F_LOCK, 0) == 0) {
        fprintf(stderr, "Got lock!\n");
        sleep(10);
        return 0;
    }

And to then run both, about 5 seconds apart. If this still doesn’t convince you, not much will.

This still allows us to implement it as a bash builtin, or to rewrite portage to take the lock and have a sub-shell actually doing the download. Either way, that’s more work than what I’m willing to do tonight.

Option 4 is thus eliminated (for the most part) seeing that lockf’s semantics isn’t similar enough. Option 2 may still be viable, but hard to motivate. The altlinux code suggests that one can modularly add bash built-ins – perhaps this is still an option, except that adding a global built-in is not a clean solution, so I’m in two minds as to whether or not to attempt it. A quick grep of the bash-4.0 source code however also suggests that altlinux hacked bash to add the required capabilities to load builtins as modules.

*cheers*. Here’s to hoping that this helps someone else looking to understand the difference and nuances of these two rather confusing locking mechanisms.

Share on Facebook

I’m not sure whether this is a result of too little testing, total ignorance or just incompetence. Either way, it would seem it’s a bit of a race condition, and hits something similar to what we in the office refer to as the “connection tracking bit bucket”. Basically it seem most connection tracking implementations (when combined with a state full firewall such as that used by Vodacom – as per their own admission in their last letter to me) results in certain flows being prematurely marked as “invalid”. In particular in the example that Kevin has captured for me the server ends up being the first entity to send a GRE packet, this then gets (or got, seeing that it’s fixed again) intercepted by the firewall, perceived as an inbound connection to the client and the uni-directional flow gets marked as invalid. When the client now sends GRE traffic to the server this gets allowed, but the return traffic still bites the “invalid” mark. I can only speculate as to the exact state (seeing that Vodacom doesn’t reveal exactly what software they are using – probably proprietary anyway) of things, making it difficult. This I will attempt to speculate as objectively as possible (not always easy).

Seeing that there are two entities involved in this dump, and I want to do a side-by side comparison, some ASCII art is in order. Essentially three columns being used, the sending agent will indicate what is being sent, and if it was received by the destination I’ll mark that column with an ACK. I’ll also add (R) to retransmits on the sending side. The ISN mods still applies to the TCP connections, however, the data itself isn’t being tampered with in this case. Note that in the GRE traffic case there are still ACK packets being sent by the server, these ACK packets however goes lost (as indicated in the packet sequence).

Client                Direction  Server
SYN                        ->    ACK
ACK                        <-    SYN
PPTP (Start Req)           ->    ACK
ACK                        <-    PPTP (Start Resp)
                           <-    GRE (PPP-LCP Conf Req)
GRE (PPP-LCP Conf Req) (R) ->    ACK (goes lost)
GRE (PPP-LCP Conf Req) (R) ->    ACK (goes lost)
... a few more of these ...
GRE (PPP-LCP Conf Req) (R) ->    ACK (goes lost)
PPTP (Call Clear Req)      ->    ACK

Once the Call Clear Req is received TCP/IP teardowns happens, surprisingly without the flurry of injected RST packets I’ve growned accustomed to, just a single out-of-order delivery between one ACK and FIN/ACK packet.

What I would want (not sure what resolution they picked) is for them to either perform a routine inspection of the PPTP control traffic (specifically the Start Request and Start Reply packets) to determine the GRE traffic parameters (based on what I can see, just mark the fact that GRE is to be expected between the two given end points) and allow that traffic, or, stop this firewalling nonsense. It’s only the Cellular “ISPs” performing actions such as these. The arguments for providing this protection is sound. But then it needs to be done sanely. For the most part I’ll have to admit that the firewall works and doesn’t cause too many problems.

Seeing that the problem has been resolved by Vodacom, I’ll let it rest, for now.

Share on Facebook

This entry is written out of some slight frustration – yet again with Vodacom. We run a PPTP VPN firewall for a (large) number of clients. As of approximately a week ago these clients are having severe issues connecting to the VPN specifically when connecting from the Vodacom 3G network. Also the reason why I went back and re-checked these ISN adjustments that were being made at the time. It seems, however, that they are still making these (justified or not) adjustments. They have now gotten the “systems provider” to fix up the sequence number adjustments inside of selective ACK options inside TCP segments fortunately.

My frustration also grows from the fact that the solutions the Vodacom call center are now giving me are in direct contradiction to what I was being told about a year and a half ago. At the time they said I need to get off the InternetVPN APN and onto the normal Internet APN because it no longer uses private IP addresses and contains full support for VPNs etc etc … … and now I’m being told to revert back to the InternetVPN APN. You try and explain how to do this to 100+ different clients, how and why it’s required and more importantly you consistently get some snotty comment down the lines of “yea yea, you’re just using this as an excuse for breaking this and that”, so explaining to them that it’s Vodacom that changed something (whilst accurate) isn’t really an option as it’s just perceived as instigation against a “company bigger than yourself”. The end result? Vodacom yet again gets away with breaking stuff, and the small guys takes the fall.

The other solution that I’ve been offered is to put the server on the unrestricted APN. Didn’t you listen the first time round? The server ISN’T on 3G for, amongst others, the fact that you’re messing with TCP/IP options, it’s damn expensive at the traffic volumes we’re requiring, not to mention slow in comparison with wired connections such as ADSL (yes I know 7.2Mbps is more than 4Mbps, I don’t care, it’s NOT comparable, be it due to latency, high packet loss or jitter, I’d rather have a 512Kbps ADSL than any kind of 3G connection @ 7.2Mbps).

Anyhow, now I return to figuring out what is causing breakage this time.

Share on Facebook

MY NAME IS DARTH VADER, THE APPRENTICE TO THE SITH MASTER

SITH MASTER, FORMER EMPOROR OF THE GALAXY, FORMERLY SENATOR PALPATINE, WAS RECENTLY KILLED BY OUTLAW HAN SOLO. I HAVE JUST RECENTLY BEEN INFORMED BY MY LATE MASTERS’S BANKING OFFICER THAT THE OLD MAN OPERATED A SECRET ACCOUNT WITH THE BANK INTO WHICH A TOTAL SUM OF SEVEN BILLION EIGHT HUNDRED MILLION THREE HUNDRED AND TWENTY ONE THOUSAND GALACTIC CREDITS (7,800,321,000) WAS TRANSFERED AND CREDITED IN HIS FAVOUR. I HAVE NOW BEEN ADVICED BY THE BANKING OFFICER TO SEEK IN CONFIDENCE A FOREIGN ACCOUNT INTO WHICH THIS FUND COULD BE TRANSFERED FOR SAFE KEEPING TO AVOID A LEAK TO THE ACCURSED JEDI.

IT HAS BEEN RESOLVED THAT 25% WILL BE YOUR SHARE FOR NOMINATING AN ACCOUNT FOR THIS PURPOSE AND ANY OTHER ASSISTANCE YOU GIVE IN THAT REGARD, 5% HAS BEEN SLATED FOR REIMBURSEMENT OF ALL LOCAL AND INTERNATIONL EXPENSES WHICH MAY BE INCURED IN THE TRANSFER PROCESS, AND 5% HAS BEEN CONCEDED TO THE LOCAL BANKING OFFICER HERE ASSISTING AND FACILITATING THE TRANSFER. FINALLY 65% WILL COME TO MYSELF AND FAMILY AND A GOOD PART OF THIS SHALL BE DIRECTED TOWARDS EXECUTING HIS WILL, WHICH IS TO BUY SHARES AND STOCK IN FOREIGN COUNTRIES AND TO SECURE HIS CHILDREN’S FUTURE. TO FACILITATE THE CONCLUSION OF THIS TRANSACTION, IF ACCEPTED, DO SEND TO ME PROMPTLY BY E-MAILLING THE FOLLOWING:

1.NAME AND ADDRESS OF YOUR BANK. 2. TELEPHONE AND FAX NUMBERS THROUGH WHICH YOU WILL BE CONTACTED PROMPTLY BY ME FOR THE COMPLETION OF THIS TRANSACTION.

PLEASE PROMISE ME YOU WILL ASSIST ME, AND REMEMBER TO KEEP THIS TRANSACTION VERY CONFIDENTIAL,NOTE THERE IS NO RISK INVOLVED. OR I WILL FORCE CHOKE YOU.

I LOOK FORWARD TO HEAR FROM YOU.

THANK YOU AND GOD BLESS.

DARTH VADER

Share on Facebook