jkroon

So recently I started bumping into an issue where I would see a buildup of traffic in the RX queue of asterisk’s SIP port 5060 (udp bound). After some scavenging of the code I quickly came to realize that asterisk only processes a single incoming SIP request (or response) at a time. So I cooked a rather crude patch (that I for the shame of it won’t share here) in an attempt to figure out what went wrong.

I set an arbitrary limit to log SIP requests taking longer than 100ms to process and it very quickly became apparent that REGISTER packets were the only ones triggering this. I dug deeper and logged the time taken for each and every SIP packet, and it was quite scary to learn that the far majority of packets was dealt with in under 100us (yes, micro-seconds), so why was REGISTERs such an order of magnitude slower?

A few questions was raised, and some flames thrown in #asterisk and the blame went everywhere except the total lack of parallel processing in chan_sip, everything from heavy network IO, reverse DNS lookups right through to the OS got blamed. As it turns out, the astdb should have took it. I got one of my clients whose an absolute wizzard with MS Excel to graph the badness of the RX queue for me and it became clear that things are worse between 22:00 and 2:00 (when our backups run), but that it was nowhere near perfect at other times. We were at times losing requests simply because there wasn’t sufficient buffer-space for the socket!

We’ve logged numerous cases of SIP REGISTER requests taking over 300ms, but typically between 150 and 250ms when the system was not under heavy IO load, but up to 2.6s during backup runs. Needless to say, this explained a few strange things we’ve experienced over the last few months.

We tested the theory by moving astdb.sqlite3 onto a tmpfs mounted filesystem, and immediately the problem went away, and processing of register packets dropped to within reasonable (<1ms) ranges. After a quick perusal of the contents of the astdb we decided that the only important information we keep in there is SIP and IAX/2 registration information, and that should the server crash (OS level) we're willing to lose that data. Our SIP clients mostly use SRV records so they can easily enough fall back to one of our other servers, and our IAX/2 clients are typically configured to register to multiple servers on our end, as well as to dial out via multiple servers, so they should not be a problem either. Our biggest concern is still the dropped calls when this happens. WARNING: Utilizing the procedure outlined below may very well end up losing your asbdb data if you’re not careful!

So, to speed up astdb in order to give reasonable performance to asterisk’s chan_sip, this is what we did (NOTE THE WARNINGS ABOVE):

Firstly, we created a folder /var/lib/asterisk/astdb. We then added the following line to /etc/fstab (broken up for ease of reading):

none /var/lib/asterisk/astdb tmpfs \
  noatime,nodev,nosuid,noexec,mode=0750,uid=asterisk,\
  gid=asterisk 0 0

Then we issued “mount /var/lib/asterisk/astdb”. In /etc/asterisk/asterisk.conf, you need to add this:

[directories]
astdbdir => /var/lib/asterisk/astdb

And then came the patient part, first we need to wait for asterisk to have no calls (ie, very, very late at night), then we did this:

cd /var/lib/asterisk/astdb
/etc/init.d/asterisk stop
cp -a ../astdb.sqlite3 .
/etc/init.d/asterisk start

After this your asterisk should be running again, but using astdb.sqlite3 on the tmpfs instance, which should result in MUCH (orders of magnitude, at least 200x in our case, from ~150ms best case to ~750us worst case) faster REGISTER request processing.

Now, if you want your sqlite database to survive reboots, then I suggest putting these commands before your asterisk start in your init script, and the inverse after asterisk stopped:

# Before start:
cp -a /var/lib/asterisk/astdb.sqlite3 /var/lib/asterisk/astdb/

# After stop:
cp -a /var/lib/asterisk/astdb/astdb.sqlite3 /var/lib/asterisk/

You may also want to periodically make a live copy of the database. I’m not sure what’s required from sqlite’s perspective to make this transactionally safe and to not make a copy corrupted copy. Consult the sqlite documentation most likely. For our purposes starting with a clean DB after every boot is good enough.

This entry was posted on Thursday, August 1st, 2013 at 10:10 pm and is filed under VoIP. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.