Right, so I moved from downloading 50+ curl handles using the simple interface to using the curl_multi variant, and suddenly I bumbed into a few problems, firstly my downloads were incomplete by the time the code stopped looping (reference example code!) and the web server took some strain trying to serve 50+ severely CPU intensive queries at the same time (an individual download can take up to a few minutes depending on various things, thus why I need concurrency – multiple cores – can just as well utilize them).
(more…)
Archive for the ‘Scripting/Programming’ Category
PHP curl – controlling multi concurrency
Wednesday, August 25th, 2010
bash file descriptors, pipes and … lockf.
Monday, June 14th, 2010Bash and long-standing file descriptors is a pain, to put it politely. For example, I’d like to spawn a sub-process into which I can “write” stuff at arbitrary points in time by doing echo fneh >&4 … sounds simple enough right? But how to spawn that sub process?
(more…)
bash, lockf and flock
Sunday, June 13th, 2010So after having had enough I decided it’s time to make torpage integrate better into the portage locking. The current state isn’t bad, it’s just not complete. The fact of the matter is that it is possible should torpage initiate the fetch that portage will (permitting it attempts to fetch the same file) ignore the fact that torpage is busy downloading the file and kick off a dual download. What’s happening is that torpage (currently) treats the mere existence of the file as a lock, whereas portage takes out a fcntl (lockf) based lock on the file. Using the existince/lack of existence of files as the basis for a lock is a bad idea anyway, so I decided it’s time I took what I learned in the last six years and fix it. (more…)
Is asterisk ready for a fork?
Wednesday, January 27th, 2010It is with a heavy heart that I start writing this entry. I’ve always enjoyed working with asterisk after overcoming an initial (unfounded) fear of it, in part due to the words from a relatively brilliant ex-coleage of mine from my days working at Techteam (CS dept at the University of Pretoria). It’s a mind-bending experience, to say the least, fun and requires a decent amount of intuition. However, recently I’ve been realizing that the current state of affairs is oh so very far from ideal. (more…)
