Farewell Old Friend

Today is March 17, the day for the wearing of the green.  But today I'm in a more sombre mood as I've just had to say goodbye to a mate of almost 17 years.  He died today in the arms of my wife, after a melanoma in the mouth finally took away his quality of life.

Clancy was an Australian Terrier.  We used to quip that Aussie Terriers were just German Shepherds that had shrunk in the wash.  The same glorious zest for life, the same loyalty, just in a more compact package. Right up until yesterday he was still trying to jump up to welcome his dinner, even if he didn't get his front feet more than an inch off the ground, the will was still there.

Clancy brought joy and comfort to everyone he met.  When he'd got a bit older he was my aging mother's favourite as he would like nothing better than to snuggle up to her and she would commiserate on the process of aging.  As a youngster it was like living in a pinball machine, he would literally bounce off walls.

When we got Clancy I spent time learning by heart "Clancy of the Overflow", a classic Australian poem by Banjo Patterson.  I don't think I'll ever forget him or that poem.

So this evening we've poured two fingers of whiskey each, and toasted that wonderful creature that both of us will miss dearly.

Farewell Old Friend.

Why your next DevOps should be an Australian

I often see job postings that allow remote workers, but only from Europe or North America.  This ignores a pool of talent that has advantages that you may not have thought of.  Australia is often seen as being in an "inconvenient" timezone, yet if you want to cover the "swing shift" between North America and Europe you need look no further than Australia.

When I get up in the morning it is in the US afternoon, when I leave work it is the European morning.  So I get to cover those times that others do not - or would be expensive to do so.  I have been doing this "swing shift" for 10 years now, and providing value for the companies I contract to.  Why don't you consider Australia for your next hire.  We even speak English!

In memory of Eileen Mary Donnison (nee Cain)

Today, just before Mother's Day, my mother died.  It was fitting in some ways, she hated Mother's Day as she thought you shouldn't need a special day to remember your mother - that should be every day.

My mother was born Eileen Mary Cain in the period between two world wars, the daughter of a man touched by war - he was a TPI (Totally and Permanently Incapacitated) pensioner after encountering a mustard gas bomb in the first world war.  She was a teenager for most of the second world war where she saw her older brothers head overseas as boys and arrive back vastly changed men - one of whom never really recovered from the experience. Shortly after the war she married my father, John William (Bill) Donnison. Her home town of Rye only saw electricity connected the year she was married.  Her life has covered an amazing amount of technological and sociological change.

At heart my mother was an anarchist.  No, I don't mean she threw bombs and tried to bring governments down. Instead she was a ferociously intelligent woman in a time when it was not seen as an asset.  In her teens Eileen received a scholarship to university, but given her age (and no doubt her gender) it was decided she should forgo it.  What a pity. This was a woman who loved learning and would have been a great teacher.  My love of books I attribute directly to her. I remember times when, while cleaning the house, she would come across a book she hadn't read or remembered had a great quotation in it, so the task at hand was put aside for the far more important one of  reading.  Standing in a queue I had seen her toe a piece of paper that had fallen to the ground so she could read it.  From the back of cereal packets to the classics, nothing was considered to be too high- or low-brow to be read.  I think she would rather read than eat.

One of my favourite stories that mum often related was of her as a young girl sitting at the dinner table reading the newspaper and asking her father "What is rape?".  Her father replied "Its an oil seed." to which the young Eileen responded "Don't be silly, Errol Flynn has been charged with it!".  Her father, rather nonplussed, spluttered and told her to put the paper down and finish her dinner.

Mum used to tell of her horror when recently married and having to vote in a council election, talking to the other women there.  She naturally asked who the women thought would be a chance in the election only to be told that they didn't think about those things, their husbands told them who to vote for.  Mum was incensed at this and told them in no uncertain terms that nobody told her what to think or do. To her marriage was a partnership, but she still insisted on keeping her own bank account, and when things got tough it was mum's foresight that meant there was money to pay the employee's Christmas bonuses.

Times were occasionally very tough. She used to tell of the time that the kids were complaining that there was gristle in the meat served for dinner, because she could only afford pet mince, and she didn't want her children to go without meat.  Mum and dad made sure all of their kids got the best education they could afford, despite the cost to them personally.  It was mum's insistence that education was the way to a better future.

Eileen couldn't abide those that thought of educating women as being a waste. To mum, educating a woman meant educating a family.  The saying "Give a man a fish, feed him for a day. Teach a man to fish, feed him for a lifetime" could be extended to "Teach his wife to fish, feed the generations that follow".  To this extent mum was a feminist. Not the bra-burning type, and she thought that feminism was basically giving women the right to work two jobs, but she nevertheless saw the empowerment of women as not only important, but vital.  As a girl mum would have to wash the work clothes of her father and brothers in a time when that meant boiling up a copper and running them through a mangle, and she was adamant that she would not allow that to happen to any daughter of hers, nor would she allow her sons to put that burden on others.  Each of us seven boys was taught to cook, clean, wash, iron, mend, knit and sew.

This is not to say my mother was the dour, earnest type. Not a bit of it. There were many a time that we were all rolling around in laughter at some shared joke, mum usually the instigator.  Eileen also would find fun in the most unusual places.  Dad getting inappropriately angry at something would at times cause mum to fall into gales of laughter at the sense of absurdity of the situation. While in the short term this would make dad angrier, he would eventually succumb to the hilarity of it all and join in. Mum loved The Goons, Spike Jones and his City Slickers, Dave Allen and Val Doonican.

Eileen loved life and loved people.  She was intensely interested in people and was a great listener.  Once I was waiting outside a supermarket while she ducked in for a few items, and coming out she told me all about the life of the woman who followed her out of the supermarket - someone she had only just met and started talking to in the checkout queue.  She would also talk to anyone - as far as mum was concerned people were people, and you treated them as you would like to be treated.  She could be found as easily chatting with a group of school children as she could with a bunch of Hell's Angels.  There was just something about her that led people to want to talk to her.

Mum's love of people was even more profound when those people were family.  Even towards the end  as her mind faded and she had no idea who I was she would proudly talk about her grandchildren and great-grandchildren. To her family was everything.  It distressed her no end at the occasional disharmony in her children, as to her family was inviolate and love absolute.  That love could be a burden on her though. In her early marriage she suffered with an interfering mother-in-law and vowed she would not do that to her children. This often meant that she held her tongue and suffered vicariously the pain of her offspring when their marriages hit a rocky patch or their children hit those troublesome years as they invariably do.  Sometimes I think it may have been better had she not been so hands-off, but then you had to admire her commitment to her ideal.

Mum was a communicator. She would say that you should be able to talk on any subject for 5 minutes without hesitation or repitition. As a relbellious teenager I remember replying "Bullshit!", so without so much as a blink of the eye she proceeded to start on that topic until we both couldn't stop laughing. I have had the privilege of reading her rare efforts in poetry that was both beautiful and poignant. We could talk about any subject, and often did.  I remember talking to her one day about either menstruation or masturbation (I don't recall which) and my father walked in, blushed and sputtered and walked straight back out. Mum shrugged and continued the discussion.

I learned a lot from my mother. Much more than I can ever repay. I learned tolerance, compassion, strength of character, tenderness, fidelity. I learned to read, write, drive, cook, knit, sew, draw, laugh, cry. More than anything I learned to love. To love life, to love others, to love myself. There is nothing else I need to know.

Eileen Mary Donnison (nee Cain) 28th May 1928 - 3 May 2014.

May the road rise up to meet you.
May the wind be always at your back.
May the sun shine warm upon your face;
the rains fall soft upon your fields 
and until we meet again, 
may God hold you in the palm of His hand.

(traditional gaelic blessing)

Arduino antics

I had heard of Arduino for a while, but it wasn't until the 2012 Linux.Conf.AU in Ballarat that I got my hands on one.  Freetronics had come up with an Arduino compatible USB stick called the LeoStick and they provided a free one to each conference attendee.  That first night I had the thing out of the packet and hooked up doing a simple blink demo.  I was impressed with the development cycle, the IDE and the ease of development. These powerful little building blocks are coupled with a huge community of developers and users and there are code samples and libraries for a plethora of interconnects.  I had dreams of doing great things with the Arduino. It then languished in the back of the electronics box for nearly 12 months.

At the moment we in western Victoria are suffering from a dry spell that is more severe than many of the drought years, although the government refuses to call it a drought as that would imply having to provide drought assistance.  So far we've been running at less than a quarter of the average rainfall with no signs of the situation improving.  Naturally thoughts come to handling water - as it is now becoming a scarce resource.  So I dusted off the LeoStick and the Arduino IDE and started tinkering.

For years I had a greywater system that I'd developed a PIC microcontroller circuit for, which included a web interface and several options for handling pumping to various outlets (like a toilet cistern and garden beds).  We then moved and the needs for the unit became a lot simpler, and  the pump driver triac died so I converted to a simple latching relay circuit, with the idea that I'd eventually update the microprocessor controller and re-install it.  With the ongoing "unusually dry period" it seemed like a good project to get something real under my belt.  As I knew exactly what I wanted to do the first cut of the code was done in a matter of hours, and I only had to wait for some components to arrive to get the hardware together.

I also decided to put together some tank sensors on our main water tanks so we didn't have to go tapping on them and listening for the change in sound where the water level sits.  I had looked at a lot of sensors out there but I figured that with an Arduino it shouldn't be that difficult.  I found this capacitive sensor library that was designed for touch sensitive plates, but could easily be adapted to measure any small capacitance.  Coupling this with a length of wire placed in the tank and a resistor, I have a reasonably reliable tank sensor.  Just to make sure I've added a float sensor at the top of the tank just below the overflow level so that I can use it as a pump cut-out.  Since then I've found some reasonably priced pressure sensors that give a far more consistent result than the capacitive sensor.

Next project was not really water related, but something that has irked me for some time.  We bought a heap of Arlec solar powered motion sensor lights a few years ago, and they worked reasonably well for a few months, but then decided to die.  Pulling one apart I found the battery case had cracked and looking further I found that the 6 volt battery was connected with just a blocking diode to the solar panel, which in full sun was generating more than 12V.  This might have worked OK in the suburbs with a few trees giving partial shade, but out here we have pretty much clear skies for more than 90% of the year.  Coupled to this was the use of a 6V 10W halogen globe.  Not really easy to replace, and the light output wasn't fantastic. I had tried putting in a charge controller and that helped, but I realised that with a few components I could put together something far more useful.

I'd purchased 5m rolls of led strip lighting that runs off 12V and you can cut it into short lengths and, with the adhesive backing, stick it up where you need it.  I'd used that for internal lighting in a coolroom and had some left over.  Coupling this with a few motion sensors (under $3 each from China), a few phototransitors and an Arduino I came up with a lighting solution that is far better both in light quality and coverage and in detection.  Almost 12 months later and it is still going strong.

Now with all of these projects I decided that knowing what was going on was important, so they all have support for XBee radio units to communicate with a control system.  This allowed me to abstract a lot of code into my own library for handling the XBee communications, configuration, remote control and a few other useful functions, leaving the sketches (as Arduino programs are called) very small and focussed only on the "business logic" rather than the lower level mechanics.   While useful, I found that unlike most of the components I had looked at, the XBees were increasing in price to the point where it was becoming uneconomic to use, and frankly their performance was pretty dreadful.  In clear line of site using peer-to-peer they work pretty well, but in a mesh network with things like trees and walls in the way they are pretty useless.  In one test I couldn't get the controller to talk to another XBee that was within 3m of it.

I have looked at alternatives and there are some promising units out there.  With some more research and tinkering I should have an answer to my communications issues.

Anyway, should you be interested in this sort of stuff, most of my code is now on GitHub in my "Smallholder Sensor Framework" repository.

Anatomy of a Failure: CFA Website


As a result of disasterous fires in our state in 2009 known as Black Saturday there was a Royal Commission into the fires, the management of them and the deaths that occurred as a result.  Coming out of this was a set of recommendations and changes were instigated by a number of parties.  One of the main parties of interest in the handling of fire services in the state is theCFA (Country Fire Authority).

The CFA has to be congratulated for at least the intent of some of their initiatives post Black Saturday.  One of these is their Fire Ready app for mobile phones, and another was a revamp of their website.  A lot of time, and presumably public funds, went into the developments that resulted in a catastrophic failure of the website and app. So what went wrong?

Initial reports suggest that capacity planning was at issue, with the statement made that they had planned for up to 350 hits per second, but the site received 700 hits per second. Much of the blame was placed on the fact that the app was connecting to the same server as the main site, and that part of the solution was that they put the app on its own server. Note the singular 'server' there.  I'm not sure if this was just a reporting problem, trying to make it easier for journalists to understand, or we are really talking about an under-resourced system that is supposedly meant to assist people in the times of greatest need and stress.  So, aside from the fact that both were on the same infrastructure (let us assume there is actually more than one server there), what can we determine about the possible problems that these 700 hits per second were causing.

Firstly, looking at the site details, the app now uses osom.cfa.vic.gov.au, which appears to be fronted by a Squid proxy cache, often used to act as a web site accelerator, so there could be more than one server behind that.  The main website, www.cfa.vic.gov.au is behind a F5 BIG IP appliance, which in its base form is a load balancer, meaning that unless they are spending money for the sake of it, there is at least more than one server providing the service.  The BIG IP appliance is quite a useful beasty, and can provide web site acceleration, using compression and other techniques to make life easier.  So why was there a problem?  And apologies for me starting to get a bit technical from here on in.

Let's first define what a "hit" might be.  When you load a web page, like www.vic.gov.au, you might think that by the time it finishes loading that is one "hit".  And you'd be wrong.  A web page is made up of a number of files. Each image on the site is likely to be a file, as is each stylesheet and javascript file. Each of these files is requested separately and each of these requests results in a "hit".  Taking a look at the structure of the CFA site shows that there are a number of unnecessary files on there.  For instance there are 6 javascript files that could be combined into one.  There are images called from CSS that could be combined into "sprites".  Just performing that alone would change the number of hits the site was getting to load a single page, and in doing so increase its capacity - without resorting to extra hardware or tweaking networking stacks.  Indeed it looks like you could increase the number of page views supported by a factor of at least three, simply by resolving these extraneous hits.

There is more intersting stuff under the covers.  There is no compression used on any of the assets (page data, images, scripts, stylesheets) so that the amout of data having to be sent down the wire is far greater than it needs to be.  A conservative estimate suggests that the site could handle twice the current load simply by turning on compression.  This is usually just a configuration change in the base web server software.  All modern browsers support compression, and it is a rookie mistake not to be using it.

On the same front, most browsers also try and help by actively caching items that don't change often, meaning the next time you go to the site it is far quicker to load as the browser already knows about it.  For this to work you need to make sure your web server is set up to help out.  The CFA site is not.  There are two factors here, ETags and Expire headers.  ETags are supposedly unique ids for assets that allow the browser to check that it has a copy of this and therefore doesn't need to re-download (providing it hasn't expired).  The trouble with these is that if you have more than one server supplying the same data - as you would behind say a BIG IP appliance,  these ETags are likely to be different for the same resource, meaning that the browser thinks it needs to download the item again as it has changed.  The CFA site has ETags turned on.

Expiry headers tell the browser how long it should hold an item in its cache before checking for a new version.  For items that don't change much (like images and stylesheets and scripts) it is common to set a "far future" expiry header, say a year or even a month into the future.  This means the browser doesn't need to worry about these items after the first load.  The CFA site doesn't use Expiry headers.  Fixing these two problems could see at least a 50% if not doubling of the capacity of the servers.

Now, without looking at hardware, operating system or web server software, it looks like the site could have handled between 9 and 12 times the traffic it received on Friday.  All it would have taken was someone with experience in developing high capacity websites, or even a sysadmin with capacity planning skills to have foreseen this and averted what was an obviously avoidable calamity.  What a pity.  I hope someone in CFA is reading this, as despite the assurances made about things being done - I don't see any evidence of even the basics being covered.

Update 2013-01-15:

The minister has announced today that the site, and the app, have been improved such that there will be no repeat of the problems previously seen.  The details of what they've changed were unclear, however a quick check shows that the only thing addressed on the site itself was the compression.  As mentioned above, this was an easy fix and should have been applied before the site went live.  However, the other points were not addressed.

The wording of the minister's announcement suggests that more hardware has been added.  Perhaps the compression module on the Big IP box?   I haven't run any tests to identify if there have been extra servers added, but I suspect that has happened as well.  Yet compression and the other suggestions I've made would take a few minutes of a competent sysadmin's time, so why is it that it took more than a week for compression to be turned on?  Why aren't the other problems being addressed?  Why are we getting vague assurances that the problems are resolved with absolutely no detail as to what was addressed?  My suspicion is that there is a salesperson somewhere driving around in a very expensive car funded by the commission they have received on upselling the CFA.

:: Next >>