Arduino antics

I had heard of Arduino for a while, but it wasn't until the 2012 Linux.Conf.AU in Ballarat that I got my hands on one.  Freetronics had come up with an Arduino compatible USB stick called the LeoStick and they provided a free one to each conference attendee.  That first night I had the thing out of the packet and hooked up doing a simple blink demo.  I was impressed with the development cycle, the IDE and the ease of development. These powerful little building blocks are coupled with a huge community of developers and users and there are code samples and libraries for a plethora of interconnects.  I had dreams of doing great things with the Arduino. It then languished in the back of the electronics box for nearly 12 months.

At the moment we in western Victoria are suffering from a dry spell that is more severe than many of the drought years, although the government refuses to call it a drought as that would imply having to provide drought assistance.  So far we've been running at less than a quarter of the average rainfall with no signs of the situation improving.  Naturally thoughts come to handling water - as it is now becoming a scarce resource.  So I dusted off the LeoStick and the Arduino IDE and started tinkering.

For years I had a greywater system that I'd developed a PIC microcontroller circuit for, which included a web interface and several options for handling pumping to various outlets (like a toilet cistern and garden beds).  We then moved and the needs for the unit became a lot simpler, and  the pump driver triac died so I converted to a simple latching relay circuit, with the idea that I'd eventually update the microprocessor controller and re-install it.  With the ongoing "unusually dry period" it seemed like a good project to get something real under my belt.  As I knew exactly what I wanted to do the first cut of the code was done in a matter of hours, and I only had to wait for some components to arrive to get the hardware together.

I also decided to put together some tank sensors on our main water tanks so we didn't have to go tapping on them and listening for the change in sound where the water level sits.  I had looked at a lot of sensors out there but I figured that with an Arduino it shouldn't be that difficult.  I found this capacitive sensor library that was designed for touch sensitive plates, but could easily be adapted to measure any small capacitance.  Coupling this with a length of wire placed in the tank and a resistor, I have a reasonably reliable tank sensor.  Just to make sure I've added a float sensor at the top of the tank just below the overflow level so that I can use it as a pump cut-out.  Since then I've found some reasonably priced pressure sensors that give a far more consistent result than the capacitive sensor.

Next project was not really water related, but something that has irked me for some time.  We bought a heap of Arlec solar powered motion sensor lights a few years ago, and they worked reasonably well for a few months, but then decided to die.  Pulling one apart I found the battery case had cracked and looking further I found that the 6 volt battery was connected with just a blocking diode to the solar panel, which in full sun was generating more than 12V.  This might have worked OK in the suburbs with a few trees giving partial shade, but out here we have pretty much clear skies for more than 90% of the year.  Coupled to this was the use of a 6V 10W halogen globe.  Not really easy to replace, and the light output wasn't fantastic. I had tried putting in a charge controller and that helped, but I realised that with a few components I could put together something far more useful.

I'd purchased 5m rolls of led strip lighting that runs off 12V and you can cut it into short lengths and, with the adhesive backing, stick it up where you need it.  I'd used that for internal lighting in a coolroom and had some left over.  Coupling this with a few motion sensors (under $3 each from China), a few phototransitors and an Arduino I came up with a lighting solution that is far better both in light quality and coverage and in detection.  Almost 12 months later and it is still going strong.

Now with all of these projects I decided that knowing what was going on was important, so they all have support for XBee radio units to communicate with a control system.  This allowed me to abstract a lot of code into my own library for handling the XBee communications, configuration, remote control and a few other useful functions, leaving the sketches (as Arduino programs are called) very small and focussed only on the "business logic" rather than the lower level mechanics.   While useful, I found that unlike most of the components I had looked at, the XBees were increasing in price to the point where it was becoming uneconomic to use, and frankly their performance was pretty dreadful.  In clear line of site using peer-to-peer they work pretty well, but in a mesh network with things like trees and walls in the way they are pretty useless.  In one test I couldn't get the controller to talk to another XBee that was within 3m of it.

I have looked at alternatives and there are some promising units out there.  With some more research and tinkering I should have an answer to my communications issues.

Anyway, should you be interested in this sort of stuff, most of my code is now on GitHub in my "Smallholder Sensor Framework" repository.

Anatomy of a Failure: CFA Website

 

As a result of disasterous fires in our state in 2009 known as Black Saturday there was a Royal Commission into the fires, the management of them and the deaths that occurred as a result.  Coming out of this was a set of recommendations and changes were instigated by a number of parties.  One of the main parties of interest in the handling of fire services in the state is theCFA (Country Fire Authority).

The CFA has to be congratulated for at least the intent of some of their initiatives post Black Saturday.  One of these is their Fire Ready app for mobile phones, and another was a revamp of their website.  A lot of time, and presumably public funds, went into the developments that resulted in a catastrophic failure of the website and app. So what went wrong?

Initial reports suggest that capacity planning was at issue, with the statement made that they had planned for up to 350 hits per second, but the site received 700 hits per second. Much of the blame was placed on the fact that the app was connecting to the same server as the main site, and that part of the solution was that they put the app on its own server. Note the singular 'server' there.  I'm not sure if this was just a reporting problem, trying to make it easier for journalists to understand, or we are really talking about an under-resourced system that is supposedly meant to assist people in the times of greatest need and stress.  So, aside from the fact that both were on the same infrastructure (let us assume there is actually more than one server there), what can we determine about the possible problems that these 700 hits per second were causing.

Firstly, looking at the site details, the app now uses osom.cfa.vic.gov.au, which appears to be fronted by a Squid proxy cache, often used to act as a web site accelerator, so there could be more than one server behind that.  The main website, www.cfa.vic.gov.au is behind a F5 BIG IP appliance, which in its base form is a load balancer, meaning that unless they are spending money for the sake of it, there is at least more than one server providing the service.  The BIG IP appliance is quite a useful beasty, and can provide web site acceleration, using compression and other techniques to make life easier.  So why was there a problem?  And apologies for me starting to get a bit technical from here on in.

Let's first define what a "hit" might be.  When you load a web page, like www.vic.gov.au, you might think that by the time it finishes loading that is one "hit".  And you'd be wrong.  A web page is made up of a number of files. Each image on the site is likely to be a file, as is each stylesheet and javascript file. Each of these files is requested separately and each of these requests results in a "hit".  Taking a look at the structure of the CFA site shows that there are a number of unnecessary files on there.  For instance there are 6 javascript files that could be combined into one.  There are images called from CSS that could be combined into "sprites".  Just performing that alone would change the number of hits the site was getting to load a single page, and in doing so increase its capacity - without resorting to extra hardware or tweaking networking stacks.  Indeed it looks like you could increase the number of page views supported by a factor of at least three, simply by resolving these extraneous hits.

There is more intersting stuff under the covers.  There is no compression used on any of the assets (page data, images, scripts, stylesheets) so that the amout of data having to be sent down the wire is far greater than it needs to be.  A conservative estimate suggests that the site could handle twice the current load simply by turning on compression.  This is usually just a configuration change in the base web server software.  All modern browsers support compression, and it is a rookie mistake not to be using it.

On the same front, most browsers also try and help by actively caching items that don't change often, meaning the next time you go to the site it is far quicker to load as the browser already knows about it.  For this to work you need to make sure your web server is set up to help out.  The CFA site is not.  There are two factors here, ETags and Expire headers.  ETags are supposedly unique ids for assets that allow the browser to check that it has a copy of this and therefore doesn't need to re-download (providing it hasn't expired).  The trouble with these is that if you have more than one server supplying the same data - as you would behind say a BIG IP appliance,  these ETags are likely to be different for the same resource, meaning that the browser thinks it needs to download the item again as it has changed.  The CFA site has ETags turned on.

Expiry headers tell the browser how long it should hold an item in its cache before checking for a new version.  For items that don't change much (like images and stylesheets and scripts) it is common to set a "far future" expiry header, say a year or even a month into the future.  This means the browser doesn't need to worry about these items after the first load.  The CFA site doesn't use Expiry headers.  Fixing these two problems could see at least a 50% if not doubling of the capacity of the servers.

Now, without looking at hardware, operating system or web server software, it looks like the site could have handled between 9 and 12 times the traffic it received on Friday.  All it would have taken was someone with experience in developing high capacity websites, or even a sysadmin with capacity planning skills to have foreseen this and averted what was an obviously avoidable calamity.  What a pity.  I hope someone in CFA is reading this, as despite the assurances made about things being done - I don't see any evidence of even the basics being covered.

Update 2013-01-15:

The minister has announced today that the site, and the app, have been improved such that there will be no repeat of the problems previously seen.  The details of what they've changed were unclear, however a quick check shows that the only thing addressed on the site itself was the compression.  As mentioned above, this was an easy fix and should have been applied before the site went live.  However, the other points were not addressed.

The wording of the minister's announcement suggests that more hardware has been added.  Perhaps the compression module on the Big IP box?   I haven't run any tests to identify if there have been extra servers added, but I suspect that has happened as well.  Yet compression and the other suggestions I've made would take a few minutes of a competent sysadmin's time, so why is it that it took more than a week for compression to be turned on?  Why aren't the other problems being addressed?  Why are we getting vague assurances that the problems are resolved with absolutely no detail as to what was addressed?  My suspicion is that there is a salesperson somewhere driving around in a very expensive car funded by the commission they have received on upselling the CFA.

License confusion and the stripping of rights.

I really thought that the time of license confusion was well past, and we had all settled down and understood the basis, and ramifications of each of the open source licenses.  It seems I was wrong.

Having a look at a piece of code recently I noticed some very familiar code - it was my code, originally released under GPL with the copyright notice clearly stating that it was "part of the collected works of Adam Donnison".  I had done this deliberately and done so with a number of pieces of code that I had built over the years, knowing that they were useful and could be used in other projects.  Imagine my surprise to see that same code, only slightly modified, in another project with the copyright notice removed and the explanation:

 * Note: Previously, this class was mis-licensed as GPL in an otherwise BSD
 *   application. The GPL attempt was in 2003 while the project itself was not
 *   relicensed from BSD to GPL in 2005. In 2007, all further development was
 *   done under the Clear BSD license and all GPL modifications were removed.

Really?  Since when did the BSD license become viral?  Wasn't that the entire reason people complained about GPL and wanted to move to BSD?  There is nothing in the BSD license, or the Clear BSD license that demands that all code in a project be covered by the same license.  Indeed even prior to 2001 dotProject had code that was under the Voxel Public License (ticketsmith) which was more restrictive than BSD, so there had been precedents for differently licensed parts of the code.  Indeed none of the BSD licenses even has the concept of a "project" or "greater work". Dropping the copyright is also a violation of the BSD license, as it is of the GPL, so no matter how you cut the dice, this action was against both the spirit and the letter of the licenses it supposes to uphold.

As to "all GPL modifications were removed", an interesting and, on the face of it, erroneous statement.

I believe I am within my rights to demand that the copyright notice be reinstated, or the code removed.  Now I don't want to get heavy with anyone, but these licenses only work based on strong copyright protection.  My copyright has been violated, and I am now considering what action to take.

This is not the first run-in I've had with this project's developers on their cavalier attitude to copyright notices, but this is by far the most egregious.  I believe their actions were to allow them to make money on the project - in which case I also believe that damages could be sought.  I have no problem with them making money - only not by stripping me of my rights.

Update: I've since spoken with the project lead on the project in question and we've come to an understanding on the issue.

Telstra BigPond Accounting Failure?

This morning I was considering downloading a site image to work on locally, something I do every now and then.  As usual I checked my download quota on BigPond as we are on a limited bandwidth, being out in the sticks and not having access to real internet.  I was astonished to see that the monthly bandwidth was exceeded, by a large margin, and that I was being shaped down to 64kbps.  In fact, the system had started shaping on Sunday.  This was around 2/3rds of the way through the current billing period, and I knew that neither my wife nor I had been doing anything that would account for the whopping 4GB supposedly downloaded since the shaping started.  Indeed, at 64kbps, 4GB in 3 days is theoretically impossible.

Speaking to Telstra accounts was, as usual, tricky.  All they could tell me was exactly what I could see in the usage graph, that I was being shaped and had exceeded my usage.  I explained how it was not theoretically possible to have downloaded the data claimed and explained why - so they decided I would be better talking to tech support.  The person on tech support was really helpful, which is not something I'm used to.  Normally they run through a set script and then give me grief because I don't use Windows, cannot run Internet Explorer, and generally don't like having to explain a technical issue to someone who is obviously non-technical.  Anyway, after exploring the issues, they suggested resetting the wifi password and SSID.  I pointed out that since I live on 20 acres, and unless someone parked their car in my front yard they would have no chance of even seeing the router, it was unlikely to be an issue - although I agreed to do so.  Just before talking to Telstra I checked the usage meter (which updates hourly).  There was 906MB downloaded today.  After the call, and at least an hour after I last checked, I checked again. 1474MB.

Whoa, 568MB in one hour?

I grabbed my calculator.  64kbps is 8KB per second, which is 28MB per hour.  568MB is 20 times that.  Sorry guys, not remotely possible, unless you are telling me that I can be both shaped and download at speeds that I find almost impossible to achieve even when the wind is in the right quarter and the schoolkids are still at school.

I managed to convince them that there was a real issue, and that we needed to monitor this.  I also changed my password to my account - which I suspect is more likely to be the issue.  They promised to lift the limit on my account although so far that hasn't yet happened.  I suspect, once again, I'll have to spend hours on the phone trying to find someone who understands the issue and can do something about it.  Stay tuned for updates.

Open letter to Yahoo! account holders

Yahoo! account holders are being witheld information that they signed up to receive. This is not the fault of the services they signed up with, but rather the arcane and obscure mechanisms by which Yahoo! supposedly identifies spam sources. If you are a Yahoo! account holder, and you suspect that you are not getting email you should, chances are you are correct. Notify Yahoo! that their mechanisms for problem resolution simply do not work and you are going to vote with your fingers and move to another email provider.

The detail behind this is that I have two virtual servers that are with a VPS provider, on two completely different subnets and from the time they were commissioned, no email has been accepted by Yahoo! for its account holders with an error message that appears to be for persistent spamming.  This is clearly an error, one that I have tried for weeks now to resolve with Yahoo! without success.  They will not admit that their detection of spamming cannot be for mail they have yet to receive, but instead most likely for a previous owner of the IP addresses in question. They will not provide any clear information even as to the reason that we are - in their quaint and completely incorrect terminology - "deprioritized".

I don't really want prioiritised mail services, any level of mail service would be fine with me, and I suspect with those users who in good faith have signed up with one of the services we host.  "Deprioritized" suggests that there is a chance the mail will get through, just not in a timely fashion. What we have is in fact a complete embargo on mail from our servers.

It doesn't matter how you wrap it up or what policies you quote ad-infinitum, Yahoo!  If your system starts blocking mail on the first attempt then there is something wrong with your system, not mine.  If it is for prior usage of the IP address, then provide me with the methods of showing that the IP address changed hands so you can reset your system. Don't lie to me and tell me that it is temporary and will resolve itself once I get my systems in line with your policies.  It is not temporary, it is a complete block.  My systems are in line with your policies and have been for quite some time.  You will not answer any of my questions with anything other than a pointer or extract from your policy documents - so I have no idea (and I suspect neither do you) of why I am listed at all, let alone how to resolve it.

So if you are a Yahoo! account holder, stop this nonsense by switching provider. I can't be the only service provider that is affected this way, so even if you are not on one of the services we host, I can assure you there is a good probability you are affected.

If you are Yahoo!, stop this nonsense by acknowledging your system is flawed and fixing it.

:: Next >>