As a result of disasterous fires in our state in 2009 known as Black Saturday there was a Royal Commission into the fires, the management of them and the deaths that occurred as a result.  Coming out of this was a set of recommendations and changes were instigated by a number of parties.  One of the main parties of interest in the handling of fire services in the state is theCFA (Country Fire Authority).

The CFA has to be congratulated for at least the intent of some of their initiatives post Black Saturday.  One of these is their Fire Ready app for mobile phones, and another was a revamp of their website.  A lot of time, and presumably public funds, went into the developments that resulted in a catastrophic failure of the website and app. So what went wrong?

Initial reports suggest that capacity planning was at issue, with the statement made that they had planned for up to 350 hits per second, but the site received 700 hits per second. Much of the blame was placed on the fact that the app was connecting to the same server as the main site, and that part of the solution was that they put the app on its own server. Note the singular 'server' there.  I'm not sure if this was just a reporting problem, trying to make it easier for journalists to understand, or we are really talking about an under-resourced system that is supposedly meant to assist people in the times of greatest need and stress.  So, aside from the fact that both were on the same infrastructure (let us assume there is actually more than one server there), what can we determine about the possible problems that these 700 hits per second were causing.

Firstly, looking at the site details, the app now uses osom.cfa.vic.gov.au, which appears to be fronted by a Squid proxy cache, often used to act as a web site accelerator, so there could be more than one server behind that.  The main website, www.cfa.vic.gov.au is behind a F5 BIG IP appliance, which in its base form is a load balancer, meaning that unless they are spending money for the sake of it, there is at least more than one server providing the service.  The BIG IP appliance is quite a useful beasty, and can provide web site acceleration, using compression and other techniques to make life easier.  So why was there a problem?  And apologies for me starting to get a bit technical from here on in.

Let's first define what a "hit" might be.  When you load a web page, like www.vic.gov.au, you might think that by the time it finishes loading that is one "hit".  And you'd be wrong.  A web page is made up of a number of files. Each image on the site is likely to be a file, as is each stylesheet and javascript file. Each of these files is requested separately and each of these requests results in a "hit".  Taking a look at the structure of the CFA site shows that there are a number of unnecessary files on there.  For instance there are 6 javascript files that could be combined into one.  There are images called from CSS that could be combined into "sprites".  Just performing that alone would change the number of hits the site was getting to load a single page, and in doing so increase its capacity - without resorting to extra hardware or tweaking networking stacks.  Indeed it looks like you could increase the number of page views supported by a factor of at least three, simply by resolving these extraneous hits.

There is more intersting stuff under the covers.  There is no compression used on any of the assets (page data, images, scripts, stylesheets) so that the amout of data having to be sent down the wire is far greater than it needs to be.  A conservative estimate suggests that the site could handle twice the current load simply by turning on compression.  This is usually just a configuration change in the base web server software.  All modern browsers support compression, and it is a rookie mistake not to be using it.

On the same front, most browsers also try and help by actively caching items that don't change often, meaning the next time you go to the site it is far quicker to load as the browser already knows about it.  For this to work you need to make sure your web server is set up to help out.  The CFA site is not.  There are two factors here, ETags and Expire headers.  ETags are supposedly unique ids for assets that allow the browser to check that it has a copy of this and therefore doesn't need to re-download (providing it hasn't expired).  The trouble with these is that if you have more than one server supplying the same data - as you would behind say a BIG IP appliance,  these ETags are likely to be different for the same resource, meaning that the browser thinks it needs to download the item again as it has changed.  The CFA site has ETags turned on.

Expiry headers tell the browser how long it should hold an item in its cache before checking for a new version.  For items that don't change much (like images and stylesheets and scripts) it is common to set a "far future" expiry header, say a year or even a month into the future.  This means the browser doesn't need to worry about these items after the first load.  The CFA site doesn't use Expiry headers.  Fixing these two problems could see at least a 50% if not doubling of the capacity of the servers.

Now, without looking at hardware, operating system or web server software, it looks like the site could have handled between 9 and 12 times the traffic it received on Friday.  All it would have taken was someone with experience in developing high capacity websites, or even a sysadmin with capacity planning skills to have foreseen this and averted what was an obviously avoidable calamity.  What a pity.  I hope someone in CFA is reading this, as despite the assurances made about things being done - I don't see any evidence of even the basics being covered.

Update 2013-01-15:

The minister has announced today that the site, and the app, have been improved such that there will be no repeat of the problems previously seen.  The details of what they've changed were unclear, however a quick check shows that the only thing addressed on the site itself was the compression.  As mentioned above, this was an easy fix and should have been applied before the site went live.  However, the other points were not addressed.

The wording of the minister's announcement suggests that more hardware has been added.  Perhaps the compression module on the Big IP box?   I haven't run any tests to identify if there have been extra servers added, but I suspect that has happened as well.  Yet compression and the other suggestions I've made would take a few minutes of a competent sysadmin's time, so why is it that it took more than a week for compression to be turned on?  Why aren't the other problems being addressed?  Why are we getting vague assurances that the problems are resolved with absolutely no detail as to what was addressed?  My suspicion is that there is a salesperson somewhere driving around in a very expensive car funded by the commission they have received on upselling the CFA.

Having a look at a piece of code recently I noticed some very familiar code - it was my code, originally released under GPL with the copyright notice clearly stating that it was "part of the collected works of Adam Donnison". ?I had done this deliberately and done so with a number of pieces of code that I had built over the years, knowing that they were useful and could be used in other projects. ?Imagine my surprise to see that same code, only slightly modified, in another project with the copyright notice removed and the explanation:

?* Note: Previously, this class was mis-licensed as GPL in an otherwise BSD
?* ? application. The GPL attempt was in 2003 while the project itself was not
?* ? relicensed from BSD to GPL in 2005. In 2007, all further development was
?* ? done under the Clear BSD license and all GPL modifications were removed.

Really? ?Since when did the BSD license become viral? ?Wasn't that the entire reason people complained about GPL and wanted to move to BSD? ?There is nothing in the BSD license, or the Clear BSD license that demands that all code in a project be covered by the same license. ?Indeed even prior to 2001 dotProject had code that was under the Voxel Public License (ticketsmith) which was more restrictive than BSD, so there had been precedents for differently licensed parts of the code. ?Indeed none of the BSD licenses even has the concept of a "project" or "greater work". Dropping the copyright is also a violation of the BSD license, as it is of the GPL, so no matter how you cut the dice, this action was against both the spirit and the letter of the licenses it supposes to uphold.

As to "all GPL modifications were removed", an interesting and, on the face of it, erroneous statement.

I believe I am within my rights to demand that the copyright notice be reinstated, or the code removed. ?Now I don't want to get heavy with anyone, but these licenses only work based on strong copyright protection. ?My copyright has been violated, and I am now considering what action to take.

This is not the first run-in I've had with this project's developers on their cavalier attitude to copyright notices, but this is by far the most egregious. ?I believe their actions were to allow them to make money on the project - in which case I also believe that damages could be sought. ?I have no problem with them making money - only not by stripping me of my rights.

Update: I've since spoken with the project lead on the project in question and we've come to an understanding on the issue.

Speaking to Telstra accounts was, as usual, tricky.  All they could tell me was exactly what I could see in the usage graph, that I was being shaped and had exceeded my usage.  I explained how it was not theoretically possible to have downloaded the data claimed and explained why - so they decided I would be better talking to tech support.  The person on tech support was really helpful, which is not something I'm used to.  Normally they run through a set script and then give me grief because I don't use Windows, cannot run Internet Explorer, and generally don't like having to explain a technical issue to someone who is obviously non-technical.  Anyway, after exploring the issues, they suggested resetting the wifi password and SSID.  I pointed out that since I live on 20 acres, and unless someone parked their car in my front yard they would have no chance of even seeing the router, it was unlikely to be an issue - although I agreed to do so.  Just before talking to Telstra I checked the usage meter (which updates hourly).  There was 906MB downloaded today.  After the call, and at least an hour after I last checked, I checked again. 1474MB.

Whoa, 568MB in one hour?

I grabbed my calculator.  64kbps is 8KB per second, which is 28MB per hour.  568MB is 20 times that.  Sorry guys, not remotely possible, unless you are telling me that I can be both shaped and download at speeds that I find almost impossible to achieve even when the wind is in the right quarter and the schoolkids are still at school.

I managed to convince them that there was a real issue, and that we needed to monitor this.  I also changed my password to my account - which I suspect is more likely to be the issue.  They promised to lift the limit on my account although so far that hasn't yet happened.  I suspect, once again, I'll have to spend hours on the phone trying to find someone who understands the issue and can do something about it.  Stay tuned for updates.

The detail behind this is that I have two virtual servers that are with a VPS provider, on two completely different subnets and from the time they were commissioned, no email has been accepted by Yahoo! for its account holders with an error message that appears to be for persistent spamming.  This is clearly an error, one that I have tried for weeks now to resolve with Yahoo! without success.  They will not admit that their detection of spamming cannot be for mail they have yet to receive, but instead most likely for a previous owner of the IP addresses in question. They will not provide any clear information even as to the reason that we are - in their quaint and completely incorrect terminology - "deprioritized".

I don't really want prioiritised mail services, any level of mail service would be fine with me, and I suspect with those users who in good faith have signed up with one of the services we host.  "Deprioritized" suggests that there is a chance the mail will get through, just not in a timely fashion. What we have is in fact a complete embargo on mail from our servers.

It doesn't matter how you wrap it up or what policies you quote ad-infinitum, Yahoo!  If your system starts blocking mail on the first attempt then there is something wrong with your system, not mine.  If it is for prior usage of the IP address, then provide me with the methods of showing that the IP address changed hands so you can reset your system. Don't lie to me and tell me that it is temporary and will resolve itself once I get my systems in line with your policies.  It is not temporary, it is a complete block.  My systems are in line with your policies and have been for quite some time.  You will not answer any of my questions with anything other than a pointer or extract from your policy documents - so I have no idea (and I suspect neither do you) of why I am listed at all, let alone how to resolve it.

So if you are a Yahoo! account holder, stop this nonsense by switching provider. I can't be the only service provider that is affected this way, so even if you are not on one of the services we host, I can assure you there is a good probability you are affected.

If you are Yahoo!, stop this nonsense by acknowledging your system is flawed and fixing it.

For events registration we use the Signup module and use a theme template function to provide a set of standard fields.  The code looks something like this:

function ourtheme_signup_user_form($node) {
  $form = array();
  // If this function is providing any extra fields at all, the following
  // line is required for form form to work -- DO NOT EDIT OR REMOVE.
  $form['signup_form_data']['#tree'] = TRUE;

  $form['signup_form_data']['FirstName'] = array(
    '#type' => 'textfield',
    '#title' => t('First Name'),
    '#size' => 40, '#maxlength' => 64,
    '#required' => TRUE,
  );
  $form['signup_form_data']['LastName'] = array(
    '#type' => 'textfield',
    '#title' => t('Last Name'),
    '#size' => 40, '#maxlength' => 64,
    '#required' => TRUE,
  );

And so on, building up the elements and then returning the form.  This is great because it allows us to have a standard set of fields for all signup pages, making life a lot simpler when creating content that requires registration.  But the Solutions Day event required an extra field.  I could have done this a number of ways, including putting logic in the template file to check for that particular node and only display the field then, or perhaps some other hack specific to this node.  I, however, don't like specifics and tend to look for a generic solution, as the exception invariably becomes the rule.

For this exercise I wanted to be able to have a way of specifying for a particular node any extra fields that are available for this form.  So I now have in the template.php file the following code:

// If there is a special field required for this, check and display
  if (!empty($node->field_signup_extra) && !empty($node->field_signup_extra[0]['value'])) {
    $extras = explode("\n", $node->field_signup_extra[0]['value']);
    foreach ($extras as $field_def) {
        $field_def = trim($field_def);
        if (empty($field_def)) {
            continue;
        }
        $elems = explode('|', $field_def);
        $field_name = array_unshift($elems);

        $form['signup_form_data'][$field_name] = array();
        foreach ($elems as $field_element) {
          list($key, $val) = explode('=',$field_element);
          if ($key == 'options') {
              $val = explode(',', $val);
          }
          $form['signup_form_data'][$field_name]['#' . $key] = $val;
        }
    }
  }

Now all I need to do is create a field that is non-displayable but contains information to build extra fields.  For example the content that describes the Dietary Requirements field is:

dietary_requirements|title=Dietary Requirements|size=40|type=textfield

The production version does a little more analysis of the input to ensure there are no possible attack vectors, but I've left that out for clarity sake.

Now, if I have an event (or other content type) that needs extra signup fields, I ensure that the content type has the new Signup Extras field and fill it on the new content with a simple field definition that Signup can use.

Here's a hint Yahoo.  If you don't tell me what the problems are, I can't fix them.  A basic tenet of support. If you keep me in the dark then I will have to start banning users with yahoo.com* addresses as it is just too hard to deal with you.  I'll have to tell them all to get a real mail account, like Gmail, so they don't have to bitch about not getting important emails. Perhaps this is why you are losing users?

While I was researching the possible reasons for by blocking I found that there are some really dodgy practices going on out there.  I checked my Reputation Score with a number of sources, and found that it was pretty good, except for one, Cisco's SenderBase.  And I can't send email to them about the problem from the problematic domain. Mind you one had information on email that went back 4 years and showed no recent activity - and I didn't own that IP address 4 years ago.   There is one, Barracuda, that suggests you sign up at EmailReg.org for $20 per year to bypass their reputation filter - um, that is uncomfortably close to blackmail for my liking.  Then there is the abuse.net system.  It appears that many reputation scores will reduce your reputation if you haven't taken the trouble to register a contact address for your domains at abuse.net.  WTF? The internet standard is that there is always a postmaster@ address that must go to a real person.  In recent times it has also become common practice to have an abuse@ address.  Since when does it become mandatory to register this somewhere?  How does that in any way contribute to the removal of SPAM?

Abuse.net also has some quirks.  With the rise of SPAM many people who set up a new domain take the precaution of using a hidden registration option offered by many domain registrars, to avoid having your contact details harvestable by spammers.  And guess what, you can't register these at abuse.net because they believe that you are likely to be a spammer if you want anonymity. Catch 22.

So the upshot appears to be don't send email - use a carrier pigeon instead.

I'll be talking about MySQL®, the companies that now exist to support it, and the third party products that are starting to proliferate in what appears to be a community effort to address perceived shortcomings in the Oracle offerings.  Many of these offerings are in the HA space, and there have been some pretty amazing developments recently.

So if you want to find some history of Oracle's effect on MySQL's development (and a hint, it started well before the Sun acquisition), or you want to find out what is out there to assist you in developing HA and Distributed database systems, come along.  I'll be wearing my SkySQL shirt so I should be easy to pick.

Replication is the technology that allows data to be stored on multiple servers. Typically this is used in "scale out" applications.  "Scale out" is used in contrast to "Scale up" where to scale a solution you buy a bigger box to run it on, where "scale out" means you buy more boxes.  Each has its benefits and drawbacks, with the usual benefit of scale out being that you get more bang for your buck.

The way replication works in MySQL is pretty simple.  One server is identified as the master, and writes every transaction to a file, the binary log.  Other servers (and there may be many) act as slaves and request information from the master. The slave keeps track of where it got up to and asks the master for the next transaction in the file.  In general the master doesn't know or care where the slave is up to, it just sends out the requested transaction to whoever has the right credentials to ask for it.

On the slave there are two threads running, one that requests events from the master binlog (binary log) and writes them to the relay log (which in reality is just another binlog), and the second thread that reads the relay log and executes the queries. In order to avoid non-deterministic outcomes the SQL thread is just that, a single thread.  Long running events can result in the replication lagging well behind the state of the master.

This is the traditional asynchronous replication.  Newer versions of MySQL now support semi-synchronous replication as well.  In this mode the master will not return control back to the querying process until at least one slave reports that it has received the transaction events and written them to its own relay log.  Note that this is written to the relay log but not necessarily written to the database.

You may wonder under what circumstance replication would be of use.  If your application is mainly read-intensive (as many web applications are) then replication gives you the ability to spread reads across multiple slaves to reduce contention.  If your application is mainly write intensive, then replication to multiple slaves will not, of itself, give you any relief.  Indeed, because the replication is single-threaded and all writes on the master must be written to the slaves, you can in fact hurt performance with a replicated setup.

What about more than one master?  This is possible but requires careful thought and planning.  MySQL replication keeps track of which server initiated the transaction, so "circular replication" where a slave acts also as a master should not cause problems, however there are some serious issues to consider.  Take for instance a customer table with the customer ID being auto incremented.  What happens if there are writes to both masters in a master<->master replication setup?  Both will grab the next ID in sequence, then the other will pull that transaction and try and will end up with a duplicate index error, causing replication to fail.  You can get around this by setting the initial auto_increment value on each server and setting the auto_increment_increment value to the number of masters in use.  Remebering however that both masters now have to write all traffic that goes to the pair, there is not a great deal of benefit in such an arrangement.  Other technologies, such as sharding, turn out to offer better scale out opportunities for multi-master deployments.

There are, however, some tools to make multi-master work more easily.  MMM (Multi-Master Replication Manager for MySQL) is one.  MHA (Master High Availability for MySQL) is somewhat different, allowing easy management of master failover.

That is probably enough theory, lets look at how to build a replication system.

For the master, the only requirement is that it writes its transactions to a binary log, and that it has a server ID set.  To do this we need to add the following to the my.cnf file:

server-id = 1
log_bin = /var/log/mysql/mysql-bin.log

The server-id is an integer that must be unique for all servers in the same replication set.  This is because of the tracking of transactions to avoid circular replication errors.  The log_bin is the path to the binary log file to use.  Each file will be appended with a sequence number to allow the server to rotate log files when they grow too large, or a flush request is received.

One other value that is useful is to set an expiry on the binlogs so that they don't take over all of your disk.  You need to ensure that you have enough binlog capacity for normal replication lag and for handling backup/recovery, but beyond that you can get MySQL to automatically delete the older binlogs.

expire_log_days = 14

Now the master will at least be writing transactions to the binary logs.  Now we have to set up to allow slaves to replicate.  First step is to create a user that the slaves will connect with.  This user must have REPLICATION SLAVE permissions to be able to read the binlog.  Note that this user is created on the master. As a privileged user, in the MySQL client:

CREATE USER 'replicator'@'192.168.%' IDENTIFIED BY 'mypass';
GRANT REPLICATION SLAVE ON *.* TO 'replicator'@'192.168.%';

The first command creates our user, the second grants the required permissions.  Note that I've used a wildcarded network address, you can also use a hostname wildcard.

For the slave we need a few things, first we need to set a server-id that is not the same as the master (or any other slave we are creating). Then we need a copy of the data on the master at the point at which the slave is to be started, and we need to then tell the slave about the master.

We can create the server-id in the same manner as we did for the master - but we don't need to specify a binlog (unless we are acting as a relay master).

server-id = 2

To get our first copy of the master data, the best idea is to use mysqldump.  Before we do, we need to stop the master from writing to the binary log and record its current position.  From the mysql command line:

FLUSH TABLES WITH READ LOCK;
SHOW MASTER STATUS;
+------------------+----------+--------------+------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| mysql-bin.000092 |      106 |              |                  |
+------------------+----------+--------------+------------------+
1 row in set (0.00 sec)

We need to keep this session open so that the lock remains in place, and from another terminal run a mysqldump.

mysqldump --all-databases > all_db.sql

Once the backup is completed we can either drop the lock session or issue:

UNLOCK TABLES;

Great, we now have a backup and we have the position in the binary log at the point at which the backup occurred.  These two together provide us with all we need to create our slave.

First step is to load the above SQL file into the slave.   On the slave you can use the mysql command line client with the SQL file copied from the master.

mysql < all_db.sql

On the slave server we now need to issue a CHANGE MASTER command to tell it where the master is, and where to start replicating from.  This information will be written to the master.info file that is read at startup and updated by the replication process so that the slave will be able to resync with the master on restart.

CHANGE MASTER TO MASTER_HOST = 'master'
  MASTER_USER = 'replicator'
  MASTER_PASSWORD = 'mypass'
  MASTER_LOG_FILE = 'mysql-bin.000092'
  MASTER_LOG_POS = 106;

The MASTER_HOST, MASTER_USER and MASTER_PASSWORD must match the master and the user created above with replication slave permission.  The MASTER_LOG_FILE and MASTER_LOG_POS come from the SHOW MASTER STATUS used when we did the backup of the master.

Now all that is left to do is to start the slave processes:

START SLAVE;

We can use the same backup and the same processes above to create as many slaves as we need.

If you have to set up a complex replication scenario that is not covered by the above, you may find the SkySQL Reference Architecture provisioning system useful.  This service creates master/slave high availability configurations and packages software and configurations together so that you can easily install a system from scratch that meets all your replication needs.

Disclaimer

I am employed by SkySQL Ab, which provides support, training and services for MySQL, MariaDB and Drizzle and is staffed by many ex-MySQL AB employees. While I try to be unbiased in all of my public opinions, and my opinions are all my own and not those of my employer, I work for SkySQL because I believe in what they are doing and therefore cannot be truly unbiased.

MySQL is a registered trademark of Oracle Corporation,  MariaDB is a trademark of MontyProgram, and Drizzle is a trademark.