Risky Thinking Blog

How much does a disruption cost?

MZB — Thu, 28 Mar 2013 14:53:29 +0000

In designing our new product, the Risk Assessment Toolkit, one of the things we needed to do was model the cost of disruption of an internal activity or process.

External facing activities are often the easiest to discuss: if our web site or phone system goes down, it’s clear that we can’t take customer orders. If the shipping department is under water, then goods can’t be shipped. But what about less defined activities such as Human Resources, or Marketing. How much does a disruption in one of these departments cost?

A naive approach would be to assume that each hour lost is equal to the cost of the staff sitting (or standing outside in the car park) idle. With this approach we must be careful to include any overheads – payroll taxes, medical insurance and other expenses – which might not show directly on a department’s budget.

But this is only half the story.

A digression here: an example from a lecture on Computer Security at Cambridge University made a lasting impression on me. (This was a real world example -although the names and details have been omitted for reasons of confidentiality).

A large utility bought an expensive laser printing and folding system to send out its monthly invoices. It was a unique and expensive system. The company could only afford one of them. With the anticipated workload, it was expected to operate 24×7 and be 99% occupied. When one of its parts failed, the system was down for a week. How long did it take before the bills are once again sent out on on time? At the time interest rates were high – around 10% – so the supplemental question was to estimate how much the company would lose if it had a million customers and the average monthly bill was $100.

Don’t worry – you don’t need to work out the answer. But do consider what the implications are for our hypothetical department.

Suppose the power goes out for half an hour. People stop working. But by the end of the day, are they significantly less productive? If your department is like most departments, there is a degree of over-staffing. People are not 100% occupied most of the day. This is required for resilience under normal operating conditions. Can the department continue operating with one person off sick or on holiday? Almost certainly. Is there a peak workload which has to be handled at certain times of year? Then at other times of year people aren’t 100% busy. People may be idle for half an hour, but by the end of the day, any small outage is unlikely to have had any noticeable effect on productivity. The monetary loss is quite likely to be zero.

How big can such an outage without loss of productivity be? An instructive example here comes from Britain during the 1970s. There was a shortage of electricity due to a miner’s strike. Businesses were switched to a three day work week to reduce electricity consumption. Surprisingly, some businesses reported no drop or even an increase in productivity. People were working shorter hours, but working harder. This demonstrates that, depending on the nature of the work, a department may be able to cope with even quite a long outage with negligible effect on productivity.

But there comes a time when a department will not be able to catch up with its backlog of work in a timely manner while working normally. At this point, overtime is required. Typically this will cost more than standard working hours – perhaps 150% or 200% of standard time. However, this cost increase will only apply to salaries and payroll taxes. It won’t apply to other overhead costs (such as medical insurance). (This is one of the reasons why some companies prefer to increase overtime rather than increase staffing). After an initial period, the amount of overtime required approximately scales with the length of the disruption.

There are only 168 hours in a week, and even with overtime it is not practical (and generally not even legal) to expect your staff to work all of these hours.

At this point you need more staff. Hopefully you can get the extra staff you need on a temporary or contract basis. Contract staff cost more on an hourly basis, but since this cost includes most overheads the difference in cost is not as great as a simple comparison of hourly rates might suggest.

The conclusion from this is that the model for staffing costs contains three phases: an initial phase (of low cost) where lost productivity can be made up without additional overtime. An intermediate phase, where overtime can be used to recover in a timely manner. And finally there is a phase where additional staff will be required to clear up the backlog of work in a timely manner.

However, there are other non-staffing costs to consider if the department must relocate to an alternative location. This will involve transport costs, purchasing of additional or replacement equipment, costs of installing telephone and network services, and perhaps an activation fee for the use of the alternative location. Assuming the department is forced to relocate, there is a sizable cost component which is independent of the length of an outage.

In addition, if facilities are rented, there will be rental costs for the new location. Equipment may need to be hired or leased. Security services may need to hired. And if the new location is a significant distance from the original location, staff will also require travel and meal expenses.

There are thus an additional fixed and variable costs which will be incurred if the department must relocate. The variable costs will continue until a new location is purchased and prepared, or the original location can be repaired for re-use.

Finally there may be some costs which are incurred if deadlines can’t be met: for example, there may be statutory fines if paperwork isn’t submitted on time. There may also be losses in corporate reputation if some non-critical activities (such as updating a website or sending out press releases) don’t happen for an extended period of time.

In summary, we have:

An initial hourly cost, when normal reserve capacity can be used to clear the backlog of work after an outage.
An interim hourly cost, when additional overtime or equipment must be rented
A final hourly cost, when additional staff or equipment must be brought in to clear the backlog.
A fixed relocation cost, which is incurred when work is moved to an alternative location.
An daily relocation cost, reflecting the increased expense of operating at a second temporary location. This will persist until a new permanent location is found. Given the time taken to identify and purchase or rent real estate, working for 30 days at an alternative location is not unlikely.
Various fixed costs which are incurred if the activity has been delayed for an extended period, such as statutory fines or loss of customer goodwill.

Obviously this is all still a highly simplified model, but it does capture some of the important distinctions. In particular, it captures the distinction between a short disruption and an extended one; and between a disruption which requires relocation and one which only requires staff to wait until the disruption ends. Without these distinctions it would be easy to over-estimate the cost of a minor disruption, or under-estimate the cost of a major one.

Strategic Risk Choices: The One Basket Approach

MZB — Tue, 09 Oct 2012 15:02:52 +0000

Are you sure you still want to eat me?

Canadian Beef has a problem. A big problem.

Right now there is the largest recall ever of beef due to to possible contamination with E. coli O157:H7. The recall list is now so long that it has become impossible to summarize.

So Canadian consumers have a problem: they can either not eat beef, carry around with them a list of affected beef products which is several pages long, or treat all beef products as potentially contaminated.

In fact US consumers also have a problem: over 900,000 pounds of beef products subject to recall were shipped across the border.

But the point of interest here is not the problems for the consumer, but the risk management choice made by XL Foods, the company concerned. As far as I can tell, XL Foods has one beef processing plant, XL Lakeside, in Brooks, Alberta. No doubt it is a very efficient plant: it’s apparently the second largest in Canada, and slaughters about one third of Canadian cattle. I don’t doubt that every reasonable effort has been made to make it safe, but this is the classic “put all your eggs in one basket” strategy in action. So with a problem of food contamination shutting down that one plant, the company is not in a good position.

In addition, the consistent branding across all product lines means that the company’s other plants are also affected. (It’s a business to business brand, which is why you or I haven’t heard of it – the food is re-packaged under another customer’s label before we buy it).

Single-branding and large processing facilities are cost-efficient when everything goes well. But when the E Coli hits the fan (yes, fecal contamination is one of the main routes by which the pathogen is transmitted) it’s also the worst possible strategy from a risk perspective: any problem affects the company as a whole and shuts down the entire production.

The XL Foods strategy in responding to the problem so far has been interesting: no press releases, no news conferences, no mention of the beef recall on their website. Perhaps they are hoping it will all just go away.

It won’t. It’s here to stay. Just try Googling XL Foods.

It’s also an interesting case of customer risk: for many ranchers in Alberta, XL Foods was their only customer.

What percentage of businesses fail following a disaster?

MZB — Wed, 26 Sep 2012 15:00:47 +0000

There are a number of statistics floating around which are often quoted in articles but never attributed. One of them is the percentage of businesses that fail in the months following a disaster.

The range for this mythical figure is quite high. I’ve seen 70% quoted quite often. Sometimes the number is 80%. Other times it is a more plausible 40%.

I’ve searched for a reliable source for this number. Was it a survey of businesses following 9/11? The Manchester bombing? Hurricane Andrew? I think now that I have finally found the answer to this question.

So what is the percentage of businesses that fail following a disaster?

The answer is simply this: it does not matter.

Suppose you had access to the data to compute this statistic, what would it actually tell you? There were 18,000 small businesses “… dislocated, disrupted, or destroyed” by 9/11 according to a congressional report. But how many of those were hot dog vendors? Retail shops with a single outlet? Manufacturers of candles? Bicycle couriers? Financial service companies?

Comparing ourselves to this magic percentage does not help us at all with the question we really need to answer: how likely are we to go out of business following a disaster?

The statistics for an arbitrary group of companies is only of interest to editorial writers and politicians. For the rest of us, what matters is only how different types of disaster will affect us – something that we can only determine through a thorough examination of our own business.

Why Computers Crash on New Year’s Day and Canada Day

MZB — Mon, 02 Jul 2012 21:53:32 +0000

There’s 60 seconds in a minute, 60 minutes in an hour, 24 hours in a day, and 365 days in a year. Any fool knows that.

Except that it’s not true.

First of all, there’s leap years. If the year is divisible by four, than an extra day is added to the end of February. Most people know that.

Except that it’s not true.

If the year in question is the zeroth year of a new century, then it’s not a leap year. So 1700, 1800, 1900 weren’t leap years. Many people know that.

Except that it’s not true.

If the century part of the year is divisible by four, then the above rule doesn’t apply and it’s still a leap year. So 2000 and 2400 are leap years. Some people do not know that.

Therefore we had non-Y2K Y2K bugs in computer systems on February 29th 2000.

But we made a mistake even before we considered leap years.

Even less people know that there aren’t always 60 seconds in a minute: sometimes there are 61, and at some point in the future there might conceivably be 59.

Because the speed of rotation of the earth varies in response to climactic and geological events, the time system derived from this (Coordinated Universal Time – UTC) isn’t good enough for high precision calculations (think GPS, spacecraft, etc.) Instead time is defined in terms of International Atomic Time, a time scale derived by measuring the frequency of vibration of atoms in hundreds of atomic clocks.

What this means is that astronomical or clock time (the time most of us use) and atomic time (the time we measure) can drift out of sync. Every few years, on either 1 July or 1 January, a correction is made to keep the difference between the two small. An extra second is inserted (or possibly removed) just before the start of the day.

So if clock time is being used, extra care has to be taken with time calculations twice a year… which is why some major web sites failed when a leap second was introduced at the end of 30 June 2012.

So be prepared for computer problems on New Year’s Day (1 January), and Canada Day (1 July), as well as on February 29th and March 1st.

Notes.

For more reading on the intricacies of time, and how various computer systems and protocols deal with it, a good introduction can be found here.
All this may change in 2015. There’s a proposal to abolish leap seconds and replace them with leap hours which will be voted on at the ITU.

The Risk of a Categorical Denial – The MilitarySingles.com hack

MZB — Mon, 02 Apr 2012 03:12:40 +0000

The first I read on the alleged MilitarySingles.com hack was this:

With the usual link to pastebin for more details. So far a simple hack story. Nothing to see here. Time to move on.

But then the same story came up at The Office of Inadequate Security, along with a denial by the company that runs MilitarySingles.com. The denial reads:

“After a thorough investigation by our company programmers, it is our conclusion that our database was not hacked and that the claims of the Lulzsec group are completely false. Here are a couple points to note:

1. The total number of users in our database does not even closely match the number they have claimed to have exposed.

2. All user passwords in our database are encrypted and secure.

3. The location of the file the above user posted is in a repository directory on our website for user’s photos. The above user simply uploaded a photo of the Lulzsec group and does not mean in any way whatsoever that they were successful in actually hacking our service.

4. MilitarySingles.com was down for a few hours on March 25th due to regularly scheduled maintenance, not due to any outside activity.

We have taken measure to confirm our website and it’s database is secure and safe for our members, and will continue to do so. We are unable to confirm that the so-called checklist of email addresses have actually come from our user database.”

Sounds like a pretty comprehensive and categorical denial.

This was followed by:

Now things are looking interesting. Hackers claim to have hacked a website and downloaded everything; the company that runs it claims that they didn’t.

So Were They Hacked?

A quick look at the files (which, since the company wasn’t hacked, presumably aren’t theirs) reveals:

A file MilitaryDatabase.rar which is a compressed SQL file containing a 500MB dump of a database. It contains an administrator password (in clear and hashed), recent chat sessions, user profiles, email addresses, and hashed passwords. The chat sessions often make references to US military deployments and bases.
A file Military.rar which is a compressed 57 MB text file containing user email addresses, names, and hashed passwords. There are 170,937 records.

The IQ Security Blog has analyzed the database dump. It contained the unsalted MD5 hashes of user passwords. This makes them particularly easy to crack, because it’s possible to pre-compute the hashes of common passwords. IQ Security report that they managed to crack 151,972 of the 163,792 password hashes in just 9 hours.

“It Never Happened”

I guess we can all breath a big sigh of relief that the breach never happened, because, as MilitarySingles.com’s FaceBook page read on April 1st.

It’s still possible LulzSec hackers had an extreme case of obsessive-compulsive disorder and put together 570MB of fabricated data in some form of elaborate April Fool’s joke.

The Problem of Denial

What will happen if the hackers were correct?

That’s the problem with a categorical denial like the one above. It’s a rookie PR mistake. You do well if you are right, but you are damned if you are wrong.

A better approach might have been to use a non-denial denial, as popularized by Woodward and Bernstein in their book on the Watergate Scandal, “All The President’s Men“. Say you don’t have any evidence yet. Say that LulzSec are known to issue misleading statements designed to mislead and confuse. Recommend everyone change their passwords as a precaution.

But don’t issue what sounds like a categorical denial and then go silent as conflicting evidence mounts. Who is going to believe or trust you now?

As it is, it’s going to be quite interesting to see how this one plays out.

The Questionable Wisdom of Crowds

MZB — Wed, 25 Jan 2012 13:00:00 +0000

Photo – James Cridland

There’s often far too much credibility given to the wisdom of crowds.

The original much-quoted observation by Francis Galton that the average of a group’s estimates of something (beans in a jar, weight of a butchered ox) can be much better than an individual estimate may often be true, but in some circumstances it may be much worse. This is frequently forgotten.

I just came across an excellent ten minute presentation by Tom Scott which explains some of those other circumstances, providing a convincing live demonstration using audience tweets. It’s worth having on hand for those occasions when you are tempted (or told) to simply average people’s estimates of the likelihood of an event or of the impact of a disruption.