The Limitations of Cyber Risk Assessment Tools

Several months ago, I was asked by an executive, “how much should we be spending on cyber insurance?". A reasonable question – one which had me searching for a tool that could help me both frame and answer the question. I decided to reach out to my network for suggestions, and several people pointed me to the FAIR Framework. This felt like a reasonable fit, I was familiar with FAIR and had used it in the past. However, after persisting with FAIR for several days, it soon became clear that FAIR wasn’t the right tool for the job.

Not all cyber risk tools are fit for the job. In this second instalment (Part 1 here), I share my thoughts on cyber risk tooling and examine situations in which certain cyber tools are preferable, and situations in which they are not.

CYBER RISK ANALYSIS

I wanted to start this discussion by briefly discussing cyber risk analysis. In Part 1, I showed that the typical cyber risk management process looks something like this:

  1. Identify risks
  2. Analyse risks using risk matrix
  3. Evaluate them on risk matrix
  4. Track them in risk register

It is in my experience that most organisations will analyse cyber risks – irrespective of their size, complexity and business outcomes – the same way. And that’s typically using a risk matrix / heatmap to qualitatively select likelihood and impact. In terms of risk analysis, I have yet to come across any cyber security standards, industry documentation or readings that make a distinction between different types of cyber risk problems and different analysis techniques.

Not all risk problems are equal. And because risk problems come in different sizes and complexity, the techniques used to analyse them will not always be the same. One size does not fit all. I summarise here the three different types of risk problems that I see:

  1. Risk problems that require risks to be tracked, prioritised, organised & communicated for the purposes of internal audit, regulation & cyber compliance.
  2. Risk problems that require a better understanding of the nature of a cyber risk. This is to help make a decision on how the security team and/or business can best manage it.
  3. Risk problems that require a better understanding of how uncertainty (including cyber) affects business-level decisions or objectives.

These risk problems should not be conflated. They are different. And because they are different, they require different analysis techniques to address them. I summarise these problems and their corresponding techniques in the following table:


In it, I have used Kaplan’s three risk categories: Preventable, External and Strategic. I spoke about these and drew some conclusions in a previous blog post here. To summarise, my view is that cyber is channel (vector) and not a risk category. Further, cyber will typically cover Preventable and External risk categories, but rarely does it cover Strategic risks (more on this later).

On to the techniques…

1. Techniques to monitor, track, prioritise, organise & communicate internal security audit findings & cyber compliance

RISK MATRIX / HEAT MAP


The risk matrix (or heatmap) is ubiquitous across the cyber security profession. As mentioned above, it is in my experience that this is the dominant tool for cyber risk analysis – irrespective of the risk problem being solved. It is my view, however, that the use of this tool should be limited for most (if not all) risk problems/analysis. There is now significant empirical evidence that casts sufficient doubt over the accuracy of this tool. If it is not possible to move away from this tool completely, I would suggest limiting this only to those risk problems that require risks to be tracked, prioritised, organised & communicated for the purposes of internal audit, regulation & cyber compliance. These risks fall within the Preventable risk category only. To maximise accuracy and decision-making ability, it is recommended the risk matrix is avoided for all other risk problem types. 

2.            Techniques to better understand the nature of a cyber risk to make a decision on how to best manage it

FAIR



The FAIR Framework has gained some popularity in recent years and attempts to provide a quantitative alternative to cyber risk analysis.  The primary FAIR use case is the analysis of External cyber risks. That is, using FAIR to perform an analysis to better understand cyber risks so that the security manager and his/her team can better manage them. Whilst this is useful for managing External cyber risks, rarely is this useful in assisting with risks that fall within the Strategic risk category, with a few limited exceptions^.

^ I was successfully able to apply FAIR (with other tools) in one Strategic risk situation. The decision was concerning whether to proceed or abandon a particular technology project, tasked with addressing security risk. Part-way through the project, the management team was eager to understand whether changing the project's course would be more cost effective than continuing with the current course of action. Using FAIR (coupled with risk buy-down charts and PV cash flow comparisons between projects) it was possible to see the risk exposure and switching costs associated with changing the project's course. 

3. Techniques to better understand how uncertainty (including cyber) affects business decisions or objectives.

To conclude this discussion, I go back to the original insurance case study mentioned at the beginning of this post. Whilst it was suggested that FAIR could assist with this type of risk problem, it is my view that risk problems of this type cannot be solved with the FAIR Framework (nor any cyber-specific tools for that case). The insurance problem mentioned is exactly that – a problem about insurance, not a problem about cyber. The risk problem here is not to better understand cyber risk, but to better understand the different factors (cyber included) influencing an insurance claim. 

In this situation, there are many factors that require analysis to understand both the size, scope and likelihood of an insurance claim (cyber being just one). This includes external factors such as time of year, regulatory directives, internal capabilities, locational/geographic factors, among others. Tools better suited to this task would be decision trees to calculate/compare expected value (EV) or sensitive/scenario analysis using modelling software.

I will conclude this post by saying that not all risk problems are equal and different techniques are needed for different problems. My hope is that this post has added something to the conversation and raised some awareness - hopefully provoking some thought on cyber risk analysis. I am keen to hear your thoughts on the topic, and I welcome comments below. 

NB. In Greek mythology, Procrustes was a rogue bandit inn keeper from Attica who was known for coercing weary travellers into his inn. Once inside, Procrustes would stretch his victims and cut off their limbs, so as to force them to fit the size of his bed.

Comments

Post a Comment

Popular posts from this blog

Defining Cyber Risk

Image
The more I learn, the more I realise how little I know. This is especially true for  risk . I have been practicing risk management (whatever that means) for a little over 10 years now. Even with this experience, risk still confuses me. You can hardly blame me; especially when the industry  itself  can't agree on what risk is. Take, for example, this list of risk definitions I compiled from industry sources. Not only does each standard have its own distinct way of describing risk, they each have their own way of calculating risk, too! Before going further, I want to briefly diverge and discuss where I see cyber risk in the broader risk picture. There’s a great HBR article by Robert Kaplan and Anette Mikes on risk strategy which I highly recommend (you can check it out  here ). In it, they discuss three broad categories of risks: Preventable Risks, External Risks and Strategic Risks. Each has their own distinct way of dealing with risk. Preventable risks  are those risks that arise f
Read more

The Pitfalls of Cyber Risk Assessment

In my previous post, I discussed the ambiguous nature of cyber risk. There was some healthy engagement on the topic, and I thought I would continue from where I left off. In this two-part post I will explore some shortcomings of the cyber risk assessment practice and follow on by looking at when certain risk techniques should and should not be used. The Cyber Risk Assessment I’m going to start off with something that would be familiar to many of you: the risk workshop. For those that are unfamiliar, the basic premise of this risk activity is to bring people together to brainstorm ideas about “bad things that might happen”. These ideas are then typically prioritised on a company risk matrix (heat map) and later organised in a list, to be tracked and periodically communicated. This process, whether done in a group or an individual setting, is fairly typical of most cyber risk assessment methodologies. A typical cyber risk assessment methodology will look something like this: Identify ris
Read more