The Limitations of Cyber Risk Assessment Tools

Several months ago, I was asked by an executive, “how much should we be spending on cyber insurance?". A reasonable question – one which had me searching for a tool that could help me both frame and answer the question. I decided to reach out to my network for suggestions, and several people pointed me to the FAIR Framework. This felt like a reasonable fit, I was familiar with FAIR and had used it in the past. However, after persisting with FAIR for several days, it soon became clear that FAIR wasn’t the right tool for the job.

Not all cyber risk tools are fit for the job. In this second instalment (Part 1 here), I share my thoughts on cyber risk tooling and examine situations in which certain cyber tools are preferable, and situations in which they are not.


I wanted to start this discussion by briefly discussing cyber risk analysis. In Part 1, I showed that the typical cyber risk management process looks something like this:

  1. Identify risks
  2. Analyse risks using risk matrix
  3. Evaluate them on risk matrix
  4. Track them in risk register

It is in my experience that most organisations will analyse cyber risks – irrespective of their size, complexity and business outcomes – the same way. And that’s typically using a risk matrix / heatmap to qualitatively select likelihood and impact. In terms of risk analysis, I have yet to come across any cyber security standards, industry documentation or readings that make a distinction between different types of cyber risk problems and different analysis techniques.

Not all risk problems are equal. And because risk problems come in different sizes and complexity, the techniques used to analyse them will not always be the same. One size does not fit all. I summarise here the three different types of risk problems that I see:

  1. Risk problems that require risks to be tracked, prioritised, organised & communicated for the purposes of internal audit, regulation & cyber compliance.
  2. Risk problems that require a better understanding of the nature of a cyber risk. This is to help make a decision on how the security team and/or business can best manage it.
  3. Risk problems that require a better understanding of how uncertainty (including cyber) affects business-level decisions or objectives.

These risk problems should not be conflated. They are different. And because they are different, they require different analysis techniques to address them. I summarise these problems and their corresponding techniques in the following table:

In it, I have used Kaplan’s three risk categories: Preventable, External and Strategic. I spoke about these and drew some conclusions in a previous blog post here. To summarise, my view is that cyber is channel (vector) and not a risk category. Further, cyber will typically cover Preventable and External risk categories, but rarely does it cover Strategic risks (more on this later).

On to the techniques…

1. Techniques to monitor, track, prioritise, organise & communicate internal security audit findings & cyber compliance


The risk matrix (or heatmap) is ubiquitous across the cyber security profession. As mentioned above, it is in my experience that this is the dominant tool for cyber risk analysis – irrespective of the risk problem being solved. It is my view, however, that the use of this tool should be limited for most (if not all) risk problems/analysis. There is now significant empirical evidence that casts sufficient doubt over the accuracy of this tool. If it is not possible to move away from this tool completely, I would suggest limiting this only to those risk problems that require risks to be tracked, prioritised, organised & communicated for the purposes of internal audit, regulation & cyber compliance. These risks fall within the Preventable risk category only. To maximise accuracy and decision-making ability, it is recommended the risk matrix is avoided for all other risk problem types. 

2.            Techniques to better understand the nature of a cyber risk to make a decision on how to best manage it


The FAIR Framework has gained some popularity in recent years and attempts to provide a quantitative alternative to cyber risk analysis.  The primary FAIR use case is the analysis of External cyber risks. That is, using FAIR to perform an analysis to better understand cyber risks so that the security manager and his/her team can better manage them. Whilst this is useful for managing External cyber risks, rarely is this useful in assisting with risks that fall within the Strategic risk category, with a few limited exceptions^.

^ I was successfully able to apply FAIR (with other tools) in one Strategic risk situation. The decision was concerning whether to proceed or abandon a particular technology project, tasked with addressing security risk. Part-way through the project, the management team was eager to understand whether changing the project's course would be more cost effective than continuing with the current course of action. Using FAIR (coupled with risk buy-down charts and PV cash flow comparisons between projects) it was possible to see the risk exposure and switching costs associated with changing the project's course. 

3. Techniques to better understand how uncertainty (including cyber) affects business decisions or objectives.

To conclude this discussion, I go back to the original insurance case study mentioned at the beginning of this post. Whilst it was suggested that FAIR could assist with this type of risk problem, it is my view that risk problems of this type cannot be solved with the FAIR Framework (nor any cyber-specific tools for that case). The insurance problem mentioned is exactly that – a problem about insurance, not a problem about cyber. The risk problem here is not to better understand cyber risk, but to better understand the different factors (cyber included) influencing an insurance claim. 

In this situation, there are many factors that require analysis to understand both the size, scope and likelihood of an insurance claim (cyber being just one). This includes external factors such as time of year, regulatory directives, internal capabilities, locational/geographic factors, among others. Tools better suited to this task would be decision trees to calculate/compare expected value (EV) or sensitive/scenario analysis using modelling software.

I will conclude this post by saying that not all risk problems are equal and different techniques are needed for different problems. My hope is that this post has added something to the conversation and raised some awareness - hopefully provoking some thought on cyber risk analysis. I am keen to hear your thoughts on the topic, and I welcome comments below. 

NB. In Greek mythology, Procrustes was a rogue bandit inn keeper from Attica who was known for coercing weary travellers into his inn. Once inside, Procrustes would stretch his victims and cut off their limbs, so as to force them to fit the size of his bed.


Popular posts from this blog

Defining Cyber Risk

The Pitfalls of Cyber Risk Assessment