Blog

One of the most challenging part of building security into your organisation – or even just improving your existing security – is determining how you will measure the effectiveness of your security controls.

Security is unusual in that truly effective security controls are invisible and prevent bad things from happening. This means that the only way you can really know the control effectiveness is by measuring how many things dont happen.

This is not easy. But it is still important and in lots of instances it is essential.

If you can’t measure it, you can’t manage it.
[various sources]

Almost all management doctrines and frameworks have, at their heart, some way of defining a set of performance indicators, measuring them and then tracking performance. This is the linchpin for quality management and the overwhelming majority of organisations across the globe employ some variation on the theme.

Except when it comes to security.

Security Information and Event Management Tools

The importance of finding some way to measure security performance has led to a whole raft of tools designed to monitor technological security activity – frequently referred to as “Security Information and Event Manager” (SIEM) variations of the name – provided by both big-name companies (IBM produces TSIEM for example) and Open Source projects.

Wonderful as this tools are (or at least can be) there remain two major problems:

1. Management metrics are arbitrarily defined, often based on what a favoured tool can deliver rather than what is important to the business.
2. SIEM tools can only measure what they are plugged into which can lead to the exclusion of areas of your IT Infrastructure (especially if you are using the tools on a volume licence) or, more frequently end up excluding entire sources of security risk such as physical access control and joiners/movers/leavers processes.

All too often, the deployment of SIEM tooling is used to plaster over cracks in an organisations Risk Management structures and this invariably leads to uncontrolled risks and security breaches.

Without a well thought out, management led, risk management approach spending money on tooling – whatever the tool – is wasting money.

What is the Security Measurement Solution?

Unfortunately there is (and probably never will be) a one-size-fits-all security measurement solution. How you address this important problem is almost entirely driven by your organisations culture, ethos and goals which is why the deployment of packaged, “off the shelf,” solutions so frequently fails. Unless you really do know what you are doing and have a robust enough risk management function that you can ensure value then you should steer clear of any supplier promising a packaged solution.

Building your own solution is may or may not be the lowest cost option but, importantly, will be the most cost effective option. Spending £100,000 on a set of SIEM tools which measure things you dont need may well be cheaper than spending £600,000 to build an organic security measurement program but only the latter actually delivers value. If you take the first option you are actually throwing away £100,000 for nothing and, eventually, you will be forced (often following a major breach) to spend the £600,000 anyway.

One of the common trends in security is “get it right the first time.” This is always the cheapest (over the longer term) option.

Top Tips for Building  a Security Measurement Plan

Keeping in mind the fact that every organisation is different, so the specifics and implementations will vary, there are some common themes you can consider with regards to developing a measurement and control plan.

• Decide, early on, what you want the measurements for. For example are you looking to judge the effectiveness of “control measures” (in general) or the effectiveness of a given tool (such as a new type of lock or a firewall).
• Identify and account for vested interests. Here you need to consider what the impact of the measurements will lead to and try to reduce the chance that they will be biased. For example, if you are measuring a firewall provided by a managed service partner, it is far from ideal to have the same service partner provide you with metrics. Even internally this is a risk if departmental budgets rely on the metrics in anyway.
• Decide what you want to measure. While wanting to measure everything makes sense, this is rarely ever going to be practical. Measurement requires resources to implement and resources to store logs, so you need to have a clear idea about what measurements are important enough to record.
• Decide on what you are going to measure against. The hardest but most important step. As mentioned, it is all but impossible to count the number of times something hasnt happened so you need to work out a way to benchmark your data to allow for some indication of progress.
• Decide what you are going to do with the data. Measurement is not an end – it is simply a way of informing your risk management decisions. Never measure for the sake of it, always make sure you have a clear idea of how you will deal with any changes, such as purchasing better security controls or adjusting business processes.

Once you have done all that, you can move onto the easy part – implementing the plan and start measuring!

Benchmarking

One final note – benchmarking. You need to determine some method by which you can work out what the effect of your security controls is. Realistically, this means you need to have some idea of what data you would expect to see without the control and then if it is different (ideally reduced) you can see the effectiveness of the control.

There are three common methods by which you can determine this – but none are ideal so frequently, each organisation will have to determine its own criteria.

1. Measure Against Past Experience. This is one of the more accurate methods to determine effectiveness, but involves a significant period of pain and only really works when you are bringing in a control to an existing situation. For example, if you have experienced several break-ins over the last year, you have data you can use to measure the effectiveness of a new door & locking system. The downside with this approach is that you need to have previous breach data.
2. Measure Against Peers. It is possible to generalise an assumption that you should be experiencing the same number of security breaches as organisations which are broadly similar to yours. You can then measure your control against this to determine its effectiveness. For example, if you know that organisations very similar to yours in both size and geography experience 10 break ins each year, you can use this to determine the effect of your controls. The problem here is that this data is rarely available and even when it is, is frequently unreliable.
3. Measure Against Industry Standards. Some organisations (such as the Information Security Forum) have developed security benchmarks which collect anonymous data from various organisations to get an indication of how many security incidents take place over a given time period. For example, if you know that you should expect 30 network intrusions a week, you can use this to measure the effectiveness of your firewall product. This approach can be excellent but as the data is largely self reported it is difficult to determine how applicable it is and this sort of benchmarking is almost totally confined to the IT Security domain.

Of course, you can opt for a combination of methods – the most important thing is that you pick a solution that works for you and your organisation.

Security Measurements – Summary

Measuring your security controls is essential to ensure they are effective rather than simply being a way to spend money. By putting together a dedicated and optimised security control measurement plan you can validate your security controls and ensure that all resources you spend on security are properly applied where they will count most.

Without one, and you really have no idea where the money you are spending is going and no idea if it is effective security.

Rather than spend money on vendor provided, “off-the-shelf” solutions it is much more effective to design one that fits in with your organisational philosophy and working practices. If you want advice or assistance in this then get in touch with Halkyn Consulting and take advantage of our experience in delivering security and risk management advice to diverse organisations across the globe.