Metrics, Dashboards and Security
Like them or not, metrics are a fundamental part of every organisation. Security doesn’t get to a free pass. It is a rare CISO who doesn’t demand dashboards showing how all the security controls are performing. Therefore for most organisations, this is a fight long lost. This may not always be a bad thing either.
Really, dashboards are a good way of showing metrics. Metrics themselves aren’t inherently evil. As a result, you’d think dashboards would enhance your infosec work.
However, all to often the opposite is true. Metrics end up collected just for the sake of it. As a result, dashboards end up being nice shiny things for people to stare at. This is not good.
What do you mean?
First off, an example to explain this. Two of the most common metrics collected in security are patching and anti-virus status. Both are generally good things so people want to measure them. As a result, these are often cited in security guidance – such as CSO Online’s article. While this seems like a great idea it has problems.
- Patching. Nearly every program will measure things like the number of systems patched to “current” levels. Normally this means they’ve had all the patches applied within 48 or so hours. For most enterprises, hitting 95% here is a really good thing and will be green on the dashboards.
- Antivirus. Another common one where people measure the number of systems with recent AV updates. Most of the time this is “updates issued in the last 24 hours” with 98% compliance target. As a result, unless things break, it is often green.
The problem is that this dashboard doesn’t tell you anything useful. If your organisation has 200 systems, you could have 10 totally unpatched and 4 without any functional AV and still show green on the dashboard. One phishing campaign and 4 – 10 machines are compromised. All the while, your dashboards show green and the attacker steals data.
So, is there really any value in this obsession with metrics?
Actually, yes. Metrics do have a place in every organisation. Just not driving dashboard showing your executive view of security. Its important to pick good, effective metrics. It is more important to truly understand the message they give you.
Dashboards, what are they good for?
Actually, lots of things.
Metrics are best at showing things which are changing towards a target. They are brilliant at project measurements. Also, they are good at showing progress towards a goal. These are all areas where metrics excel.
When it comes to “steady state” measurements, it is a bit different. They can do it, but you need to realise they are telling you something different. Metrics tell you what your risk level is and help drive improvements. They help support compliance programs. This is all useful stuff.
However, most dashboards don’t give you situational awareness. Don’t let them trick you into thinking they do. Real operational dashboards take a lot of effort to create and manage. If you have an out-of-the box product, you don’t have this.
What should you do?
If your dashboards are basically compliance reports, then accept it. Compliance is good but it isn’t security. Educate yourself that green doesn’t mean secure, it just means things are operating. Use them to inform your risk management but remember 1 vulnerable device is enough to compromise your entire network.
Take time to decide if you want security metrics. If you do, fully understand what you want them for. Without this, your dashboards will be pointless. Try to avoid simply googling for ideas. Good security metrics come from your organisations controls & requirements – not a template.
If you really want security monitoring, then don’t go for dashboards, monitor your enterprise. Centrally log events, look for malicious activity and threat hunt. You can measure this but it will never look good on a dashboard.