Vulnerability Data Analytics

Let’s start with metrics and KPIs (key performance indicators).

Metrics vs KPI

#

Metrics and KPI are closely related concepts. It is commonly used in performance measurement and management, but they have distinct differences.

Metric (definition)

• A metric is a quantifiable measurement that provides information about a specific aspect of a business or process.
• Can be used to track and assess performance but may not necessarily be directly tied to the strategic goals.
• Can be objective or subjective. For example, data points like revenue, website traffic, or customer satisfaction, which can be measured using quantitative or qualitative methods.

KPI (Key Performance Indicator)

• A KPI is a specific type of metric that is directly linked to strategic goals and objectives.
• Used to evaluate the performance of a specific area of a business in achieving its strategic targets.
• Typically objective and well-defined. They are quantifiable and tied to specific, predetermined targets that are relevant to a strategic direction.

Metrics can cover wide range of data points, and not all metrics are critical to success, such as some metrics may be useful for day-to-day operations but not necessarily indicative of overall performance.

KPIs are carefully selected metrics that have a direct impact on strategic goals and they are crucial in measuring progress toward those goals.

Usages/Examples

#

Metrics are:

• Used for tracking and monitoring various aspects of a business.
• (but) not all metrics are equally important. DSome metrics may have more significance than others.

For instance, the web traffic metric that measures the number of visitors to a website. While it provides valuable information about website performance, it may not be directly linked to the organization’s strategic goals.

Another example. The employee satisfaction score metric that assesses the satisfaction of employees within a company. It is important for HR management but may not be a KPI (unless employee satisfaction is a strategic goal).

KPIs are:

• the few key indicators that are essential for measuring success and ensuring alignment with strategic objectives.

For instance, the Monthly Revenue Growth Rate is a KPI, and is directly used to measures the company financial performance. Its alignment with the strategic goal of increasing revenue by a certain percentage each month.

Another example is Customer Churn Rate KPI. It measures the percentage of customers who stop using a product or service, directly impacting the strategic goal of retaining customers and increasing customer lifetime value.

VulnMgmt Report

#

In Vulnerability Management, both metrics and KPIs are used to track/measure the security state.

A Vulnerability Management Report is aimed to achieve several objectives to ensure effective identification, assessment, and mitigation of security vulnerabilities within an organization’s systems and infrastructure. It also provides holistic view including root cause analysis (RCA), as well as recommended actions to address different purposes and implications.

By carefully selecting and tracking the right KPIs, organizations can improve their security state and achieve their strategic goals in term of vulnerability and risk management.

VM Metrics

#

Metrics are used to provide the raw vulnerability data within a specific timeframe (such as weekly or monthly).

Vulnerability Scan

#

Below are the common metrics.

1. The number of scanned assets.
2. The number of live assets.
3. The number of live assets by OS.
4. The number of new assets discovered.
5. The number of assets removed.
6. The number of successful authenticated scan assets.
7. The number of vulnerability instances (unclosed).
8. The number of new vulnerabilitites found.
9. The number of vulnerabilities closed (remediated).
10. The number of re-open vulnerabilities.

Vulnerability Management

#

Below are the common metrics that categorized into 5 groups.

1. Total (accumulated):

   • The number of (unique) CVE by severity.
   • The number of exploitable CVE.
   • The number of CVE by different OS.
   • The number of vulnerable products by vendor.
   • The number of vulnerable assets.

2. New:

   • The number of new (unique) CVE.
   • The number of new exploitable CVE.
   • The number of new CVE by different OS.
   • The number of new vulnerable products by vendor.
   • The number of new vulnerable assets.

3. Top-10 (or top-X):

   • The top-10 CVE by occurence.
   • The top-10 exploitable CVE.
   • The top-10 CVE by different OS.
   • The top-10 vulnerable products.
   • The top-10 vulnerable vendors.

4. The Most:

   • The top-10 assets with most vulnerabilities.
   • The top-10 assets with most vulnerabilities by different OS.

5. Exceptions:

   • The number of identified False Positive (FP) findings.
   • The number of risk-accepted (RA) findings.

The 5 categories can be repeated for different departments or business units or applications.

VM KPI

#

KPIS are specific VM metrics to track progress towards strategic goals in Vulnerability Management. Usually it provides a rating against our strategic goals, such as the progress of remediation.

1. Mean Time To Remediate (MTTR) or Time to Remediate (TTR)

   • Measure the average or median time to address and remediate identified vulnerabilities.
   • Shorter MTTR/TTR indicates higher efficiency in reduceing the likelihood of exploitation.

2. Vulnerability Resolution Rate

   • Measure the percentage of vulnerabilities that have been remediated within spcific timeframe.
   • Provides insight into the effectiveness of VM program where higher resolution rate indicates a proactive and successful approach.

3. Risk Reduction

   • Measure the overall overall risk reduction in term of percentage.
   • Align risk reduction as organization security objectives in enhancing overall security posture.

4. Vulnerability Discovery Rate

   • Measure the rate at which new vulnerabilities are discovered over time.

5. Re-open Vulnerability Rate

   • Identify the percentage of vulnerabilities that reoccur after being remediated, indicating potential weakness in remediation process.

6. Authenticated Scan Rate

   • Measure the rating of successful authenticated scan which indicate the comprehensiveness and accuracy of the vulnerability assessment process.

7. Compliance SLA

   • Measure against the security policy or regulatory requirements.

8. False Positive (FP) Rate

   • Measure the percentage of reported vulnerabilities that, upon further investigation, are determined to be false positives.

9. Risk-Accepted (RA) Rate

   • Measure the percentage of reported vulnerabilities that cannot be remediated due to whatever reason.

Again, the KPI above can be repeated for different departments or business units or applications.

False Sense of Security

#

Metric and KPI are both crucial measurements that provide valuable insights in term of VM operation and management. However, IMO, below are some of the useless or ineffective measurements that may lead to false sense of security.

1. Average CVSS Score

   • May over-simplify the analysis, as vulnerabilities affect system differently based on characteristics and functions.
   • May create false-sense of security when low-severity vulnerabilities are numerous.

2. Vulnerability Closure Rate

   • A high closure rate does not necessary indicate the most critical vulnerabilities are being addressed.
   • May lead to false sense of security and does not consider the risk associated with each vulnerability.
   • May also lead to quantity over quality in remediation efforts.

3. Total number of vulnerabilities.

   • The sheer quantity of vulnerabilities does not necessarily correlate with the severity or risk associated with those vulnerabilities and does not provide a clear understanding of a company risk posture..
   • May lead to inefficient resource allocation and inability to prioritize remediateion efforts effectively.

4. Scan Coverage Rate

   • May not provide insights into the effectivesness of scan quality.

Why This is Important?

#

How would you handle those vulnerabilities with no patch available? Have you ever seen a Vulnerability Management team simply groups them as False Positive?

I have seen this, and I call it Security through obscurity.

The reason behind this is, they have mixed up between Vulnerability Management and Patch Management.

And usually this is commonly found at those company where VM is managed by (untrained) Patch Management team. And it is also partly due to the KPI (setup by top management) to track those monthly vulnerability patching progress. Remember Goodhart’s Law?

To me, monthly patching progress is not a KPI (and shouldn’t be). It is just one of the metrics. This is the reason why it is important to understand the distinct between metric and KPI.

Summary

#

In general,

• Metrics provide raw data in trending and measure the coverage and effectiveness in VM.
• KPI provides rating towards some strategic goals and measures the progress and efficiency in VM.
• Report provides holistic view of the entire exercise including effectiveness, efficiency, RCA and recommendations.
• Ineffective metrics and KPIs may lead to false sense of security.

In summary, metrics are general measurements used to track various aspects of a business, while KPIs are a subset of critical metrics that are specifically chosen to evaluate an organization’s progress toward strategic goals. KPIs play a more focused and strategic role in performance management.