RMIT Classification: Trusted
Lecture 9
Introduction to Cybersecurity
Governance
1
RMIT Classification: Trusted
Session Objectives
After reading completing the week contents, you should be able to :
• Analyses the challenges of security measurement
• Review kinds of measures used
• Perform security cost analysis
22
RMIT Classification: Trusted
1a. Security Metrics
• The lack of standards in the measurement of security is because it
has no dimensions or characteristics of its own.
• The size of a company’s security can be measured through its
financial impact but this is not enough without measuring other
aspects: profit, flexibility, added business value, asset protection,
etc.
33
RMIT Classification: Trusted
1b. Security Metrics
• However, measuring security is essential for good governance.
• Senior management requests reports that contain stable KPIs of
the adequacy of security.
• enable to adjust program and decide on security investments,
which need to be justified (explained) by some kind of
quantitative measurements of the benefits.
44
RMIT Classification: Trusted
1c. Security Metrics
• Metrics are also requested by the regulatory framework.
• Metrics will be used in reporting, dashboards, or targeted studies for
governance bodies to help them make decisions based on hard
facts.
55
RMIT Classification: Trusted
2a. Why Is It Difficult to Measure
Security?
• there are no universally accepted indicators to measure the
adequacy of the controls in place
• Incident observations (or their absence) or examining statistics
generated by technical security devices (e.g. the number of viruses
or attempted intrusions stopped) do not allow an opinion to be
formed on the adequacy of security.
66
RMIT Classification: Trusted
2b. Why Is It Difficult to Measure
Security?
• the diversity of security activities (from the strategic to the
operational level)
• the impossibility of delineating its scope (technical, human,
application, physical, etc.)
• problems with the measurement units or objects being measured.
77
RMIT Classification: Trusted
2c. Why Is It Difficult to Measure
Security?
• Incidents, risk level, leakage, threat, loss, etc. mean different
things to different people.
• companies do not share their data or statistics on vulnerability and
incidents because of the negative image this information conveys.
88
RMIT Classification: Trusted
2d. Why Is It Difficult to Measure
Security?
• Easily available indicators often do not answer the questions asked
by senior executives.
• Security devices generate numerous traces of activity, such as
patches applied, vulnerabilities detected, alerts, intrusion attempts,
volume of emails processed by antivirus tools, authentication errors,
signs of system access, privilege changes, etc.
99
RMIT Classification: Trusted
2e. Why Is It Difficult to Measure
Security?
• Log management solutions are able to correlate these traces,
generate reports, and thus ensure compliance with legal and
normative requirements.
• However, high-level metrics require additional efforts to link and
aggregate these different indicators.
10
10
RMIT Classification: Trusted
2f. Why Is It Difficult to Measure
Security?
• They must provide information on the expected results or
improvements that benefit the company.
11
11
RMIT Classification: Trusted
3a. Security Metrics Categories
• Financial metrics are concerned with the financial impact of
investments in security controls.
• Modelling allows investment decisions to be made based on a
model that is used instead of real values.
12
12
RMIT Classification: Trusted
3b. Security Metrics Categories
• Assessment (measurement) of the state of security consists of
evaluating the state of security as a whole or in a specific domain
(e.g. cybersecurity, continuity, or application security) using various
qualitative evaluation tools.
13
13
RMIT Classification: Trusted
3c. Security Metrics Categories
• Assumption-based metrics consists of guiding the measurement
process according to what we want to demonstrate (or are not able
to demonstrate).
• Posture or maturity evaluation compared with the standards or
regulatory framework is a qualitative metric on the level of maturity
compared with the standards or benchmarks of good practices.
14
14
RMIT Classification: Trusted
3d. Security Metrics Categories
• Audits can be classified in this category. Operational metrics
measures the operational effectiveness of security activities.
• Metrics of progress toward a goal is the establishment of metrics
or KPIs to measure the degree of progress toward a set objective.
• Cost analysis of costs in different security categories often
provides highly relevant indicators for governance.
• Benchmarking allows measurements among similar companies.
15
15
RMIT Classification: Trusted
3.1a. financial Metrics
• help assess investment opportunities in protection solutions
• security investments are considered justified if they cost less than
potential losses (inherent risk)
• Annualized loss expectancy (ALE) is an evaluation of potential
annual losses as the result of risks impacts.
16
16
RMIT Classification: Trusted
3.1b. financial Metrics
• Total cost of ownership ( TCO ) includes all
costs, hardware, software, and human
resources within the security
• Economic value added (EVA) can be
Expected trends of ALE and
calculated indirectly by observing changes TCO
in ALE and TCO I.e EVA=ΔALE/ΔTCO.
17
17
RMIT Classification: Trusted
3.1b. Calculation of ROSI Based on
Risk Analyses
• The benefit obtained from countermeasures is the difference
between the inherent risk (without security controls) and the
residual risk (after setting up countermeasures).
• If we consider that countermeasures modify the probability ( P ) of
occurrences, risk impacts (or ALE) can then be calculated as
follows:
18
18
RMIT Classification: Trusted
3.1c. Calculation of ROSI Based on
Risk Analyses
• Impact of the inherent risk = Cost of incident x P ( inherent risk )
• Impact of the residual risk = Cost of incident x P ( residual risk )
• Benefit ( EVA ) = Impact of inherent risk - Impact of residual risk
• ROSI = Benefit - Cost of countermeasures
19
19
RMIT Classification: Trusted
3.1d. Calculation of ROSI Based on
Risk Analyses
• The main difficulty with these methods is that the estimate of a loss
must be associated with its probability of occurrence for all the
business units and all the controls deployed, which may be very
inaccurate.
20
20
RMIT Classification: Trusted
3.1.2a. Protection Capacity Index
• Protection capacity (PC)=(inherent risk (IR)-residual risk
(RR))/inherent risk (IR)
• Protection capacity is ideal if the index tends to 1.
• This will happen if the impact of the residual risk is very small (or
tends to 0) or if the difference between the inherent risk and the
residual risk is very large.
21
21
RMIT Classification: Trusted
3.1.2b. Protection Capacity Index
• The idea behind this indicator is to attach a value to the means of
protection in place or the degree of risk mitigation.
• If we know the operational costs or the cost of controls to reduce
the risk, we will be able to follow the evolution of our protection
capacity with the evolution of investments.
• The IR − RR difference presents the risk reduction capacity.
22
22
RMIT Classification: Trusted
3.2a. Modelling
• Used to represent reality so as to
simulate or observe the behavior of a
system, make “what if” analyses, or
extrapolate the effects of decisions.
• Project setup phase increases cost while
operational phase decreases real cost
23
23
RMIT Classification: Trusted
3.2b. Modelling
• Another advantage of a model like this is its simulation capabilities.
• We can observe changes in the BE point and the evolution of
ROI by changing parameters such as the cost of the project and
the evolution of operational expenses
24
24
RMIT Classification: Trusted
Example (1)
• A new role-based access rights management system will
• Improve efficiency in granting privileges
• Automatically propagate privileges to target platforms
• Validate privileges at regular frequencies
• Offer more flexibility for new needs
25
25
RMIT Classification: Trusted
Example (2)
• A simple calculation of ROI can be made based on the following
data: project cost, annual operational cost of the new solution, and
annual operational cost of the current solution for the same
volume of processed access rights (same result). Operational
costs will be obtained by adding labor costs, infrastructure costs,
and indirect costs (e.g. errors).
26
26
RMIT Classification: Trusted
Example (3)
• Since it is difficult to accurately estimate the operational benefit of
a new solution, some assumptions and simulations can be made
to arrive at an ROI confidence interval
27
27
RMIT Classification: Trusted
3.3a. Measuring the State of Security
• Security program posture, capacity, and maturity –state of security
• analyzing risks allows us to understand our “enemies” (threats),
while security posture allows
• being compliant with a standard does not mean having adequate
security.
28
28
RMIT Classification: Trusted
3.3b. Measuring the State of Security
• The different standards or good practices can, however, be used
under certain conditions to assess security posture.
• The standards present the processes that are required but do not
propose evaluation criteria or gradations of conformity.
• There are also no recommendations on how to satisfy the
requirements.
29
29
RMIT Classification: Trusted
3.3c. Measuring the State of Security
• Some standards provide criteria for evaluating the security level.
• ISO 15408— Evaluation criteria for IT security (Common
Criteria) enables the security certification of a computer system
or product.
30
30
RMIT Classification: Trusted
3.3.1a. Maturity Models
• Maturity models generally evaluate processes according to a scale
of values
• Standards such as ISO 27001 or NIST may be used to establish a
list of processes or control objectives for which maturity will be
assessed.
31
31
RMIT Classification: Trusted
3.3.1b. Maturity Models
• The maturity model then proposes
evaluation criteria for each process
on a scale of values.
• Many examples of value scales
exist, such as the one proposed by
Different process maturity levels according to ISO
ISO/IEC 15504 Information 15504.
technology— Process assessment
32
32
RMIT Classification: Trusted
Example
Example of Maturity Evaluation Criteria (Control objective 5 Information
security policies of the ISO 27001)
33
33
RMIT Classification: Trusted
Example of representing the maturity
of controls (ISO 27001) (1)
• Tools to measure the maturity of a security program
• “Open Information Security Management Maturity Model (O-ISM3)”
from The Open Group,
• “A New Approach for Assessing the Maturity of Information
Security” from ISACA.
34
34
RMIT Classification: Trusted
Example of representing the maturity
of controls (ISO 27001) (2)
• Large consulting firms also offer
their own maturity measurement
models.
• The “CERT Resilience
Management Model (CERT-
RMM)”,form SW Eng institute for
evaluating the resilience
35
35
RMIT Classification: Trusted
3.3.2a. Security Index
• Indexes are used in different domains to aggregate the indicators
or values of their components.
• They are primarily used to present trends (stock market indexes,
real estate indexes, price evolution indexes, etc.).
36
36
RMIT Classification: Trusted
3.3.2b. Security Index
• As aggregates, they conceal the underlying details. Thus, a stock
market index can remain stable despite the opposite evolution of
two stock prices that are part of it.
• A security index will be able to summarize various indicators: risks,
operational effectiveness, costs, etc.
37
37
RMIT Classification: Trusted
3.3.2c. Security Index
• However, an index only makes sense if it aggregates indicators or
measurements of the same type.
• A security risk index could be constructed according to the
following formula:
• Risk index =High risk weight/Weight of all risks, where risk
wight= Σ(probability x impact) of risk
38
38
RMIT Classification: Trusted
3.4a. Assumption-Based Metrics
• “You can’t improve what you can’t measure”
• “You can’t measure if you don’t know what you’re looking for.”
• Setting measurement objectives facilitates the choice of metrics.
• A measurement goal that is simple and detailed makes it easier to
define the associated metrics.
• The best would be an objective defined as an initial assumption.
39
39
RMIT Classification: Trusted
3.4b. Assumption-Based Metrics
• “You can’t improve what you can’t measure”
• “You can’t measure if you don’t know what you’re looking for.”
• Setting measurement objectives facilitates the choice of metrics.
• A measurement goal that is simple and detailed makes it easier to
define the associated metrics.
40
40
RMIT Classification: Trusted
3.4c. Assumption-Based Metrics
• The best would be an objective defined as an initial assumption.
• In such a case, the metrics will aim to confirm or refute the
assumption.
• For example, if we consider that the level of awareness is not
adequate in a company department, we can set up a questionnaire
or a survey to “measure” it precisely.
41
41
RMIT Classification: Trusted
Assumption-Based Metrics (1)
• To facilitate metric definition, a strategy would be to:
• subdivide the initial assumption into subassumptions
• and then define the metrics in relation to these subassumptions
so as to be able to confirm or refute them
• Different assumption-based measuring methods exist, such as the
“McKinsey Diagnostic Method”, Goal– Question– Metric (GQM)
42
42
RMIT Classification: Trusted
Assumption-Based Metrics (2)
43
43
RMIT Classification: Trusted
Example.
To ensure that all critical
processes are taken into account
as part of disaster recovery
procedures following major
incidents, it was decided to
check whether service-level
agreements (SLAs) were defined
and tested.
44
44
RMIT Classification: Trusted
3.5a. Measuring Progress toward
Security Goals
• The balanced scorecard (BSC) is a popular tool for tracking
performance and advancing toward goals that support the
business strategy
• As a well-known management tool, it provides a formal basis to
establish and communicate results.
45
45
RMIT Classification: Trusted
3.5b. Measuring Progress toward
Security Goals
• It can be used to monitor security performance, thereby helping to
position security as an equal partner with other business units.
• As a tool for monitoring security objectives, it can also facilitate
management’s appropriation of security issues.
46
46
RMIT Classification: Trusted
3.5c. Measuring Progress toward
Security Goals
• Financial performance alone does not provide all the information
needed to assess the contribution of a unit to the consolidated
results of the company.
• The BSC approach advocates benchmarking according to the
objectives set in the following four perspectives: Operations, Client
Relationship, Evolution (Learning and Growth), and Finance.
47
47
RMIT Classification: Trusted
3.5d. Measuring Progress toward
Security Goals
• The four perspectives must contribute to supporting the company’s
strategy and vision.
• A core question is associated with each perspective to guide the
user in choosing the objectives and performance indicators that will
be applied.
48
48
RMIT Classification: Trusted
Four perspectives in the balanced
scorecard (1)
49
49
RMIT Classification: Trusted
Four perspectives in the balanced
scorecard (2)
• Operations Perspective:
• “How can we improve our security processes?”
• measure to improve the performances of our security processes
to better support the business and align with the company’s
strategy.
• measure process efficiency and associated costs.
50
50
RMIT Classification: Trusted
Four perspectives in the balanced
scorecard (3)
• Client Perspective:
• “How should security be perceived by our customers?”
• focus our goals and metrics on security processes impacting
internal or external customers or on activities that support
customer-centric business processes.
• Security operations must be perceived by our customers as
contributors to their own success
51
51
RMIT Classification: Trusted
Four perspectives in the balanced
scorecard (4)
• Evolution Perspective:
• “How can we improve our capacity to react to threats and
contribute to business opportunities?”
• measure our level of preparation and training to support the
changes imposed by the evolution of business. In this context,
the maturity objectives stated earlier may prove useful.
. 52
52
RMIT Classification: Trusted
Four perspectives in the balanced
scorecard (5)
• Finances Perspective:
• “How can security contribute to improving the financial
performance of the company?”
• measure returns on security investments or financial objectives.
• Since it is difficult to measure ROSI directly, we can focus on
objectives that improve the effectiveness of controls
contributing to the financial performance of business processes.
53
53
RMIT Classification: Trusted
3.6a. Measuring Operational
Performance
• Security operations can be assimilated into the controls set up
within the framework of a security program.
• For example, a Security Operation Center (SOC) monitors threat
status by means of intrusion detection technologies, data loss
prevention solutions, consoles, or incident tracking.
54
54
RMIT Classification: Trusted
3.6b. Measuring Operational
Performance
• Operational performance can be measured and presented through
figures, ratios, and trends.
• Operational efficiency metrics can be used as complementary
indicators for risk management and maturity analyses.
55
55
RMIT Classification: Trusted
Examples of Operational Efficiency
Measures
56
56
RMIT Classification: Trusted
3.7a. Security Cost Analysis
• The cost of security or TCO can be an important indicator for
governance, especially if it is related to other factors, such as :
• the evolution of the company’s overall expenses
• the number of employees,
• the evolution of risks,
• factors generating cost, etc.
57
57
RMIT Classification: Trusted
3.7b. Security Cost Analysis
• Security expenses
• Labor, overhead, installations, depreciation (direct costs)
• Internal services used (Indirect costs)
• IS service concerned (according to IS services offered)
• Key of the distribution of charges to other departments in the
company
58
58
RMIT Classification: Trusted
Security Cost Analysis (1)
• Cost accounting methods already in practice for other units of the
company can be used.
• In this way, security costs (expenses) can be analyzed using the
same standards and presented in the same familiar format to the
management and the board of directors.
59
59
RMIT Classification: Trusted
Security Cost Analysis (2)
• The costs or expenses can be distributed in different ways, taking
into account the objectives of the analyses that may be made later.
• Expenses distribution:
• Distribution of costs in different categories (direct, indirect)
• Detail and evolution of the costs of each category
60
60
RMIT Classification: Trusted
Security Cost Analysis (3)
• Change in expenses compared with other indicators such as
changes in turnover, number of employees, evolution of threats,
etc.
• Distribution of expenses related to business or geographic
units
61
61
RMIT Classification: Trusted
Security Cost Analysis (4)
• Evolution of expenses over several years
• Breakdown of expenses by service areas provided (IS,
continuity, SOC, physical security, IAM, etc.)
62
62
RMIT Classification: Trusted
3.8a. Benchmarking
• Refers to comparing companies or their respective processes that
offer the same services to gain insight into potential improvement.
• A set of quantified comparison indicators is produced, which
facilitates decision-making and the definition of objectives.
• Comparing companies in the same sector is a widespread
technique that is appreciated by company executives.
63
63
RMIT Classification: Trusted
3.8b. Benchmarking
• It makes it possible to measure and compare the results of
different strategies.
• Apart from comparing public financial results, it is difficult to
compare companies in operational areas, particularly IS
processes, because data on incidents or the means of protection
are not publicly available.
64
64
RMIT Classification: Trusted
3.8c. Benchmarking
• Nevertheless, surveys or studies conducted on cybersecurity or
maturity of others, and have knowledge bases on the practices of
their customers may help for benchmarking
65
65
RMIT Classification: Trusted
Benchmarking (1)
• there are several methods enabling benchmarking analyses in the
field of security.
• Studies or surveys conducted by different specialized firms on
• specific topics such as cybersecurity
• the maturity of certain processes
• knowledge bases on the practices of their customers.
66
66
RMIT Classification: Trusted
Benchmarking (2)
• These data are confidential, but they can offer anonymized
benchmarking services.
• Thematic seminars or conferences are also a place to exchange
and compare practices in different companies.
• Surveys are often conducted at these meetings, and the
results are made available to the participants.
67
67
RMIT Classification: Trusted
Benchmarking (3)
• Some companies agree to share information about
security processes with each other, especially if they are in
the same sector.
• This enables not only a comparison and clarification of
strategic approaches but also an exchange of experiences
in solving problems.
68
68
RMIT Classification: Trusted
Benchmarking (4)
• Business associations of companies in the same sector often
conduct studies of the practices of their members and share this
information.
• Chief information security officer (CISO) forums or associations
provide opportunities to conduct mini-surveys, providing
information on what others are doing.
69
69
RMIT Classification: Trusted
Benchmarking (5)
• External auditors also provide a source of information on practices
in other companies.
• They cannot transmit confidential data, but they may recall certain
trends observed elsewhere in their comments related to audit
findings.
70
70
RMIT Classification: Trusted
Benchmarking (6)
• In connection with a finding, they might mention how the company
compares with an average observed elsewhere.
• The company can also mandate an expert to carry out a study on
the positioning of a security service compared with similar offers at
other companies.
71
71
