CERT PODCAST SERIES: SECURITY FOR BUSINESS LEADERS: SHOW NOTES

Comparing IT Risk Assessment and Analysis Methods

Key Message: Choose the IT risk assessment and analysis methods that are the best cultural fit for your organization.

Executive Summary

"Technical professionals are often called on to research, recommend, implement, and execute IT risk assessment and analysis processes. These processes provide important data used by management to responsibly grow and protect the business through good decision making for mitigating, accepting, transferring, or avoiding risk. These decisions must account for IT risks caused by emerging threats to the enterprise and vulnerabilities in the people, processes and technologies required for digital business." [1]

"Which method you choose for IT risk assessment and risk analysis is far less important than ensuring that the selected methodology is operationalized and a good fit for the corporate culture. The selected approach must be able to produce output that is meaningful to management, and supporting processes must account for assumptions, documentation, and potential gaming of the system. Tools should be leveraged, where possible, to ease method adoption." [1]

In this podcast, Ben Tomhave and Erik Heidt, research directors with Gartner Technical Professionals, discuss methods for IT risk assessment and analysis and comparison factors for selecting the methods that are the best fit for your organization.


PART 1: RISK ASSESSMENT METHODS AND COMPARISON FACTORS

Key Questions and Considerations

IT and information security professionals regularly ask:

Much of the interest in answering these questions is being driven by compliance, promises from marketing, and public relations hype.

Professionals often start with idealistic criteria for selecting the right process. It is more useful to make this selection based on the method’s ability to inform better business decisions.

An effective risk method should help determine priorities and help make the right choices. Some examples are:

Risk Assessment and Analysis Methods

Some of the risk assessment and analysis methods that the Gartner team analyzed include the following:

The hierarchy of risk management terms is risk management program or process, a subset of which is risk assessment. Risk analysis is a subset of risk assessment.

Comparison Factors

The factors that the Gartner team used to compare methods include the following:

  1. Method type: risk assessment framework/process or risk analysis method
    1. Framework: typically defines an overall risk management process including risk assessment and analysis. COBIT 5 is one example.
    2. Method: typically focused on performing a specific set of analysis functions. FAIR is one example.
  2. Analysis type: quantitative, qualitative, or both. MAGERIT is one example that supports both types of analysis.
  3. Cost
  4. Tools: available or not. OCTAVE provides spreadsheets to support its process. FAIR provides a range of automated tools.
  5. Special skills required: by risk analysts
    1. The more prescriptive methods are designed for a general audience. The more flexible methods typically require customization and therefore depend on more experienced risk analysts and training.
  6. Method flexibility: flexible/customizable vs. prescriptive
  7. Ramp-up time: time to complete the first analysis
  8. Cycle time: time required for each subsequent analysis

Observations from the Field

One of the most important factors to consider is the fit between the method and the way in which decisions are made in the organization. Most organizations do not have a one-size-fits-all risk assessment approach. In most cases, organizations use a two-tiered approach:

A two-tiered approach is important because you can’t run every decision through an in-depth, quantitative risk analysis. This may take days or week to gather data and perform.


PART 2: TRADEOFFS – METHODS VS. FACTORS

Specialized Skills and Method Flexibility

These 2 factors are linked to the extent to which a given method is more or less prescriptive. Those that are more prescriptive tend to be relatively straightforward to understand and implement.

A counterpoint to this is that the separate risk documents that are derived from ISACA COBIT 5 provide extensive depth of coverage in terms of specific processes. However, COBIT 5 and its supporting documents are all intended to be customized, which does require a higher level of skill.

Additional examples include:

Investing in customization may be worthwhile if the method will be used for some time and has broad applicability across the organization. It is important to take the time to tie the method into the way the organization does business.

Additional Tradeoffs and Considerations – Part 1

It is critically important to start with the end in mind. Leaders need to determine what they want in terms of assessment results. For example, an organization that has to complete 2,000 risk assessments on current third party suppliers every year must consider these time and resource constraints.

Another consideration is being able to use assessment results to categorize and prioritize areas for improvement.

If your market sector is highly regulated or regularly has high impact risks to mitigate, investing the time and energy in a method such as FAIR may be the right thing to do. On the other hand, if such compliance requirements call for hundreds, thousands, or tens of thousands of risk assessments in a short period of time, some compromises regarding method selection are necessary.

With many assessments to perform, outside formal training may not be an option. In addition, the use of automated tools for capturing and scoring risk questionnaires may be required to collect high volumes of data. Short cycle times are essential.

It is important to select a method that fits within the time and energy of available resources.

Ramp-up Time vs. Cycle Time

Qualitative methods tend to have shorter ramp-up times while quantitative methods tend to have longer times. Because they are more qualitative, NIST 800-30 and OCTAVE likely have shorter ramp-up times.

Of the methods reviewed, COBIT 5 has the longest ramp-up time due to the degree of customization required. NIST 800-30, OCTAVE, and MAGERIT have mid-range ramp-up times.

OCTAVE has a medium to long cycle time due to the amount of analysis that is required for each assessment. FAIR has a shorter cycle time once you get through the somewhat longer ramp-up time. This is due to the use of automated tools.

Some methods are more iterative. For example, FAIR provides the ability to determine the size and impact of hazards and threats early in the process, which can be fine-tuned as you go. The more questionnaire-based methods are generally “one and done.”

High impact and high residual risks are almost always going to require additional, more in-depth analysis.

Annual vs. Continuous Risk Assessment

While there may be situations where an annual risk assessment still makes sense, effective risk management is conducted as an ongoing process. Cycle time needs to be considered when you are assessing key systems and key applications on a continuous basis.

Some risk assessment tools and methods are now being integrated earlier into the system and software development process, for example, as part of applications security testing.


PART 3: IMPORTANT CHARACTERISTICS FOR SELECTING THE “BEST FIT”

Differentiators

Qualitatively, there is not a great deal of difference in terms of how all of these methods function.

The most important factor is cultural fit. For example:

Additional Tradeoffs and Considerations – Part 2

The following factors are also essential to consider:

It is important to differentiate between risk management activities and compliance checks. These tend to get conflated by business professionals.

Given it is expensive to collect, consider how you are going to use assessment results to fulfill the immediate need as well as how you might use this data in the future. Think about what you can add to an assessment today to add value and leverage for the future.

Resources

[1] Tomhave, Ben; Heidt, Erik; Robins, Anne. “Comparing Methodologies for IT Risk Assessment and Analysis.” Gartner, 30 January 2014.

Refer to the links to publicly available methods listed in Part 1 above, under Risk Assessment and Analysis Methods.


Copyright 2014 Carnegie Mellon University