CERT PODCAST SERIES: SECURITY FOR BUSINESS LEADERS: SHOW NOTES

A Workshop on Measuring What Matters

Key Message: To make better-informed decisions, take action, and change behaviors, leaders and staff need to demonstrate that metrics for managing cybersecurity risks derive from business objectives.

Executive Summary

This podcast summarizes the inaugural Measuring What Matters Workshop conducted in November 2014, and the team’s experiences in planning and executing the workshop and identifying improvements for future offerings. The Measuring What Matters Workshop introduces the Goal-Question-Indicator-Metric (GQIM) approach that enables users to derive meaningful metrics for managing cybersecurity risks from strategic and business objectives. This approach helps ensure that organizational leaders have better information to make decisions, take action, and change behaviors. [1]

Katie Stewart, Michelle Valdez, Lisa Young, and Julia Allen, the developers and facilitators of this workshop, are all members of CERT’s Cyber Resilience Management team. Further details about this workshop can be found in our workshop report [1].


PART 1: PURPOSE; GQIM PROCESS; WORKSHOP STRUCTURE

The first offering of the Measuring What Matters Workshop was conducted at the ISACA Information Security Risk Management Conference in Las Vegas, NV in November 2014 with 40 participants.

Purpose

Why do we measure?

But how do organizations, senior leaders, and managers figure out what are the right things to measure?

The purpose of this workshop is to describe a measurement approach that is driven by strategic and business objectives. Given this, participants learn how to derive operational metrics that help ensure that the organization is managing cybersecurity risk in the appropriate manner.

GQIM Process

The Goal-Question-Indicator-Metric (GQIM) Process used in the workshop is derived from CERT’s resilience measurement research that has been ongoing since 2010. It is based on earlier work performed by Vic Basili and Dieter Rombach [2], applied to software engineering.

The SEI’s Process Program added the “I” to Basili’s and Rombach’s GQM method in support of their Capability Maturity Model Integration (CMMI) model development and implementation.

CERT has been applying the GQIM process to operational resilience based on the CERT Resilience Management Model (CERT-RMM).

The key question when using the GQIM process is not “What metrics should I use?” A much more productive question is “What do I want to know or learn?”

Starting with a strategic or business objective, participants perform the following:

Workshop Structure

The workshop took place over one full day, consisting of

Based on pre-work and workshop discussions, participants selected a specific objective to work on in their small groups, with the intent of completing the GQIM process for that objective and having results to take back to their organizations.


PART 2: SCENARIOS; BUSINESS OBJECTIVES

Scenarios

We used the following scenarios during the workshop to illustrate and demonstrate the GQIM process:

We plan to add additional scenarios based on recent cybersecurity incidents to this body of work.

Business Objectives

About 50% of the workshop participants provided strategic and/or business objectives in advance of the workshop. The CERT team used these to help select candidate table topics for small group work.

Developing a well-formed objective to use for the GQIM process is one of the more challenging steps.

The objectives that were used in the workshop are as follows:


PART 3: WORKSHOP INSIGHTS; FUTURE PLANS

Workshop Insights

We made the following observations and had the following insights during the conduct of the workshop:

Future Plans

We are in the process of turning this workshop into a 2-day CERT public course. The participants identified a number of key improvements that we are in the process of implementing. Some of these include:

Course announcements will be published on the SEI and CERT websites in the coming months.


PART 4: CUSTOMER EXPERIENCES

We have had the opportunity to use the GQIM process with several customers, for the following purposes and with the following insights:

Resources

[1] Stewart, Katie; Allen, Julia; Valdez, Michelle; Young, Lisa. Measuring What Matters Workshop Report (CMU/SEI-2015-TN-002). Software Engineering Institute, Carnegie Mellon University, January 2015.

[2] Allen, Julia & Davis, Noopur.Measuring Operational Resilience Using the CERT Resilience Management Model (CMU/SEI-2010-TN-030). Software Engineering Institute, Carnegie Mellon University, September 2010.

Allen, Julia & Curtis, Pamela. Measures for Managing Operational Resilience (CMU/SEI-2011-TR-019). Software Engineering Institute, Carnegie Mellon University, June 2011.

Allen, Julia; Curtis, Pamela; Gates, Linda. Using Defined Processes as a Context for Resilience Measures (CMU/SEI-2011-TN-029). Software Engineering Institute, Carnegie Mellon University, October 2011.

CERT Podcast: Measuring Operational Resilience, October 2011.

SEI Training Course: Measuring What Matters: Security Metrics Workshop




Copyright 2015 Carnegie Mellon University