Software Engineering Institute | Carnegie Mellon University
Software Engineering Institute | Carnegie Mellon University

Digital Library

Javascript is currently disabled for your browser. For an optimal search experience, please enable javascript.

Advanced Search

Basic Search

Content Type

Topics

Publication Date

Technical Report

Quantifying Uncertainty in Expert Judgment: Initial Results

  • Abstract

    The work described in this report, part of a larger SEI research effort on Quantifying Uncertainty in Early Lifecycle Cost Estimation (QUELCE), aims to develop and validate methods for calibrating expert judgment. Reliable expert judgment is crucial across the program acquisition lifecycle for cost estimation, and perhaps most critically for tasks related to risk analysis and program management. This research is based on three field studies that compare and validate training techniques aimed at improving the participants' skills to enable more realistic judgments commensurate with their knowledge. 

    Most of the study participants completed three batteries of software engineering domain-specific test questions. Some participants completed four batteries of questions about a variety of general knowledge topics for purposes of comparison. Results from both sets of questions showed im-provement in the participants ' ' recognition of their true uncertainty. The domain-specific training was accompanied by notable improvements in the relative accuracy of the participants ' ' answers when more contextual information to the questions was given along with "reference points" about similar software systems. Moreover, the additional contextual information in the domain-specific training helped the participants improve the accuracy of their judgments while also reducing their uncertainty in making those judgments.

  • Download

Cite This Report

SEI

Goldenson, Dennis; & Stoddard, Robert. Quantifying Uncertainty in Expert Judgment: Initial Results (CMU/SEI-2013-TR-001). Software Engineering Institute, Carnegie Mellon University, 2013. http://resources.sei.cmu.edu/library/asset-view.cfm?AssetID=41102

IEEE

Goldenson. Dennis, and Stoddard. Robert, "Quantifying Uncertainty in Expert Judgment: Initial Results," Software Engineering Institute, Carnegie Mellon University, Pittsburgh, Pennsylvania, Technical Report CMU/SEI-2013-TR-001, 2013. http://resources.sei.cmu.edu/library/asset-view.cfm?AssetID=41102

APA

Goldenson, Dennis., & Stoddard, Robert. (2013). Quantifying Uncertainty in Expert Judgment: Initial Results (CMU/SEI-2013-TR-001). Retrieved December 19, 2014, from the Software Engineering Institute, Carnegie Mellon University website: http://resources.sei.cmu.edu/library/asset-view.cfm?AssetID=41102

CHI

Dennis Goldenson, & Robert Stoddard. Quantifying Uncertainty in Expert Judgment: Initial Results (CMU/SEI-2013-TR-001). Pittsburgh, PA: Software Engineering Institute, Carnegie Mellon University, 2013. http://resources.sei.cmu.edu/library/asset-view.cfm?AssetID=41102

MLA

Goldenson, Dennis., & Stoddard, Robert. 2013. Quantifying Uncertainty in Expert Judgment: Initial Results (Technical Report CMU/SEI-2013-TR-001). Pittsburgh: Software Engineering Institute, Carnegie Mellon University. http://resources.sei.cmu.edu/library/asset-view.cfm?AssetID=41102