Advanced Search

Content Type

Topics

Publication Date

Quantifying Uncertainty in Expert Judgment: Initial Results

Abstract

The work described in this report, part of a larger SEI research effort on Quantifying Uncertainty in Early Lifecycle Cost Estimation (QUELCE), aims to develop and validate methods for calibrating expert judgment. Reliable expert judgment is crucial across the program acquisition lifecycle for cost estimation, and perhaps most critically for tasks related to risk analysis and program management. This research is based on three field studies that compare and validate training techniques aimed at improving the participants' skills to enable more realistic judgments commensurate with their knowledge. 

Most of the study participants completed three batteries of software engineering domain-specific test questions. Some participants completed four batteries of questions about a variety of general knowledge topics for purposes of comparison. Results from both sets of questions showed im-provement in the participants ' ' recognition of their true uncertainty. The domain-specific training was accompanied by notable improvements in the relative accuracy of the participants ' ' answers when more contextual information to the questions was given along with "reference points" about similar software systems. Moreover, the additional contextual information in the domain-specific training helped the participants improve the accuracy of their judgments while also reducing their uncertainty in making those judgments.

Cite This Report

Show Citation Formats

SEI

Goldenson, Dennis; & Stoddard, Robert. Quantifying Uncertainty in Expert Judgment: Initial Results (CMU/SEI-2013-TR-001). Software Engineering Institute, Carnegie Mellon University, 2013. http://resources.sei.cmu.edu/library/asset-view.cfm?AssetID=41102

IEEE

Goldenson. Dennis, and Stoddard. Robert, "Quantifying Uncertainty in Expert Judgment: Initial Results," Software Engineering Institute, Carnegie Mellon University, Pittsburgh, Pennsylvania, Technical Report CMU/SEI-2013-TR-001, 2013. http://resources.sei.cmu.edu/library/asset-view.cfm?AssetID=41102

APA

Goldenson, Dennis., & Stoddard, Robert. (2013). Quantifying Uncertainty in Expert Judgment: Initial Results (CMU/SEI-2013-TR-001). Retrieved July 24, 2014, from the Software Engineering Institute, Carnegie Mellon University website: http://resources.sei.cmu.edu/library/asset-view.cfm?AssetID=41102

CHI

Dennis Goldenson, & Robert Stoddard. Quantifying Uncertainty in Expert Judgment: Initial Results (CMU/SEI-2013-TR-001). Pittsburgh, PA: Software Engineering Institute, Carnegie Mellon University, 2013. http://resources.sei.cmu.edu/library/asset-view.cfm?AssetID=41102

MLA

Goldenson, Dennis., & Stoddard, Robert. 2013. Quantifying Uncertainty in Expert Judgment: Initial Results (Technical Report CMU/SEI-2013-TR-001). Pittsburgh: Software Engineering Institute, Carnegie Mellon University. http://resources.sei.cmu.edu/library/asset-view.cfm?AssetID=41102