search menu icon-carat-right cmu-wordmark

Uncertainty Quantification in Machine Learning: Measuring Confidence in Predictions

August 2021 Podcast
Eric Heim

Eric Heim, a senior machine learning research scientist at the Software Engineering Institute at Carnegie Mellon University, discusses the quantification of uncertainty in machine-learning (ML) systems.

“When you are developing and training these machine-learning systems, it is not easy to predict when these models are going to be wrong. One of the goals of uncertainty quantification is to develop techniques that can express some degree of uncertainty in their predictions. The consumers of those predictions, either a human being or a software component, will be more informed about when the predictions will potentially be incorrect.”

Publisher:

Software Engineering Institute

Listen

Watch

Abstract

Eric Heim, a senior machine learning research scientist at the Software Engineering Institute at Carnegie Mellon University, discusses the quantification of uncertainty in machine-learning (ML) systems. ML systems can make wrong predictions and give inaccurate estimates for the uncertainty of their predictions. It can be difficult to predict when their predictions will be wrong. Heim also discusses new techniques to quantify uncertainty, identify causes of uncertainty, and efficiently update ML models to reduce uncertainty in their predictions. The work of Heim and colleagues at the SEI Emerging Technology Center closes the gap between the scientific and mathematical advances from the ML research community and the practitioners who use the systems in real-life contexts, such as software engineers, software developers, data scientists, and system developers.

 

About the Speaker

Eric Heim

Eric Heim


Dr. Eric Heim is a senior machine learning research scientist at the SEI’s Emerging Technology Center.  Before arriving at ...


Dr. Eric Heim is a senior machine learning research scientist at the SEI’s Emerging Technology Center.  Before arriving at SEI, Heim led a basic and applied machine learning research group at the Air Force Research Laboratory, Information Directorate. Heim earned a doctoral degree in computer science in 2015 from the University of Pittsburgh. Heim’s research focuses on practical issues of applying machine learning methods to real-world environments.  Such issues include: data-scarcity, model robustness, learning from structured data, and enabling models to express and reason about uncertainty in their predictions.  Heim has applied his work to problems ranging from identifying man-made structures from LiDAR measurements of terrain to estimating RF signal propagation effects from aerial imagery.

Read more