Skip to content
Analytica > Blogs > Expert elicitation & probability assessment through treatment of Uncertainty

Expert elicitation & probability assessment through treatment of Uncertainty

Wherein lies the uncertainty with experts – in the situation that is being assessed or in the assessment itself? Expert elicitation and probability assessment can be a useful adjunct to empirical data on a problem or opportunity. However, the inherent subjectivity brings its own challenges. How expert opinions or beliefs are combined – and how they are selected in the first place – can have a major effect on the quality of the outcome.

Getting to grips with the probability distribution functions

The preliminary result of expert elicitation and probability assessment is the definition of a probability density function (PDF) that represents the expert’s belief about the matter at hand. The process for getting to such a PDF may vary, although some generally accepted methods exist, like the Stanford/SRI Protocol. To maximize compatibility between expert judgments, the quantity being assessed and the scale and unit are defined in the same way. To then model an expert’s opinion, the extremes of the expert’s degrees of belief are defined; so are the shape and percentiles of the PDF of the belief. Analytica gives modelers a wide variety of possibilities to define any particular probability distribution function, and to represent it graphically. This allows them to get confirmation from the expert that it is a fair representation of his or her degree of belief.

Combining degrees of belief from different experts

From a modeling point of view, Analytica makes it easy to combine probability distribution functions and model an outcome based on this aggregation. Uncertainty can be evaluated; sensitivity to  inputs detected; and an importance analysis run to see how inputs compare in terms of their influence on the overall result. However, the inputs themselves are conditioned by the subjectivity inherent in the expert elicitation and probability assessment. Further points to note are:

  • The proportion of experts with the same view is not necessarily an indication of the chance that view is correct
  • If experts have different opinions that cannot be reconciled, then different weights may be attributed to the probability distributions before combining them. But this is only possible if it is in accordance with the competence of the expert in each case – competence being again a subjective measure…

Surprise, surprise

Their individual opinions may differ, but expert elicitation and probability assessment typically have at least one thing in common: elicitation overconfidence. Tests done with experts using questions with known answers show that their answers fall outside the credible elicited range far more often than expected. This ‘surprise index’ would in theory be at 2 per cent of the total results (outside a 98% credible interval). Studies by Morgan and Henrion showed that real results ranged in fact between 5 and 55 per cent of the time. Other potential problems to overcome include ‘availability’ (subjective ease of recall of comparable situations), ‘anchoring’ (previous answers affect the next answer), representativeness bias (being fooled by an irrelevant ‘likeness’) and motivational bias (‘that’s the answer that I want’).

A possible solution – Heterogeneity and size of the expert panel

If you can sample randomly for a large number of simulations in an Analytica model, why not sample (quasi) randomly among a large number of different experts? When empirical data is more complete, a small expert panel may be acceptable. When there is little empirical data however, a single expert or even a small panel may not suffice; in this case, a broader range of expertise, experience and professional backgrounds may be the only way to results of reasonable quality for expert elicitation and probability assessment.

If you’d like to know how Analytica, the modeling software from Lumina, can help you to analyze and manage uncertainty of any kind, then try the free edition of Analytica to see what it can do for you.

Share now 

See also

air conditioner outdoor unit

Building electrification: heat pump technology

Lumina set out to build a useful tool to assess the benefits of heat pumps. Learn more about heat pumps and their impact.
More...

Heat pumps 101

Heat and cool your home while saving energy and reducing emissions by adopting heat pump technology. Learn more.
More...
Heatpump

Navigating the heat pump landscape

Fort Collins, Lumina, and Apex Analytics have created a tool to help reduce greenhouse gas emissions by optimizing building electrification programs.
More...

US gas leaks much larger than previously estimated

A new Stanford-led study on natural gas leak rates from oil and gas activity across a large fraction of the US are about 3x more than previous government estimates. The
More...

See also

Building electrification: heat pump technology

Lumina set out to build a useful tool to assess the benefits of heat pumps. Learn more about heat pumps and their impact.

More…

Decision making when there is little historic precedent

Learn how to make decisions and strategic plans in uncertain situations, where historical data is not available. See how to model this in Analytica with clarity and insight.

More…

Does GPT-4 pass the Turing test?

In 1950, Alan Turing proposed “The Imitation Game”, today known as the Turing test, as a hypothetical way of measuring whether a computer can think [1]. It stakes out the...

More…

What is Analytica

Analytica is a decision analysis modeling tool that helps you generate clearer and more justified results.

More…