Blog

Martin Pergler shares an article on interviewing experts to ensure best results.

Building strategic or financial planning models invariably involves agreeing on forward-looking estimates for crucial assumptions using #expertinsight, sometimes as single-point estimates, sometimes as multiple #scenarios, and sometimes as (approximate) probability distributions. Unfortunately, the way experts are interviewed often significantly limits the information gained from them, especially due to a style of questioning that activates human biases in dealing with #uncertainty and ignores experts’ wisdom about underlying dynamics.

Here is a sanitized and simplified role-play example. It starts with a typical interview script and the induced unhelpful monologue in the expert’s mind. Then we improve it.

Of course, this isn’t the full story. There are important considerations about the use of data to complement expert insight; and methods to encourage fruitful dialogue between experts, aggregate information and resolve differences of opinion, handling multiple factors rather than one, etc. In this brief note, I’m drilling down specifically on the estimate-elicitation questioning script. The example assumes the goal is to elicit a probabilistic estimate, which of course is not universally the case. But I find modelers often shy away from #probabilisticmodeling in part due to flaws precisely in their form of questioning, so please bear with me even if your ingoing hypothesis is that you’d be happy to get a (better) single point estimate or a couple of scenarios or a simple range.

Not-so-good dialogue example

Questioner (Q): Welcome Madam Expert. We’re soliciting your input today about a crucial cost assumption for us, crude oil price. We’re going to incorporate your insight in our marvelous predictive model. As our first question, what is your best estimate of Brent Oil price 3 years from now?

Expert’s internal monologue (M): Heck, how should I know? I hate these garbage-in, garbage-out exercises. You call your model marvelous, but if it’s based on guesses like this, it’s junk.

Expert, out loud (E): Well, Mr Questioner, it’s impossible to tell. There are so many uncertain factors, like A,B,C…

Q (interrupting): …Yes, of course. But we’d really need a quantitative estimate for 3 years from now….

Key points include:

• Forward-looking estimates
• Problemistic modeling 
• An improved dialogue example

Read the full article, Interviewing experts for “Riskifyied” Assumptions, on LinkedIn.