How can we help experts estimate probabilities more accurately, and test whether AI models update their estimates appropriately as new evidence arrives?
This idea brings together three concepts.
Expert calibration means checking whether an expert’s probability estimates match observed outcomes. Across many events assigned an 80% chance, approximately 80% should occur. Assessing this relationship can reveal overconfidence or underconfidence and provide feedback to improve future judgments.
Jev is an AI decision model developed by TypeSafe. Given information and a focused question, it returns structured answers and probabilities. This makes it a candidate for studying how AI probability estimates compare with expert judgments. [1]
Bayesian conjugate pairs provide a mathematically convenient way to update uncertainty as evidence accumulates. For example, a Beta prior paired with a Binomial likelihood lets us update our belief about an event’s probability after observing successes and failures. [2]
The research idea is to connect these approaches in two directions.
First, could structured exercises, Bayesian updates, and feedback help experts become better calibrated? Experts could make initial estimates, receive new evidence, revise their judgments, and learn from how their estimates compare with outcomes.
Second, could conjugate Bayesian models provide controlled benchmarks for AI decision models such as Jev? We could specify an initial belief, introduce evidence step by step, and compare the model’s revised probabilities with the corresponding Bayesian updates.
You may begin with simple probability tasks, then explore more realistic decision scenarios. Calibration against outcomes and agreement with Bayesian updates may be assessed separately, or through a gamified system.
How to participate
Interested in developing this idea? See how to submit your Ideathon entry. Subscribe to EPM Research and learn how to receive future Ideathon posts.