Saturday 2012-06-16

Tetlock ran a prediction game last year and will run another one this year as last year's "blew the lids off" expectations. The game is IARPA funded, so there's probably some gov't analysts someplace who are upset.

Because now their metrics have an external comparison, and the pay-off is asymmetric:
if they beat the amateurs => yay, they beat some guys who drink beers and talk about the news;
if they lose => the professionals lost to a bunch of guys who drink beers and talk about the news.

In it Tetlock/his team offers the following mnemonic advice:

How You Can Maximize Your Forecasting Accuracy
Our guidelines are captured by the acronym CHAMP, which builds on five key insights:
- Comparison classes offer useful clues in initially estimating probabilities;
- Historical trends can help you update your probabilities;
- Averaging across opinions often boosts accuracy;
- Mathematical and statistical models often offer useful guidance; and
- Predictable biases are a fact of life and the best forecasters are aware of their biases.