Our summer experiment

If you’re a regular to scrumandkanban.co.uk, you’ll know that James and I have been talking about estimation and forecasting quite a bit. We’ve also run a few experiments on the topic.

Two books in particular have raised interesting ideas. Firstly, Surowiecki’s The Wisdom of Crowds talked about how crowds can beat individual ‘experts’ (given the correct characteristics). Secondly, Tetlock’s Superforecasting discussed how to be better than most in predicting events: one of the key points was to update your forecasts as new information comes to light.

So we decided to run our own experiment around those two topics. Conveniently, a major football competition begins on 10 June and delivers us a perfect opportunity to get predictions from stalwart fans and professional pundits, as well as from those who have a passing interest / no idea about it. James falls into the former camp; I’m in the latter.

We will measure each person’s forecasting ability using two factors:

1. Whether they pick the winner
2. How confident they were in their answer

Ignoring draws, we then use the above factors to calculate a participants score out of 100 (it’s an inverted Brier score multiplied by 100 for anyone who’s wondering). To play on people’s competitive nature, a leader board of the top 5 forecasters will be updated each day – with the overall winner getting a trophy-like prize.

We will report back our findings after the competition ends.

We are no longer accepting requests to join the experiment. Thanks to all who have agreed to participate.

Image by David Lowe, modified from images by nvodicka and Piotr Siedlecki