Skip to content
Forecasting · retrospective_forecast

Retrospective ForecastComing soon

Resolved events inferred from dated information snapshots; graded by Brier skill.

Source
Authored
Score
Brier skill (normalized)
Ladder
horizon · horizon days · ≤30, 31–90, 91–180
Items
Not built yet
Dimensions
dossier_size (3), question_type (3)

No results yet: this benchmark is planned and is not part of any score. Everything on this page describes the plan and may change before it ships.

The plan

Draft spec

Written up in a draft specification; details may change once items are built and piloted.

What it will measure

Given a fixed set of documents dated before an event, how well a model infers what happened next. The events have already resolved, so grading is immediate, but they resolved after the model's knowledge cutoff.

How items will be built

  • About 100 dossiers of 6–20 documents (wire reports, official releases, statistics, filings), all dated on or before an information date, each carrying 2–4 questions.
  • A question is used for a model only if its information date falls after the model's knowledge cutoff (plus a margin). A closed-book check drops questions whose outcome the model already knows.
  • Every document is checked for dates and wording that give away the outcome, and outcomes must have been genuinely uncertain (a market price or base rate between 10% and 90%).
  • No search in any arm: the open web cannot be limited to what was known at the time.

How it will be scored

  • Brier skill against baselines: a uniform guess, the base rate, the prediction-market price at the information date where one existed, and the same model without the dossier.
  • Dossier lift (does the model use the documents?), calibration and sharpness.
  • Whether forecasts are asked as probabilities, as outcomes, or both is still to be settled by an experiment.

Example (burned: never used in a scored release)

Information date 2026-04-01. Twelve documents from January to March: two past rate statements, the latest inflation and employment releases, minutes, two board members' speeches, three wire reports and a budget statement. Will the central bank of Country X announce a cut to its policy rate at its meeting ending 2026-05-14?

AnswerYes. (Market price at the information date: 0.41; base rate: 0.35.)

A structural placeholder with fictional values. Real dossiers cover real resolved events.

Planned sizeAbout 100 dossiers carrying 300 questions, refreshed every quarter.