State TrackingComing soon
Generated narratives of object locations, quantities and knowledge across superseding events.
- Source
- Generated
- Score
- ModelAxes score, λ = 1
- Ladder
- structural · event count · 6, 12, 24, 48, 96, 192
- Items
- Not built yet
- Dimensions
- question_type (5)
No results yet: this benchmark is planned and is not part of any score. Everything on this page describes the plan and may change before it ships.
The plan
Draft specWritten up in a draft specification; details may change once items are built and piloted.
What it will measure
Keeping the state of a small world straight across a narrated sequence of events: where things are, how many there are, what each person believes, and which information is current.
How items will be built
- A generated world of 2–6 rooms and people, with containers, objects, countable goods and messages. Events move, give, consume, tell and supersede.
- Every story states the observation rule: people see only what happens in the room they are in, and believe what they last saw or were told.
- Questions ask for a location, a count, what someone believes (or what someone believes someone else believes), or a current value after a correction.
- The thing asked about always changes at least once, so answering from the starting state fails. Higher rungs add events, people and irrelevant events (up to 70%).
How it will be scored
- Answers come from a closed set (places, names, numbers, values) and are graded exactly; +1 / 0 / −1.
- x50 in events, per question type.
- The gap between where things are and where people think they are, and tagged errors for using a superseded value or losing count.
Example (burned: never used in a scored release)
Ana and Ben are in the kitchen. People see only what happens in the room they are in at the time. People believe what they last saw or were told, whichever came later. 1. Ana puts the key in the red box. 2. Ana leaves the kitchen. 3. Ben moves the key from the red box to the drawer. 4. Ben eats two of the five apples on the table. 5. Ana returns to the kitchen. 6. A note arrives saying the meeting is at 3 pm. 7. A second note arrives: "Ignore the previous note. The meeting is at 4 pm." Where does Ana think the key is?
The same story also asks: where the key is (the drawer), how many apples are on the table (3), and what time the meeting is (4 pm).
Planned size1,200 items (6 rungs × 200).