IncentivesComing soon
Finding planted perverse incentives and gaming vectors in described plans and mechanisms.
- Source
- Authored
- Score
- Recall (normalized)
- Ladder
- Not decided yet
- Items
- Not built yet
- Dimensions
- document_type (6)
No results yet: this benchmark is planned and is not part of any score. Everything on this page describes the plan and may change before it ships.
The plan
Draft specWritten up in a draft specification; details may change once items are built and piloted.
What it will measure
Finding what is wrong with a plan, policy or design: perverse incentives, ways to game it, unintended consequences and missing caveats.
How items will be built
- A 200–800 word document (an incentive scheme, a public policy, a business plan, a technical design, a study design or a contract) with 3–6 planted flaws.
- Each flaw has a severity (1 minor, 2 significant, 3 plan-breaking) and match criteria that name the mechanism, not just the topic.
- Two reviewers read each document without the key. Every planted flaw must be found by at least one of them, and any serious unplanted problem they find is added to the key or edited out.
- The model lists at most 8 problems, most important first, each in at most 40 words.
How it will be scored
- A judge matches each listed problem to a keyed flaw, or marks it as a valid unkeyed problem, a repeat or invalid. The judge must agree with human labels at κ ≥ 0.85.
- Severity-weighted recall, recall of plan-breaking flaws, recall within the first three listed items, and the rate of invalid claims.
Example (burned: never used in a scored release)
To reduce support backlog, the company will pay each support agent a $5 bonus for every ticket they close, starting next month. Tickets can be closed by the agent at any time. Customers can reopen a closed ticket within 7 days. Agents choose which tickets to take from the shared queue.
Planned size60 documents to start (10 per type, about 250 planted flaws), growing toward 120.