Evidence desk
Can simulated responses justify a new fee?
The panel prefers it. What do people do?
Waymark · experiment archive
Authored practice dataEach arm has 1,000 observations. Human bookings and simulated responses are separate outcomes.
| Experiment | Human bookings | Simulated responses |
|---|---|---|
| 100 → 120 | 100 → 110 | |
| 200 → 240 | 200 → 220 | |
| 180 → 210 | 180 → 195 | |
| Sealed holdout150 → 140 | 150 → 160 | |
| Not measured | 200 → 230 |
April copy
- Human
- +2 pp
- Simulated
- +1 pp
May copy
- Human
- +4 pp
- Simulated
- +2 pp
June copy
- Human
- +3 pp
- Simulated
- +1.5 pp
July copy
- Human
- Sealed holdout-1 pp
- Simulated
- +1 pp
New fee policy
- Human
- Not measured
- Simulated
- +3 pp
What this comparison measures
Each contrast is treatment responses divided by 1,000 minus control responses divided by 1,000. Multiplying by 100 gives percentage points. A +2-point human contrast and a +1-point simulated contrast are different observations. This exercise fits no calibration model, supplies no significance test and leaves the fee’s human effect unestimated.
Spotify: human effects and AI proxies