Can simulated responses justify a new fee?
Copy experiments agree until a held-out test changes direction. The fee policy has no human outcome.
The question
Waymark is considering a new cancellation fee. Its historical experiments changed booking-page copy. A simulated panel produces response counts for both copy and the proposed fee. Human counts come from separate randomized user experiments with 1,000 eligible users per arm. All counts are authored teaching data; none came from a model or an employer.
| Human control | Human treatment | Simulated control | Simulated treatment | |
|---|---|---|---|---|
| April copy | 100 | 120 | 100 | 110 |
| May copy | 200 | 240 | 200 | 220 |
| June copy | 180 | 210 | 180 | 195 |
| July copy | 150 | 140 | 150 | 160 |
| New fee policy | Not measured | Not measured | 200 | 230 |
Each numeric cell is a response count out of 1,000 in its own arm. Human bookings and simulated responses are different outcomes. July is held out in the interactive evidence desk.
The simulated fee response rises by 3 percentage points. The early copy tests agree, so ship the fee.
“Audit the claim and recommend the next evidence to collect.”
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: Name the target outcome
What would you do here? Open the move you’d make.
Treat all response counts as one outcome
Tempting, but it misses something. The two counts were produced by different processes and do not estimate the same effect by design.
Separate human bookings from simulated responses
The strong move. The target estimand must stay distinct from the available proxy.
See how a strong candidate says it
“We need the effect of the fee on human booking behavior, with a defined eligible population and observation window. The simulated response is a proxy, not a measured human booking.”
Why it works: The target estimand must stay distinct from the available proxy.
Step 2: Inspect the held-out experiment
What would you do here? Open the move you’d make.
Compare whole held-out experiments
The strong move. An independent later experiment can expose disagreement hidden by the early cases.
Split control and treatment arms across training and validation
Tempting, but it misses something. Arms from one experiment do not provide an independent experiment-level check.
See how a strong candidate says it
“July’s human contrast is 140/1,000 − 150/1,000 = −1 point. Its simulated contrast is 160/1,000 − 150/1,000 = +1 point. The directions differ. These point estimates alone are not a significance test.”
Why it works: An independent later experiment can expose disagreement hidden by the early cases.
Step 3: Name what cannot be checked
What would you do here? Open the move you’d make.
Convert the simulated +3 points into a human lift
Tempting, but it misses something. No human policy data or justified transfer model is supplied.
Keep fee-policy human outcomes missing
The strong move. Surrogacy and comparability remain assumptions for the new intervention.
See how a strong candidate says it
“The fee changes an economic consequence, not just wording. We have no human fee outcomes. Historical copy checks cannot prove that the proxy captures every fee effect or that its mapping to behavior stays stable.”
Why it works: Surrogacy and comparability remain assumptions for the new intervention.
Step 4: Use the proxy for a bounded job
What would you do here? Open the move you’d make.
Shortlist, investigate and collect human evidence
The strong move. A useful screening tool can still be insufficient evidence for a rollout.
Ship the highest simulated response
Tempting, but it misses something. The winning proxy could harm an unmeasured user outcome.
See how a strong candidate says it
“Use the panel to shortlist variants. For a ship decision, pre-specify a human randomized test, the booking outcome, refund and retention guardrails, and the observation window. Investigate the July disagreement first.”
Why it works: A useful screening tool can still be insufficient evidence for a rollout.
The answer, in short
- Decision
- Use simulated responses to screen candidates; collect human fee-policy evidence.
- Because
- The held-out copy result disagrees in direction, and the proposed policy has no measured human outcome.
- What would change it
- Relevant human experiments, a defensible transfer analysis and guardrail evidence could support a later decision.
Common traps
- Reporting the simulated +3 points as human booking lift.
- Calling a sign difference a significance test.
Check your answer
- Did you keep the two outcomes separate?
- Did you calculate July’s −1 versus +1 point?
- Did you state the new-treatment assumptions?
- Did you leave the missing fee outcome missing?
Last reviewed Nov 10, 2026