Does capping spend erase the intended effect?
A feature for travel agencies changes a few legitimate large orders.
The question
Waymark tests an agency booking feature. The pre-specified primary outcome is uncapped spend per eligible business account over 14 days. The randomization unit is the account. A separate capped-spend diagnostic limits each account to £300. The following small authored groups are not enough for a precise effect estimate.
| Control account (£) | Treatment account (£) | |
|---|---|---|
| Account 1 | 100 | 100 |
| Account 2 | 100 | 100 |
| Account 3 | 100 | 100 |
| Account 4 | 100 | 100 |
| Account 5 | 1000 | 2000 |
Rows align positions for display, not matched account pairs. The large values are verified legitimate agency purchases. The cap was chosen before the test.
“Compare the two metrics and explain what can be concluded.”
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: Calculate the planned outcome
What would you do here? Open the move you’d make.
Delete both large accounts as outliers
Tempting, but it misses something. The intervention was built for agencies; deleting their legitimate purchases can remove the intended mechanism.
Keep the large accounts in the primary mean
The strong move. The declared unit is the eligible account, including legitimate large spenders.
See how a strong candidate says it
“Control total is £1,400 across five accounts: £280 each on average. Treatment total is £2,400: £480 on average. The observed uncapped contrast is +£200 per account.”
Why it works: The declared unit is the eligible account, including legitimate large spenders.
Step 2: Calculate the capped outcome
What would you do here? Open the move you’d make.
Replace the primary result with the zero contrast
Tempting, but it misses something. A diagnostic does not silently replace the pre-specified uncapped spend outcome.
Report the capped result as a separate outcome
The strong move. Capping changes the quantity being estimated.
See how a strong candidate says it
“At £300 per account, each group totals £700 and averages £140. The capped contrast is £0. This is a different outcome, not a cleaner estimate of the same uncapped mean.”
Why it works: Capping changes the quantity being estimated.
Step 3: Keep uncertainty visible
What would you do here? Open the move you’d make.
Check validity and uncertainty
The strong move. Legitimate does not mean precisely estimated.
Declare success from the positive average
Tempting, but it misses something. A point estimate with a tiny heavy-tailed sample cannot establish a reliable business effect.
See how a strong candidate says it
“The +£200 observed contrast is driven by one large account per arm. I would inspect assignment, exposure and transaction validity, then quantify uncertainty with a method suited to this heavy-tailed outcome. Five accounts per arm do not prove the feature works.”
Why it works: Legitimate does not mean precisely estimated.
Step 4: Preserve the business question
What would you do here? Open the move you’d make.
Pre-specify the agency follow-up
The strong move. The next study should target the hypothesized mechanism without redefining success after seeing outcomes.
Choose the cap that makes the result most stable
Tempting, but it misses something. Variance reduction alone does not show the chosen outcome matches the decision.
See how a strong candidate says it
“Keep the uncapped primary result and capped diagnostic together. Use a pre-treatment definition of agencies to plan a sufficiently powered follow-up, with margin and refunds as guardrails. Do not select a cap or segment because it produces the desired answer.”
Why it works: The next study should target the hypothesized mechanism without redefining success after seeing outcomes.
The answer, in short
- Decision
- Report both outcomes and investigate a pre-specified agency follow-up.
- Because
- The raw and capped means answer different questions; neither removes small-sample uncertainty.
- What would change it
- Invalid transactions, a different primary estimand or adequately powered follow-up evidence could change the recommendation.
Common traps
- Calling a legitimate large purchase a data error.
- Choosing a cap after looking for a favorable result.
Check your answer
- Did you calculate £280 and £480?
- Did you calculate two capped means of £140?
- Did you state why the estimand changes?
- Did you retain the uncertainty of five accounts per arm?
Last reviewed Nov 10, 2026