Should you extend the test until it turns green?
The planned fourteen-day result is inconclusive. The team wants a third week and a new success metric.
The question
Waymark pre-specifies a 14-day fixed-horizon randomized copy test. The primary outcome is completed bookings per eligible user; a +1 percentage-point improvement is the business threshold. The plan also requires a power calculation and uncertainty analysis. At day 14, the team proposes extending the same test because the result looks weak. Only the authored counts below are available.
| Users per arm | Control bookings | Treatment bookings | |
|---|---|---|---|
| Day 7 | 500 | 60 | 70 |
| Day 14 — planned end | 1000 | 120 | 125 |
| Day 21 — unplanned extension | 1500 | 180 | 210 |
Authored cumulative data, not three independent experiments. No interval, statistical test or power calculation is supplied.
Day 21 is up 2 points. Extend the test when needed and report a winning result. If bookings stay uncertain, use page views as the primary metric.
“Review the readout against the original design and state what evidence is still missing.”
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: Read the planned endpoint
What would you do here? Open the move you’d make.
Call the treatment ineffective because it missed +1 point
Tempting, but it misses something. The count table alone does not rule out larger or smaller effects.
Report +0.5 points and missing uncertainty
The strong move. An observed point estimate is not proof of either equivalence or a real improvement.
See how a strong candidate says it
“At day 14 the rates are 12% and 12.5%, a +0.5-point observed contrast. That is below the stated +1-point business threshold, but the packet does not show how much uncertainty surrounds it.”
Why it works: An observed point estimate is not proof of either equivalence or a real improvement.
Step 2: Keep the stopping rule
What would you do here? Open the move you’d make.
Retain the pre-specified endpoint
The strong move. A repeated look is not an independent replication.
Extend until the point estimate clears the threshold
Tempting, but it misses something. A data-dependent extension changes the procedure and its error properties.
See how a strong candidate says it
“Day 7 shows +2 points and day 21 again shows +2 points. These are cumulative looks at one test. Selecting the favorable look after seeing the data changes the fixed-horizon procedure.”
Why it works: A repeated look is not an independent replication.
Step 3: Separate a new question
What would you do here? Open the move you’d make.
Keep exploration separate from the primary claim
The strong move. Changing the outcome after seeing results changes the question.
Swap to whichever metric looks favorable
Tempting, but it misses something. The new metric was not the declared decision criterion.
See how a strong candidate says it
“Page views can be exploratory if clearly labeled, but cannot replace completed bookings as the original primary outcome. Preserve the day-14 readout, record the deviation and use later evidence as exploratory unless a valid sequential design was specified.”
Why it works: Changing the outcome after seeing results changes the question.
Step 4: Plan the next test
What would you do here? Open the move you’d make.
Copy the same arbitrary fourteen-day duration
Tempting, but it misses something. Duration depends on traffic, detectability and the design; fourteen days is not a universal requirement.
Set the design before collecting more data
The strong move. A useful next study starts with a decision and design, not a desired result.
See how a strong candidate says it
“Before rerunning, use the baseline, business-relevant effect, chosen uncertainty/error targets and assignment structure for a power plan. Pre-specify duration or a valid sequential rule and include the relevant refund/retention guardrails.”
Why it works: A useful next study starts with a decision and design, not a desired result.
The answer, in short
- Decision
- Keep the day-14 primary readout; label the extension and metric changes as exploratory.
- Because
- The procedure and question changed after results were seen, and uncertainty is not supplied.
- What would change it
- A pre-specified valid sequential plan or a new well-designed experiment could support a different analysis.
Common traps
- Adding cumulative sample sizes as if each row were independent.
- Confusing +2 percentage points with statistical significance.
Check your answer
- Did you calculate +0.5 points at the planned end?
- Did you keep the cumulative looks distinct from replications?
- Did you preserve the primary outcome?
- Did you identify the missing uncertainty and power plan?
Last reviewed Nov 10, 2026