Can this model predict a cancellation?
The notebook uses a refund field written after the prediction.
The question
Waymark wants to prioritize outreach to new customers at booking time. The target is cancellation within the next 24 hours. An AI-generated notebook proposes a row-random split and includes every exported field. No model has been trained in this authored packet.
| Value available | Available at booking? | |
|---|---|---|
| Previous bookings | Before booking | Yes |
| Quoted fee | At booking | Yes |
| Refund amount | After cancellation | No |
| Final trip status | After the 24-hour window | No |
Authored schema. The cutoff is the instant a booking is accepted, not the time the export runs.
| Customers | Bookings | When | |
|---|---|---|---|
| Historical existing customers | 80 | 320 | January–February |
| Returning customers | 80 | 80 | March |
| New customers | 40 | 40 | March |
Authored counts. The 80 returning customers also appear in the historical set. New customers have no earlier bookings.
“Choose the usable fields and a validation design for outreach to new March customers.”
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: Freeze the prediction time
What would you do here? Open the move you’d make.
Remove post-cutoff fields
The strong move. A field can explain the label while being unavailable for the real decision.
Keep all fields that correlate with cancellation
Tempting, but it misses something. Correlation does not make a future value available at booking.
See how a strong candidate says it
“I am predicting a cancellation during the 24 hours after booking. Refund amount and final trip status are not available at that cutoff, so they cannot be predictors.”
Why it works: A field can explain the label while being unavailable for the real decision.
Step 2: Match the deployment population
What would you do here? Open the move you’d make.
Pool all 120 March customers
Tempting, but it misses something. Two-thirds of the pooled March customers have prior history; the intended outreach audience does not.
Evaluate the new-customer cohort separately
The strong move. A validation set must answer the deployment question.
See how a strong candidate says it
“The intended users are new customers. The 40 new March customers match that population more closely than 80 returning customers. I would not report a pooled March result as new-customer performance.”
Why it works: A validation set must answer the deployment question.
Step 3: Separate time and identity
What would you do here? Open the move you’d make.
Randomize booking rows across train and test
Tempting, but it misses something. Rows from the same customer can appear on both sides; this does not test new-customer generalization.
Use a future, new-customer holdout
The strong move. A row-random split can share identities and future context across the boundary.
See how a strong candidate says it
“Train on January–February information available at each booking, then validate on later new customers. Keep repeated customer identities out of an independent new-customer holdout. Wait for every 24-hour label window to close.”
Why it works: A row-random split can share identities and future context across the boundary.
Step 4: Define the outreach decision
What would you do here? Open the move you’d make.
Validate prediction, then test outreach impact
The strong move. A cancellation predictor is not an estimate of outreach impact.
Ship outreach if prediction accuracy is high
Tempting, but it misses something. A useful prediction can still trigger an ineffective or harmful intervention.
See how a strong candidate says it
“I would check calibration and the cost of missed versus unnecessary outreach at the actual contact capacity. Then test whether the outreach changes cancellations. Prediction quality alone does not establish the intervention’s benefit.”
Why it works: A cancellation predictor is not an estimate of outreach impact.
The answer, in short
- Decision
- Rebuild the feature set and use a later new-customer holdout.
- Because
- The export contains outcome information and a row-random split misses the target population.
- What would change it
- A different deployment audience or verified point-in-time availability would change the validation design.
Common traps
- Using export time as the prediction cutoff.
- Treating returning-customer performance as new-customer evidence.
Check your answer
- Did you remove the two unavailable fields?
- Did you identify the 40-customer target cohort?
- Did you separate time and customer identity?
- Did you distinguish prediction from outreach impact?
Last reviewed Nov 10, 2026