Label Design and Windows

#The three windows

 history start                 cutoff = prediction time
      |--------- observation window ---------|-- gap --|------ outcome window ------|
      features use only this                  optional   the label is computed here

• Observation window: data the features may use. Often several sub-windows (last 7, 30, 90 days).
• Gap (blackout): time between prediction and the start of the outcome window. Use it when the action takes time (call scheduling, shipping) or when data arrives late (yesterday's events land today).
• Outcome window: where the label is measured. Its length is the horizon.

#Choosing the horizon

Try two or three horizons and compare positives per window, offline metric and business usefulness. A 7-day churn label may have 1% positives and high precision; a 90-day label 12% positives and weaker signal.

#Censoring

A row is right-censored when its outcome window has not fully elapsed. Treating it as negative biases the label toward 0 for the most recent rows.

Options:

1. Drop scoring dates whose window has not ended (score_date <= today - horizon). Simple, loses fresh data.
2. Use survival analysis with (duration, event) labels.
3. For monitoring, compute a partial-window metric and compare like with like.

#Label delay and training freshness

Training data ends at today - horizon - label_delay. With a 90-day chargeback delay and a 30-day horizon, the newest trainable row is four months old. Mitigations: a faster proxy label for monitoring (analyst labels in 2 days), weighting recent rows, or nowcasting the label.

#Multi-snapshot datasets

Scoring weekly means the training table should also be weekly snapshots: one row per entity per scoring date.

• Pros: matches deployment, multiplies positives, captures changing features.
• Cons: rows of one entity are correlated (group by entity in CV), long-lived entities dominate (weight by 1 / snapshots per entity if needed), and churners appear as negatives in early snapshots, which is correct.

import pandas as pd

def build_snapshots(customers, events, cancels, score_dates, horizon_days=30):
    """One row per (customer, score_date); features from events strictly before score_date."""
    rows = []
    for d in score_dates:
        active = customers[(customers.start_date <= d) & ((customers.end_date.isna()) | (customers.end_date > d))]
        past = events[events.ts < d]
        feats = (past[past.ts >= d - pd.Timedelta(days=90)]
                 .groupby("customer_id").agg(events_90d=("ts", "size"), last_event=("ts", "max")))
        snap = active[["customer_id"]].merge(feats, on="customer_id", how="left")
        snap["days_since_event"] = (d - snap.pop("last_event")).dt.days
        snap["score_date"] = d
        end = d + pd.Timedelta(days=horizon_days)
        churned = cancels[(cancels.cancel_date > d) & (cancels.cancel_date <= end)].customer_id
        snap["label"] = snap.customer_id.isin(churned).astype(int)
        rows.append(snap)
    return pd.concat(rows, ignore_index=True)

#Proxy labels

The thing you care about is often not observable (or too slow), so you train on a proxy:

For each proxy write down: when it diverges from the goal, how big the gap is, and which guardrail metric catches the divergence.

#Label noise

Human labels, delayed labels and heuristics are noisy. Estimate noise with a double-labelled sample (agreement rate, Cohen's kappa). Noise above ~10% caps achievable accuracy; prefer models and losses robust to it, and clean the most-confident disagreements first.

#Label SQL patterns

Event within a window after a snapshot

CASE WHEN EXISTS (
  SELECT 1 FROM events e
  WHERE e.entity_id = s.entity_id AND e.type = 'cancel'
    AND e.ts > s.score_ts AND e.ts <= s.score_ts + INTERVAL '30 days'
) THEN 1 ELSE 0 END AS label

Quantity over a forward window

COALESCE((SELECT SUM(o.amount) FROM orders o
          WHERE o.customer_id = s.customer_id
            AND o.ts > s.score_ts AND o.ts <= s.score_ts + INTERVAL '90 days'), 0) AS spend_next_90d

Survival label

LEAST(COALESCE(c.cancel_date, CURRENT_DATE), CURRENT_DATE) - s.start_date AS duration_days,
CASE WHEN c.cancel_date IS NOT NULL THEN 1 ELSE 0 END                      AS event

Eligibility filter (always): active at the snapshot, not already labelled positive, in the population you will score in production.