ml-problem-framing-advisor

Most failed ML projects predict the wrong thing well: a target nobody can act on, a label that leaks the answer, a horizon shorter than the time needed to respond. This skill turns a business goal into a framing you can build, measure and ship: one decision, one action, one prediction made at a known moment for a known unit, a label defined as an observable event in a window, and a baseline plus metric that show whether the model is worth running. scripts/frame_check.py reviews a framing spec for 14 classic gaps; scripts/target_profile.py reads a candidate label column and says which task it implies.

#Trigger Conditions

Activate this skill when the user:

• Has a business goal and asks how to turn it into an ML problem ("use ML to reduce churn / predict demand / catch fraud / prioritise leads / route tickets")
• Asks what the target or label should be, how to define churn, default, conversion or "active", or how long the label window should be
• Asks whether a problem is classification, regression, ranking, forecasting, clustering or anomaly detection
• Wants a framing review of a project brief, notebook or PRD before modelling starts
• Has a model with good offline scores that nobody uses, or that degraded after launch because acting on it changed the data

Feasibility of the framed problem goes to ml-feasibility-assessor; picking the metric to ml-success-metric-selector; whether you need ML at all to ml-vs-rules-decision-advisor; the written scope document to ml-project-scoping-writer.

#Workflow

1. DECISION   -> who will do what differently because of the prediction, how often, at what capacity
2. UNIT/TIME  -> one row = which entity at which moment; when is the model called (batch or event)
3. LABEL      -> observable event or quantity, outcome window, label delay; write it as a query
4. TASK       -> map label to task type; python scripts/target_profile.py data.csv --target y
5. FEATURES   -> only data that exists at prediction time; mark when each source is written
6. BASELINE   -> today's rule or human process, measured on the same rows
7. METRIC     -> the decision's operating point (top-K, threshold, cost), plus guardrails
8. RISKS      -> proxy gap, feedback loop, selection bias, fairness, label drift
9. CHECK      -> python scripts/frame_check.py spec.json   (fill --template first)

#The Framing Canvas

Answer every line before writing model code. A blank line is a finding.

Domain-by-domain filled canvases (churn, demand, fraud, lead scoring, ticket routing, predictive maintenance, credit, pricing) are in framing-canvas.md.

#Business Ask -> Task Type

Two re-framings fix most bad projects:

• Predict the decision's input, not the KPI. "Increase revenue" is a KPI; "probability this lead converts in 14 days" feeds a ranking that sales acts on.
• Predict who to treat, not who will churn. If calling a customer changes the outcome, a churn score ranks people who leave anyway (lost causes) and people who never would (sure things) next to the persuadable ones. When treatment effect is the decision, frame it as uplift (uplift-model-builder).

Full catalog with label shapes, losses, metrics and traps per task: task-type-catalog.md.

#Define the Label as a Query

Write the label as the SQL or code that produces it. If you cannot, the label is not defined.

          observation window          gap      outcome window
  |----------------------------------|  |  |------------------------|
  features built from here           ^  scoring date (prediction time)
                                     |
                                     `- nothing after this line may enter a feature

-- One row per active subscriber per weekly scoring date; label = cancels within 30 days.
SELECT s.customer_id,
       d.score_date,
       CASE WHEN c.cancel_date >  d.score_date
             AND c.cancel_date <= d.score_date + INTERVAL '30 days' THEN 1 ELSE 0 END AS churned_30d
FROM scoring_dates d
JOIN subscriptions s
  ON s.start_date <= d.score_date
 AND (s.end_date IS NULL OR s.end_date > d.score_date)          -- active at prediction time only
LEFT JOIN cancellations c ON c.customer_id = s.customer_id
WHERE d.score_date <= CURRENT_DATE - INTERVAL '30 days';          -- outcome window fully observed

Rules that keep labels honest:

• Population = who you would score. Only rows that were eligible at prediction time (active, not already churned, not already flagged). Training on everyone inflates scores and breaks at deployment.
• Window fully observed. Drop scoring dates whose outcome window is not over yet; otherwise recent rows look negative (right censoring). If waiting is not an option, use survival analysis.
• Voluntary vs involuntary. "Cancelled" mixes customers who chose to leave with failed payments; split them if the action differs.
• Snapshot repetition. Weekly snapshots of the same customer are correlated; group by customer in CV (cross-validation-strategy-designer).
• Label delay sets the retraining and monitoring cadence; if it is months, add a faster leading indicator.

Window sizing, censoring, label delay, proxy labels and multi-snapshot designs: label-design-and-windows.md.

#Features Must Exist at Prediction Time

For every source ask: when is this field written, and can it change after the prediction?

Build features as-of the scoring date from event logs or point-in-time snapshots (time-based-feature-builder); scan a dataset for suspicious predictors with target-leakage-detector.

#Framing Pitfalls

Long-form explanations with detection steps: framing-pitfalls.md.

#Output Format

When framing a problem, answer with:

1. One-sentence framing - "At <prediction time>, for each <unit>, predict <label in a window> so that <who> can <action> for the top <capacity>."
2. Framing canvas - the table above filled in, with unknowns marked TBD (owner).
3. Label definition - SQL/pandas producing the label, the population filter and the outcome window.
4. Task type and why, plus the rejected alternatives (e.g. why not regression, why not uplift).
5. Feature availability table - source, when it is written, safe/unsafe.
6. Baseline and metric - today's process, the primary metric at the operating point, guardrails.
7. Risks - leakage, proxy gap, feedback loop, selection bias, fairness, each with a mitigation.
8. Next steps - feasibility check, baseline, first model; paste frame_check.py output when a spec exists.

#Scripts

python scripts/frame_check.py --template > spec.json     # filled-in example spec to edit
python scripts/frame_check.py spec.json                  # human-readable findings
python scripts/frame_check.py spec.json --json           # machine-readable
cat spec.json | python scripts/frame_check.py -          # stdin

Checks 14 rules: FR001-FR004 blocking (no decision, unmeasurable target, no unit, no prediction time), FR005-FR008 high (no horizon or horizon shorter than time to act, label delay far longer than horizon, features written after prediction time or named like outcomes, no baseline), FR009-FR012 medium (model-score metric instead of a decision metric, unknown action capacity, unexplained proxy label, feedback loop without a holdout), FR013-FR014 low (no owner, task type disagrees with the target wording). Verdict: NOT READY, NEEDS WORK or READY TO PROTOTYPE. Exit codes: 0 no blocking/high findings, 1 blocking or high findings, 2 usage or read error.

python scripts/target_profile.py data.csv --target churned
python scripts/target_profile.py data.csv --target price --json
python scripts/target_profile.py - --target label < data.csv

Infers the label kind (binary, multiclass, ordinal, count, continuous, ID-like, constant) and the task type, prints the class balance or the distribution (quantiles, skew, zero share) and the metric family, and warns about too few minority examples, missing labels, rare classes, heavy skew, zero inflation and extreme values. Standard library only. Exit codes: 0 usable target, 1 blocking problem (constant, ID-like, < 30 minority examples), 2 usage or read error.

#Additional Resources