Skip to content
v0.1.3latest release121 skills4 categories6 sub-categories247 tested scripts397 reference docsContact us
Vector Agentic Skills
Submit a skill

ml-project-scoping-writer

Write the scoping document (one-pager, PRD section or project charter) for a machine learning project: summary, the decision and its users, success criteria with a primary metric at the operating point, target, baseline, guardrails and online confirmation, data sources with availability at prediction time and the label definition, a phased simplest-first approach, explicit in-scope and out-of-scope lists, milestones with dates and exit criteria, risks with mitigations (leakage, labels, feedback loops, adoption, drift), team and cost, go/no-go kill criteria and open questions. Generates the Markdown from a JSON spec with visible TODOs and lints existing scope docs for missing sections and vague commitments ("leverage AI", "real-time", "state-of-the-art", metrics with no target). Use when someone asks to write, review or tighten an ML project proposal, scope, charter, one-pager or PRD.

AI & Machine Learning / Classical Machine Learningv1.0.0Updated OCT 25 20264 references2 scripts

A scope document is a contract: what decision the model changes, how success is measured against today's process, what is deliberately not being built, and when the team stops if the evidence is not there. Good ML scopes are short, numeric and phased (baseline first), with kill criteria written before the work starts. scripts/scope_doc.py renders the full document from a JSON spec and marks every gap as a TODO; scripts/scope_lint.py reviews an existing document for missing sections and vague promises.

Trigger Conditions

Activate this skill when the user:

  • Asks to write an ML project proposal, scope, charter, one-pager, PRD section or "pitch" for leadership
  • Wants an existing ML proposal reviewed, tightened or made "more concrete"
  • Needs success criteria, milestones, risks or go/no-go criteria for an ML initiative
  • Is turning a framing and feasibility assessment into a plan others can approve

Inputs normally come from ml-problem-framing-advisor (decision, label, unit), ml-feasibility-assessor (data, value, risks) and ml-success-metric-selector (metric, target, guardrails). After launch, the model's documentation belongs in a model card (model-card-writer).

Workflow

text
1. GATHER    -> framing, feasibility verdict, metric choice, stakeholder answers
2. SPEC      -> python scripts/scope_doc.py --template > scope.json; fill it
3. RENDER    -> python scripts/scope_doc.py scope.json -o SCOPE.md   (TODOs listed on stderr)
4. TIGHTEN   -> numbers in success criteria, out-of-scope list, dated milestones with exit criteria
5. LINT      -> python scripts/scope_lint.py SCOPE.md   (also for documents written by others)
6. REVIEW    -> owner, ML lead, data owner, the team that acts on predictions; resolve open questions
7. FREEZE    -> --strict passes (no TODOs); version the doc; changes go through the owner

The Sections and What "Good" Looks Like

SectionGoodWeak
Summary"Rank active subscribers weekly by 30-day cancellation risk so retention calls the top 400.""Use AI to reduce churn."
Problema number: "voluntary churn 2.1%/month, $1.2M/month lost""churn is high"
Decision + userswho acts, how, how often, capacity"the business"
Primary metricat the operating point, on a named split: "recall@400, latest 3-month holdout""accuracy", "AUC" alone
Target + baseline">= 0.45 vs 0.30 for the current rule""as high as possible"
Guardrailscomplaint rate, per-segment floors, PSInone
Online metricA/B or holdout plan with duration"monitor after launch"
Datasources, owners, freshness, available at prediction time; label definition"we have all the data"
Approachphased: baseline -> model -> shadow -> test"train a deep learning model"
Scopein scope AND out of scopeonly in scope
Milestonesdate + exit criterion each"Q1: model"
Riskslikelihood, impact, mitigation"data quality"
Team + costroles, FTE, run costmissing
Go/no-gokill criteria per phasemissing

Fully worked example (churn) and a second one (demand forecasting): scope-examples.md.

Writing Rules

  1. Lead with the decision, not the algorithm. Algorithms belong in Approach, phrased as options.
  2. Numbers everywhere: problem size, target, baseline, capacity, latency, dates, FTE.
  3. Simplest first: the first milestone is always a baseline table; the model must beat the current process, not just a dummy.
  4. Out of scope is mandatory. List the tempting extensions you will not do (real-time, other segments, personalisation, other decisions).
  5. Every milestone has an exit criterion that someone could check without asking you.
  6. Kill criteria up front: "if offline recall@400 is not >= 0.40 by Dec 13, stop and report".
  7. Risks are specific to ML: leakage, label delay, selection bias, feedback loops, train/serve skew, adoption by the acting team, drift, fairness.
  8. One page of body text for the core; detail goes to appendices or linked docs.
  9. Plain words: replace "leverage", "optimise", "insights", "state-of-the-art", "real-time" with what they mean and a number. scope_lint.py flags them.

Milestone Pattern for Most Tabular ML Projects

#MilestoneTypical lengthExit criterion
1Data + label table1-3 weekslabel query reviewed; point-in-time features; leakage scan clean
2Baseline table1 weekcurrent rule, dummy, simple model on a deployment-like split
3Candidate model2-4 weeksbeats baseline at the operating point with CI; error analysis done
4Shadow mode2-4 weeksfeature availability, latency, drift within limits
5Online test4-8 weekspre-registered analysis; go/no-go decision
6Launch + monitoringongoingdashboards, alerts, retraining cadence, model card

Effort split and estimation heuristics: estimation-and-milestones.md.

Risk Register: ML-Specific Entries

RiskTypical mitigation
Target leakage inflates offline resultspoint-in-time features; leakage scan; time-based holdout
Labels arrive late (chargebacks, churn windows)proxy label for monitoring; schedule around delay
Feedback loop (acting changes labels)random untreated holdout; log actions
Selection bias (labels only where reviewed)random exploration sample
Train/serve skewone feature definition for both; shadow mode comparison
Acting team does not adopt the scoresinvolve them in design; explanations; capacity-sized lists
Drift after launchPSI/metric monitoring; retraining cadence; rollback to rules
Fairness / regulatory exposuresliced metrics; legal review; reason codes
Data owner changes a sourcedata contracts; schema checks in the pipeline

Reviewing Someone Else's Scope

Run scope_lint.py, then ask, in this order:

  1. Which decision changes, and who acts? (If unclear, send back to framing.)
  2. What does the current process achieve on the same metric?
  3. Where is the out-of-scope list?
  4. What happens if milestone 3 misses its exit criterion?
  5. Is every feature available at prediction time?
  6. Who owns the model after launch?

Review checklist and comment templates: review-checklist.md.

Output Format

When writing a scope, deliver:

  1. The document in Markdown, sections in the order above, generated with scope_doc.py or by hand.
  2. TODO list - every unresolved item with the owner who must answer it.
  3. Lint result - scope_lint.py output, with errors fixed.
  4. Open questions for the review meeting.

When reviewing, deliver the lint findings, the six questions above answered from the document (or marked "not answered"), and suggested rewrites for the three weakest sentences.

Scripts

bash
python scripts/scope_doc.py --template > scope.json          # complete example spec (churn retention)
python scripts/scope_doc.py scope.json > SCOPE.md
python scripts/scope_doc.py scope.json -o SCOPE.md --strict  # exit 1 while TODOs remain
cat scope.json | python scripts/scope_doc.py -

Renders summary, owner/date/status, problem and decision, users, a success-criteria table (primary metric, target, baseline, guardrails, online metric), data sources with availability at prediction time, label and history, known data issues, phased approach, in-scope/out-of-scope table, milestones with dates and exit criteria, risks with likelihood/impact/mitigation, team and cost, go/no-go criteria and open questions. Missing items appear as TODO(owner): ... in the document and are listed on stderr. Exit codes: 0 rendered, 1 TODOs remain with --strict, 2 usage error.

bash
python scripts/scope_lint.py SCOPE.md
python scripts/scope_lint.py proposal.md --json

SC001-SC003 errors (missing required section, no out-of-scope list, no baseline), SC004-SC009 warnings (no numbers in success criteria, metric without operating point or split, undated milestones, risks without mitigations, vague language outside the Scope section, no go/no-go criteria), SC010 info (TODO/TBD count). Exit codes: 0 no errors, 1 errors, 2 read error.

Additional Resources

FileContent
scope-examples.mdTwo complete scopes (churn retention, store-SKU demand forecasting) and before/after rewrites of weak sentences
estimation-and-milestones.mdWhere ML effort goes, estimation heuristics, milestone and exit-criteria patterns, staffing
review-checklist.mdReviewer checklist, red-flag phrases, comment templates, approval sign-offs
spec-reference.mdEvery field of the JSON spec read by scope_doc.py, with types and examples
More in Classical Machine Learning

anomaly-detection-model-builder

Build and evaluate anomaly detection for tabular, sensor and transaction data with few or no labels: define the anomaly type (point, contextual, collective), choose detectors (robust z-score/MAD and IQR rules, Mahalanobis with robust covariance, k-NN distance, LOF, Isolation Forest, one-class SVM, PCA reconstruction / autoencoders, forecasting residuals for time series), combine them by rank, explain each alert by its deviating features, set thresholds from an alert budget or the score distribution, and evaluate with precision@k, PR-AUC and recall on known incidents, or detector agreement when no labels exist. Includes pure-Python detectors (robust z, Mahalanobis, kNN, Isolation Forest, ensemble) and an evaluation script.

catboost-model-builder

Build CatBoost models that use its strengths correctly: raw categorical columns as cat_features (ordered target statistics, no manual encoding), text_features, Pool construction with NaN-safe categories, ordered boosting and has_time for temporal data, symmetric (oblivious) trees, overfitting detector with eval_set and use_best_model, losses for classification, regression (Quantile, Poisson, Tweedie) and ranking (YetiRank), class weights, monotone constraints, feature importance (PredictionValuesChange, LossFunctionChange) and SHAP values, GPU training and model export. Plans cat/text/numeric columns from a CSV and generates a runnable training script, and checks parameter sets for the conflicts CatBoost rejects (iterations vs n_estimators, Bayesian bootstrap with subsample, multiple class-weight options, depth > 16, use_best_model without eval_set).

categorical-encoding-selector

Choose and implement the right encoding for every categorical column given the model family and the data: one-hot (with infrequent-level grouping and handle_unknown), ordinal with an explicit order, cross-fitted target encoding (scikit-learn TargetEncoder, CatBoost ordered statistics, category_encoders inside CV), frequency/count encoding, feature hashing, native categorical support in LightGBM, HistGradientBoosting, XGBoost and CatBoost, prefix levels for codes (zip3, NAICS-2), and embeddings. Measures cardinality, rare-level share, unseen-level rate on a time-ordered holdout and target signal per column, emits the ColumnTransformer, and quantifies target-encoding leakage (in-sample vs out-of-fold vs holdout AUC).

classification-model-selector

Choose the right classification algorithm for tabular, text or mixed data and prove the choice with a fair bake-off: logistic regression (L2, L1, elastic net), linear SVM, naive Bayes, k-NN, RBF SVM, decision trees, random forest, extra trees, HistGradientBoosting, LightGBM, XGBoost, CatBoost, explainable boosting machines (EBM/GAM), points scorecards and MLPs, matched to data size, width, categorical and high-cardinality share, missingness, class imbalance, text, and constraints such as interpretability, monotonic constraints, calibrated probabilities, latency budgets and tiny deployable models. Ranks candidates with starting hyperparameters and cautions, then generates a bake-off script running every candidate on identical CV folds with per-fold metrics, fit time and prediction latency.

clustering-algorithm-selector

Choose the right clustering algorithm and check that clustering makes sense at all: cluster tendency (Hopkins statistic with a correlated-features caveat, nearest-neighbour isolation), then k-means / MiniBatchKMeans, Gaussian mixtures with BIC, agglomerative (Ward, average, complete, single) with dendrograms, DBSCAN, HDBSCAN, OPTICS, spectral clustering, BIRCH, mean shift, k-prototypes and k-modes for mixed or categorical data, and Gower distance, matched to data size, expected shapes, noise, varying density, known k, soft memberships, hierarchy and streaming. Covers preprocessing (scaling, encoding, dimensionality, distance metrics), validation (internal indices, bootstrap stability, external labels, business profiling) and assigning new points.

collaborative-filtering-implementer

Implement collaborative filtering recommenders that beat popularity: item-based kNN with cosine, Jaccard or asymmetric similarity, shrinkage, neighbourhood size, positive-only or mean-centred rating signals and 'because you liked X' explanations; implicit ALS with confidence weighting, BPR pairwise ranking and the closed-form EASE model in plain NumPy, tuned on a temporal validation split by recall@k and NDCG@k and refit on all data; plus user-kNN, explicit matrix factorisation, the implicit/LightFM libraries, hyperparameter ranges, cold-start fallbacks, scaling with sparse matrices and approximate nearest neighbours. Produces top-k lists in the format of the recommendation evaluator.