A CV score estimates performance under the conditions the folds simulate. If production predicts new
customers, folds must hold out whole customers; if it predicts next month, folds must train on the past and
test on the future, with a gap when labels look ahead. Random K-fold on such data measures something easier
than the real task, so scores look better than production delivers. scripts/cv_advisor.py reads the data
(repeated-entity columns, time columns, target balance) and recommends a splitter with code;
scripts/fold_preview.py builds or audits folds and reports group overlap, time-order and gap violations,
and missing classes.
Trigger Conditions
Activate this skill when the user:
- Asks how to split data, how many folds to use, or which scikit-learn splitter fits
- Sees CV scores much better than holdout or production performance
- Has repeated rows per entity (customer snapshots, patient visits, sessions, images per subject)
- Works with time-ordered data (churn snapshots, transactions, forecasts) or overlapping label windows
- Tunes hyperparameters and needs nested CV
- Wants existing fold assignments audited
Holdout-split audits (duplicates, drift between train and test): train-test-split-auditor; leakage in features and code: target-leakage-detector.
Workflow
1. DEPLOYMENT -> how will predictions be made? new entities / same entities later / next period / i.i.d.
2. ADVISE -> python scripts/cv_advisor.py data.csv --target y [--group-col id --time-col ts --label-window 30]
3. PREVIEW -> python scripts/fold_preview.py data.csv --target y --strategy stratified-group --group-col id
4. HOLDOUT -> keep a final untouched set built the same way (latest period / unseen entities)
5. TUNE -> nested: inner splitter with the same rules; outer scores are the estimate
6. REPORT -> scheme, folds, repeats, mean ± sd, and the holdout confirmationScheme by Deployment
| Production predicts | Splitter | Notes |
|---|---|---|
| i.i.d. rows (no time, no repeated entities) | StratifiedKFold(5, shuffle=True) (classification), KFold (regression) | repeat (RepeatedStratifiedKFold) for small data |
| New entities (unseen customers, patients, devices) | StratifiedGroupKFold / GroupKFold with groups=entity | one entity's rows never on both sides |
| Each entity has one held-out group of its own (sites, hospitals) | LeaveOneGroupOut | estimates transfer to a new site |
| The next period | TimeSeriesSplit(n_splits, gap, test_size) on time-sorted rows | expanding window; max_train_size for sliding |
| Next period with look-ahead labels (churn in 30 d) | time split with gap >= label window (purging) | prevents training labels that overlap test time |
| Same entities, later time | time split (entities may appear in both) | do not group by entity unless new entities matter |
| Spatial generalisation | group by region / spatial blocks | nearby points are correlated |
| Fixed, pre-defined validation set | PredefinedSplit | e.g. a validation month |
cv_advisor.py on the transactions fixture: detects card_id repeats (~10 rows/card) and the timestamp
column, recommends StratifiedGroupKFold for new-card generalisation and warns that forward prediction needs
time folds.
What the Fold Preview Catches
stratified (random) folds on transactions:
PROBLEM: fold 2: 133 card_id value(s) shared between train and test (random folds with repeated entities)
stratified-group folds: group overlap 0 in every fold; minority rate 0.017-0.036 per fold
time folds, label window 2 days, gap 0:
PROBLEM: fold 1: gap 0.0 days < label window 2: training labels overlap the test period
time folds with --gap 120 rows: gap 2.38 d -> soundAudit someone else's folds with --fold-col fold.
Time-Series CV
- Sort by time; train on everything before the test block (expanding) or a fixed window (sliding,
max_train_size) when old data is stale. - Gap / purge: if a row's label covers [t, t + w], drop training rows whose label window overlaps the test start (gap >= w). Embargo: also drop rows just after the test block when training on both sides (finance-style combinatorial CV).
- Test blocks should match the deployment horizon (one month if retrained monthly).
- Weight recent folds more in the decision; report per-fold scores, not only the mean.
Detail (rolling origin for forecasting, purged K-fold, multiple horizons): time-series-cv.md.
How Many Folds and Repeats
| Data | Choice |
|---|---|
| < 1,000 rows | 5-fold x 5-10 repeats (or 10-fold); report the spread |
| 1k-100k | 5-fold (x 2-3 repeats if comparing close models) |
| > 1M | 3-fold or a single large holdout |
| < ~30 minority rows per fold | fewer folds, repeats, stratification, intervals |
| Expensive models | 3-fold for search, 5-fold to confirm finalists |
More folds lower the bias (larger training sets) and raise the cost; repeats lower the variance of the estimate. Leave-one-out is rarely worth it (high variance, expensive) except for tiny datasets with cheap models. Nested CV and variance estimates: nested-cv-and-variance.md.
Special Cases
Multilabel stratification (iterative stratification), hierarchical groups (patients within hospitals), duplicates and near-duplicates, spatial blocking, very small data, distribution-shift-aware validation (adversarial validation), and the final holdout: special-cases.md.
Output Format
When designing CV, deliver:
- Deployment assumption - what the folds simulate (new entities, next period, i.i.d.).
- Splitter - class and parameters (folds, repeats, gap, groups) with code (
cv_advisor.py). - Fold audit - per-fold sizes, class rates, group overlap, time ranges (
fold_preview.py). - Tuning plan - nested CV or validation split inside training folds.
- Holdout - how the final test set is built and when it is used (once).
- Reporting - mean ± sd across folds, per-fold scores for time splits, holdout result.
Scripts
python scripts/cv_advisor.py data.csv --target churned
python scripts/cv_advisor.py snapshots.csv --target churned --group-col customer_id --time-col score_date --label-window 30
python scripts/cv_advisor.py data.csv --target price --deploy iid --tuning --jsonDetects repeated-entity columns (ID-like names with repeats) and date columns, profiles the target, applies
--deploy (forward, new-entities, same-entities, iid; auto by default) and prints the splitter, folds/repeats,
gap, nested-CV advice, warnings (random folds with repeated entities, time columns ignored, missing gap for
look-ahead labels, tiny classes) and scikit-learn code. Exit codes: 0 printed, 2 usage error.
python scripts/fold_preview.py data.csv --target churned --strategy stratified --folds 5
python scripts/fold_preview.py txns.csv --target is_fraud --strategy stratified-group --group-col card_id
python scripts/fold_preview.py snaps.csv --target churned --strategy time --time-col score_date --label-window 30 --gap 200
python scripts/fold_preview.py data.csv --target y --fold-col fold --group-col customer_id --time-col tsGenerates kfold / stratified / group / stratified-group / time folds (scikit-learn-like logic) or audits an
existing fold column; per fold: sizes, minority rate or target mean, group overlap, train max vs test min time and
gap; problems for overlap, time-order violations, gaps shorter than --label-window, missing classes and tiny
folds. Exit codes: 0 sound, 1 problems found, 2 usage error.
Additional Resources
| File | Content |
|---|---|
| time-series-cv.md | Expanding vs sliding windows, gaps, purging and embargo, rolling-origin evaluation, multiple horizons, panel data |
| nested-cv-and-variance.md | Nested CV code, inner/outer rules, number of folds vs bias/variance, repeats, confidence intervals for CV scores |
| special-cases.md | Multilabel and hierarchical stratification, spatial blocking, duplicates, adversarial validation, small data, final holdout policy |
anomaly-detection-model-builder
Build and evaluate anomaly detection for tabular, sensor and transaction data with few or no labels: define the anomaly type (point, contextual, collective), choose detectors (robust z-score/MAD and IQR rules, Mahalanobis with robust covariance, k-NN distance, LOF, Isolation Forest, one-class SVM, PCA reconstruction / autoencoders, forecasting residuals for time series), combine them by rank, explain each alert by its deviating features, set thresholds from an alert budget or the score distribution, and evaluate with precision@k, PR-AUC and recall on known incidents, or detector agreement when no labels exist. Includes pure-Python detectors (robust z, Mahalanobis, kNN, Isolation Forest, ensemble) and an evaluation script.
catboost-model-builder
Build CatBoost models that use its strengths correctly: raw categorical columns as cat_features (ordered target statistics, no manual encoding), text_features, Pool construction with NaN-safe categories, ordered boosting and has_time for temporal data, symmetric (oblivious) trees, overfitting detector with eval_set and use_best_model, losses for classification, regression (Quantile, Poisson, Tweedie) and ranking (YetiRank), class weights, monotone constraints, feature importance (PredictionValuesChange, LossFunctionChange) and SHAP values, GPU training and model export. Plans cat/text/numeric columns from a CSV and generates a runnable training script, and checks parameter sets for the conflicts CatBoost rejects (iterations vs n_estimators, Bayesian bootstrap with subsample, multiple class-weight options, depth > 16, use_best_model without eval_set).
categorical-encoding-selector
Choose and implement the right encoding for every categorical column given the model family and the data: one-hot (with infrequent-level grouping and handle_unknown), ordinal with an explicit order, cross-fitted target encoding (scikit-learn TargetEncoder, CatBoost ordered statistics, category_encoders inside CV), frequency/count encoding, feature hashing, native categorical support in LightGBM, HistGradientBoosting, XGBoost and CatBoost, prefix levels for codes (zip3, NAICS-2), and embeddings. Measures cardinality, rare-level share, unseen-level rate on a time-ordered holdout and target signal per column, emits the ColumnTransformer, and quantifies target-encoding leakage (in-sample vs out-of-fold vs holdout AUC).
classification-model-selector
Choose the right classification algorithm for tabular, text or mixed data and prove the choice with a fair bake-off: logistic regression (L2, L1, elastic net), linear SVM, naive Bayes, k-NN, RBF SVM, decision trees, random forest, extra trees, HistGradientBoosting, LightGBM, XGBoost, CatBoost, explainable boosting machines (EBM/GAM), points scorecards and MLPs, matched to data size, width, categorical and high-cardinality share, missingness, class imbalance, text, and constraints such as interpretability, monotonic constraints, calibrated probabilities, latency budgets and tiny deployable models. Ranks candidates with starting hyperparameters and cautions, then generates a bake-off script running every candidate on identical CV folds with per-fold metrics, fit time and prediction latency.
clustering-algorithm-selector
Choose the right clustering algorithm and check that clustering makes sense at all: cluster tendency (Hopkins statistic with a correlated-features caveat, nearest-neighbour isolation), then k-means / MiniBatchKMeans, Gaussian mixtures with BIC, agglomerative (Ward, average, complete, single) with dendrograms, DBSCAN, HDBSCAN, OPTICS, spectral clustering, BIRCH, mean shift, k-prototypes and k-modes for mixed or categorical data, and Gower distance, matched to data size, expected shapes, noise, varying density, known k, soft memberships, hierarchy and streaming. Covers preprocessing (scaling, encoding, dimensionality, distance metrics), validation (internal indices, bootstrap stability, external labels, business profiling) and assigning new points.
collaborative-filtering-implementer
Implement collaborative filtering recommenders that beat popularity: item-based kNN with cosine, Jaccard or asymmetric similarity, shrinkage, neighbourhood size, positive-only or mean-centred rating signals and 'because you liked X' explanations; implicit ALS with confidence weighting, BPR pairwise ranking and the closed-form EASE model in plain NumPy, tuned on a temporal validation split by recall@k and NDCG@k and refit on all data; plus user-kNN, explicit matrix factorisation, the implicit/LightFM libraries, hyperparameter ranges, cold-start fallbacks, scaling with sparse matrices and approximate nearest neighbours. Produces top-k lists in the format of the recommendation evaluator.