"Model B scores 0.861 vs 0.858" is not a result; it is a pair of noisy estimates. A comparison needs
pairing (same folds or same test rows), a test that respects how the scores were produced (CV folds
share training data, so ordinary t-tests are overconfident), a practical threshold (is +0.003 AUC worth
anything?), and the costs (latency, complexity) next to the gains. scripts/compare_models.py analyses
per-fold CV scores for any number of models; scripts/paired_predictions.py compares two models on one
test set row by row. Both are standard-library Python and match scipy / reference implementations.
Trigger Conditions
Activate this skill when the user:
- Asks "which model is better?", "is this improvement significant?", "can we replace the champion?"
- Has CV results from several models (a bake-off, a tuning run, feature-set variants) and wants a verdict
- Has two models' predictions on the same holdout and wants a test (AUC difference, accuracy difference)
- Needs a comparison table, slide or written report for stakeholders
- Compares many models/datasets and needs multiple-comparison control
Producing per-fold results: classification-model-selector (bake-off script); baselines to include: ml-baseline-model-builder; metric choice: ml-success-metric-selector.
Workflow
1. ALIGN -> identical folds (or identical test rows) for every model; record per-fold scores
2. SUMMARISE -> python scripts/compare_models.py folds.csv --metric average_precision --rope 0.005
3. HOLDOUT -> python scripts/paired_predictions.py holdout_preds.csv --a challenger --b champion --k 400
4. PRACTICAL -> define the smallest difference that matters (ROPE) before looking at results
5. COSTS -> fit time, latency, size, maintenance next to the metric deltas
6. DECIDE -> simplest model not distinguishable from the best, unless the gain exceeds the ROPE
7. REPORT -> table + tests + decision + caveats (references/report-template.md)Which Test for Which Evidence
| Evidence | Test | Why |
|---|---|---|
| k-fold CV scores, 2 models | corrected resampled t-test | fold scores are correlated (shared training data); the correction inflates the variance by (1/k + n_test/n_train) |
| same, non-parametric | Wilcoxon signed-rank on fold differences | no normality assumption; low power with 5 folds (minimum p = 0.0625) |
| same, decision-oriented | Bayesian correlated t-test with ROPE | P(A better), P(equivalent), P(B better) instead of a p-value |
| CV scores, 3+ models | Friedman test on ranks, then pairwise corrected tests vs the best | omnibus first, then post-hoc |
| One test set, ROC-AUC | DeLong test | exact variance of correlated AUCs on the same rows |
| One test set, decisions at a threshold / top-K | McNemar on discordant pairs | only rows where the models disagree carry information |
| One test set, any metric (PR-AUC, log loss, MAE) | paired (stratified) bootstrap CI of the difference | resample rows once, score both models on them |
| Online A/B | two-sample test on the business metric | randomised traffic, pre-registered |
Formulas, assumptions and when each fails: statistical-tests.md.
Reading compare_models.py
model mean std rank
lightgbm 0.8578 0.0110 1.20
random_forest 0.8536 0.0090 1.80
logreg 0.8364 0.0081 3.00
model diff wins p corr-t p naive-t p Wilcoxon P(best>) P(equiv)
random_forest +0.0042 8/10 0.154 0.050 0.064 0.39 0.61
logreg +0.0214 10/10 0.000 0.000 0.002 1.00 0.00- The naive paired t-test says LightGBM beats random forest (p = 0.050); the corrected test does not (p = 0.154). The correction is the honest one for CV.
- With ROPE ±0.005 the Bayesian test puts 61% on "practically equivalent": prefer the simpler/cheaper model.
- LightGBM beats logistic regression in every fold, by 0.021: real and practically large.
Reading paired_predictions.py
DeLong: AUC A 0.9788 B 0.9130 diff +0.0657 (95% CI +0.0355 .. +0.0960), z 4.26, p 0.0000
McNemar (top-50 of each model): A right/B wrong 32, A wrong/B right 8, p 0.0002 (exact binomial)
pr_auc +0.2057 95% CI [+0.1114, +0.3091]Use the test that matches the decision: DeLong for ranking quality, McNemar at the operating point (top-K or threshold), bootstrap CIs for everything else.
Practical Significance and Multiplicity
- Fix the ROPE before looking: e.g. "±0.005 PR-AUC is noise for this business", or derive it from the value model (the difference that changes expected profit by less than the cost of switching).
- Many comparisons (10 models, 5 feature sets): control the family-wise error (Holm) or false discovery rate (Benjamini-Hochberg), or compare everything against one pre-declared baseline.
- Test-set reuse: every look at the test set to choose a model makes its score optimistic; choose on CV, confirm once on the holdout.
- Statistical ≠ practical: with 1M test rows, +0.0005 AUC is "significant" and worthless.
Details, Holm/BH code and the decision rule: decision-rules-and-pitfalls.md.
Output Format
A comparison report contains:
- Setup - data, split/folds, metric(s), ROPE, models and their tuning budgets (equal?).
- Table - mean ± sd (or holdout value with CI) per model, mean rank, cost columns (
--markdownoutput). - Tests - corrected t / Wilcoxon / Bayesian vs the best, Friedman for 3+ models; DeLong / McNemar / bootstrap on the holdout.
- Decision - the chosen model and the rule that chose it (best, or simplest within ROPE).
- Caveats - fold count, few positives, test-set reuse, distribution shift, online validation still needed.
Template and an example: report-template.md.
Scripts
python scripts/compare_models.py bakeoff_folds.csv --metric average_precision
python scripts/compare_models.py cv_scores.csv --metric roc_auc --rope 0.005 --test-train-ratio 0.25
python scripts/compare_models.py folds.csv --metric log_loss --markdown
python scripts/compare_models.py folds.csv --metric rmse --lower-better --jsonLong CSV (model, fold, metrics). Mean, std, mean rank; for each model vs the best: mean paired difference, wins,
corrected resampled t-test (Nadeau-Bengio; --test-train-ratio defaults to 1/(k-1)), naive paired t-test for
contrast, exact Wilcoxon signed-rank (normal approximation above 20 folds), Bayesian correlated t-test
probabilities with --rope; Friedman test for >= 3 models. Lower-is-better metrics are detected from the name.
Exit codes: 0 printed, 2 usage error or fewer than 2 models / 3 shared folds.
python scripts/paired_predictions.py holdout.csv --a challenger --b champion
python scripts/paired_predictions.py holdout.csv --a lgbm --b logreg --k 400 --n-boot 2000
python scripts/paired_predictions.py reg_holdout.csv --regression --a new --b oldBinary: DeLong AUC difference with SE, CI and p; McNemar at --threshold or each model's top---k (exact below
50 discordant pairs); stratified paired bootstrap CIs for ROC-AUC, PR-AUC, log loss and Brier differences.
Regression: bootstrap CIs for MAE/RMSE differences and a sign test. Exit codes: 0 printed, 2 usage or data error.
Additional Resources
| File | Content |
|---|---|
| statistical-tests.md | Corrected resampled t-test, 5x2cv, Wilcoxon, Friedman/Nemenyi, Bayesian correlated t-test, DeLong, McNemar, bootstrap: formulas, assumptions, scipy/statsmodels code |
| decision-rules-and-pitfalls.md | ROPE choice, one-SE and simplest-within-noise rules, Holm/BH corrections, test reuse, unequal tuning, champion/challenger |
| report-template.md | Report structure, markdown table, wording for significant / equivalent / inconclusive results, worked example |
anomaly-detection-model-builder
Build and evaluate anomaly detection for tabular, sensor and transaction data with few or no labels: define the anomaly type (point, contextual, collective), choose detectors (robust z-score/MAD and IQR rules, Mahalanobis with robust covariance, k-NN distance, LOF, Isolation Forest, one-class SVM, PCA reconstruction / autoencoders, forecasting residuals for time series), combine them by rank, explain each alert by its deviating features, set thresholds from an alert budget or the score distribution, and evaluate with precision@k, PR-AUC and recall on known incidents, or detector agreement when no labels exist. Includes pure-Python detectors (robust z, Mahalanobis, kNN, Isolation Forest, ensemble) and an evaluation script.
catboost-model-builder
Build CatBoost models that use its strengths correctly: raw categorical columns as cat_features (ordered target statistics, no manual encoding), text_features, Pool construction with NaN-safe categories, ordered boosting and has_time for temporal data, symmetric (oblivious) trees, overfitting detector with eval_set and use_best_model, losses for classification, regression (Quantile, Poisson, Tweedie) and ranking (YetiRank), class weights, monotone constraints, feature importance (PredictionValuesChange, LossFunctionChange) and SHAP values, GPU training and model export. Plans cat/text/numeric columns from a CSV and generates a runnable training script, and checks parameter sets for the conflicts CatBoost rejects (iterations vs n_estimators, Bayesian bootstrap with subsample, multiple class-weight options, depth > 16, use_best_model without eval_set).
categorical-encoding-selector
Choose and implement the right encoding for every categorical column given the model family and the data: one-hot (with infrequent-level grouping and handle_unknown), ordinal with an explicit order, cross-fitted target encoding (scikit-learn TargetEncoder, CatBoost ordered statistics, category_encoders inside CV), frequency/count encoding, feature hashing, native categorical support in LightGBM, HistGradientBoosting, XGBoost and CatBoost, prefix levels for codes (zip3, NAICS-2), and embeddings. Measures cardinality, rare-level share, unseen-level rate on a time-ordered holdout and target signal per column, emits the ColumnTransformer, and quantifies target-encoding leakage (in-sample vs out-of-fold vs holdout AUC).
classification-model-selector
Choose the right classification algorithm for tabular, text or mixed data and prove the choice with a fair bake-off: logistic regression (L2, L1, elastic net), linear SVM, naive Bayes, k-NN, RBF SVM, decision trees, random forest, extra trees, HistGradientBoosting, LightGBM, XGBoost, CatBoost, explainable boosting machines (EBM/GAM), points scorecards and MLPs, matched to data size, width, categorical and high-cardinality share, missingness, class imbalance, text, and constraints such as interpretability, monotonic constraints, calibrated probabilities, latency budgets and tiny deployable models. Ranks candidates with starting hyperparameters and cautions, then generates a bake-off script running every candidate on identical CV folds with per-fold metrics, fit time and prediction latency.
clustering-algorithm-selector
Choose the right clustering algorithm and check that clustering makes sense at all: cluster tendency (Hopkins statistic with a correlated-features caveat, nearest-neighbour isolation), then k-means / MiniBatchKMeans, Gaussian mixtures with BIC, agglomerative (Ward, average, complete, single) with dendrograms, DBSCAN, HDBSCAN, OPTICS, spectral clustering, BIRCH, mean shift, k-prototypes and k-modes for mixed or categorical data, and Gower distance, matched to data size, expected shapes, noise, varying density, known k, soft memberships, hierarchy and streaming. Covers preprocessing (scaling, encoding, dimensionality, distance metrics), validation (internal indices, bootstrap stability, external labels, business profiling) and assigning new points.
collaborative-filtering-implementer
Implement collaborative filtering recommenders that beat popularity: item-based kNN with cosine, Jaccard or asymmetric similarity, shrinkage, neighbourhood size, positive-only or mean-centred rating signals and 'because you liked X' explanations; implicit ALS with confidence weighting, BPR pairwise ranking and the closed-form EASE model in plain NumPy, tuned on a temporal validation split by recall@k and NDCG@k and refit on all data; plus user-kNN, explicit matrix factorisation, the implicit/LightFM libraries, hyperparameter ranges, cold-start fallbacks, scaling with sparse matrices and approximate nearest neighbours. Produces top-k lists in the format of the recommendation evaluator.