On tabular data the realistic shortlist is short: a regularised logistic regression (fast, explainable,
often surprisingly close), a gradient-boosted tree model (LightGBM / XGBoost / CatBoost /
HistGradientBoosting: usually the most accurate), and, when explanations are mandatory, a glass-box
model (EBM / GAM or a points scorecard). Data shape and constraints move the ranking: small data favours
simpler models, many categoricals favour CatBoost, text favours linear models on TF-IDF, latency and
regulation constrain everything. scripts/select_classifier.py ranks candidates from data traits and
needs; scripts/bakeoff_scaffold.py writes a script that runs the shortlist on identical folds and records
per-fold metrics, fit time and prediction latency.
Trigger Conditions
Activate this skill when the user:
- Asks "which classifier should I use?", "XGBoost or random forest?", "is logistic regression enough?"
- Starts a classification project and needs a shortlist and a fair comparison plan
- Has constraints: regulated decisions, monotonicity, calibrated probabilities, < 5 ms latency, SQL/edge deployment, tiny data, very wide data, many high-cardinality categoricals, text fields
- Wonders whether neural networks / deep learning help on tabular data
Regression targets go to regression-model-trainer; tuning the GBM chosen here to xgboost-lightgbm-tuner or catboost-model-builder; statistical comparison of bake-off results to ml-model-comparison-reporter; rare positives to imbalanced-class-handler.
Workflow
1. TRAITS -> python scripts/select_classifier.py --csv data.csv --target y --drop id [--interpretable ...]
2. SHORTLIST -> top 3-4 + logistic regression baseline (+ the current production model if any)
3. BAKE-OFF -> python scripts/bakeoff_scaffold.py data.csv --target y --models logreg hgb lgbm catboost -o bakeoff.py
4. RUN -> python bakeoff.py -> bakeoff_folds.csv (same folds for every model)
5. COMPARE -> paired tests on per-fold scores (ml-model-comparison-reporter); weigh fit/predict cost
6. DECIDE -> simplest model within noise of the best that meets constraints; then tune itQuick Decision Table
| Situation | First choices | Why |
|---|---|---|
| Typical tabular, 10k-10M rows, mixed types | LightGBM / XGBoost / HistGBM, + logistic baseline | GBMs dominate tabular benchmarks |
| Many or high-cardinality categoricals | CatBoost | ordered target statistics, little preprocessing |
| < ~2,000 rows | logistic regression (regularised), random forest, CatBoost, EBM | variance dominates; simple models win often |
| Very wide (p > n), sparse | L1 / elastic-net logistic, linear SVM | regularised linear models scale with width |
| Text fields | TF-IDF + logistic regression / linear SVM; embeddings + GBM | sparse linear models are strong text baselines |
| Regulated / must explain exactly | EBM, scorecard, monotone GBM with reason codes, logistic | glass-box or constrained models |
| Monotonic business logic (price up -> risk up) | GBM with monotone constraints, EBM, scorecard | constraint enforced by construction |
| Calibrated probabilities consumed downstream | logistic, GBM + calibration check | forests and SVMs need calibration |
| < 1 ms latency or SQL/edge deployment | logistic, shallow tree, small GBM compiled (treelite/ONNX) | model size and depth matter |
| Images, audio, raw text sequences | not classical ML | deep learning |
The bake-off on the 400-row churn fixture illustrates the small-data row: regularised logistic regression (ROC-AUC 0.80, PR-AUC 0.62) beat every default GBM (0.73-0.76) on identical folds.
Algorithm Profiles
| Model | Strengths | Weaknesses | Needs |
|---|---|---|---|
| Logistic regression | fast, stable, interpretable, good probabilities | linear in features; needs interactions/splines | scaling, one-hot, regularisation (C) |
| Linear SVM | wide sparse data, text | no probabilities without calibration | scaling |
| Naive Bayes | tiny data, text, instant training | strong independence assumption; overconfident | counts / TF-IDF |
| k-NN | no training; local patterns | slow inference, curse of dimensionality | scaling, small p |
| RBF SVM | small dense data | O(n²) training, tuning C/gamma | scaling |
| Decision tree | readable rules | unstable, weak alone | depth / leaf limits |
| Random forest | robust, little tuning | big, poorly calibrated, slower inference | few params |
| Extra trees | fast, noise-tolerant | same as RF | few params |
| HistGradientBoosting | strong, native NaN/categoricals, in scikit-learn | < 256 categories per feature | early stopping |
| LightGBM | fastest at scale, native categoricals | overfits small data with defaults | min_child_samples, early stopping |
| XGBoost | robust, rich regularisation, GPU | categorical support needs category dtype | early stopping |
| CatBoost | best with categoricals, good defaults | slower training | cat_features |
| EBM (InterpretML) | glass-box GAM with interactions, near-GBM accuracy | slower training, large data takes time | little tuning |
| MLP / tabular DL | very large data, embeddings, multimodal | rarely beats GBMs on tabular; tuning-heavy | scaling, early stopping |
Detail per algorithm (key hyperparameters, defaults that matter, failure modes): algorithm-profiles.md.
Running a Fair Bake-Off
Rules the generated script enforces:
- Identical folds for every model (stratified, grouped by entity, or time-ordered).
- Each model gets the preprocessing its family needs, all fitted inside the fold.
- Same metrics (ROC-AUC, PR-AUC, log loss, Brier) and cost columns (fit seconds, prediction ms per 1k rows).
- Untuned defaults first: a heavily tuned favourite against untuned rivals is not a comparison; tune the top 2 afterwards with the same budget.
- Per-fold results saved (
bakeoff_folds.csv) for paired statistical comparison: fold-to-fold variance is often larger than the gap between models.
Design details (time-based bake-offs, nested tuning, budgets, reporting): bakeoff-design.md.
Choosing Among Close Models
When two models are within noise on the primary metric, prefer, in order:
- the one that satisfies the hard constraints (explainability, monotonicity, latency, deployment target);
- the simpler / cheaper one (fewer features, faster, smaller);
- the better-calibrated one if probabilities are consumed;
- the more stable one (lower fold-to-fold std, stable over time periods).
Interpretability and constraint options in depth: interpretable-and-constrained-models.md.
Output Format
When recommending a classifier, answer with:
- Data traits - rows, features, categorical/high-cardinality share, missingness, imbalance, text.
- Constraints - explainability, monotonicity, calibration, latency, deployment.
- Ranked shortlist - 3-4 models with why, starting configuration and cautions (
select_classifier.py). - Bake-off plan - folds, metrics, the generated script, budget for tuning the top two.
- Decision rule - "simplest model within one standard error of the best that meets constraints".
Scripts
python scripts/select_classifier.py --csv data.csv --target churned --drop customer_id
python scripts/select_classifier.py --rows 2000000 --features 300 --cat-share 0.4 --hc-share 0.1 --minority 0.002 --latency-ms 5
python scripts/select_classifier.py --csv data.csv --target y --interpretable --monotone --calibrated --jsonDerives rows, features, categorical and high-cardinality share, missing share, minority share and text columns
from a CSV (or takes them as flags), applies needs (--interpretable, --monotone, --calibrated,
--latency-ms, --tiny-model) and ranks up to 16 algorithms with a reason, a starting configuration and
cautions. Exit codes: 0 printed, 2 usage or read error.
python scripts/bakeoff_scaffold.py data.csv --target churned --models logreg hgb rf lgbm xgb catboost ebm -o bakeoff.py
python scripts/bakeoff_scaffold.py data.csv --target churned --group-col customer_id
python scripts/bakeoff_scaffold.py data.csv --target churned --time-col signup_date --folds 5Writes a script that runs each chosen model (logreg, hgb, rf, et, lgbm, xgb, catboost, ebm; optional libraries
skipped if missing) on identical StratifiedKFold / StratifiedGroupKFold / TimeSeriesSplit folds with
family-appropriate preprocessing, and saves per-fold ROC-AUC, PR-AUC, log loss, Brier, fit seconds and
prediction ms per 1k rows to bakeoff_folds.csv. Verified with scikit-learn 1.9, LightGBM 4.7, XGBoost 3.4,
CatBoost 1.2 and interpret 0.7. Exit codes: 0 written, 2 usage error.
Additional Resources
| File | Content |
|---|---|
| algorithm-profiles.md | Each classifier: how it works in two lines, key hyperparameters, good defaults, data needs, failure modes |
| bakeoff-design.md | Fair comparison protocol, folds, tuning budgets, time-based bake-offs, cost metrics, reporting template |
| interpretable-and-constrained-models.md | EBM/GAMs, scorecards, monotone constraints in LightGBM/XGBoost/HistGBM/CatBoost, reason codes, regulation |
| tabular-deep-learning.md | When neural nets help on tabular data, what to try (MLP, FT-Transformer, TabPFN-style models), and why GBMs usually win |
anomaly-detection-model-builder
Build and evaluate anomaly detection for tabular, sensor and transaction data with few or no labels: define the anomaly type (point, contextual, collective), choose detectors (robust z-score/MAD and IQR rules, Mahalanobis with robust covariance, k-NN distance, LOF, Isolation Forest, one-class SVM, PCA reconstruction / autoencoders, forecasting residuals for time series), combine them by rank, explain each alert by its deviating features, set thresholds from an alert budget or the score distribution, and evaluate with precision@k, PR-AUC and recall on known incidents, or detector agreement when no labels exist. Includes pure-Python detectors (robust z, Mahalanobis, kNN, Isolation Forest, ensemble) and an evaluation script.
catboost-model-builder
Build CatBoost models that use its strengths correctly: raw categorical columns as cat_features (ordered target statistics, no manual encoding), text_features, Pool construction with NaN-safe categories, ordered boosting and has_time for temporal data, symmetric (oblivious) trees, overfitting detector with eval_set and use_best_model, losses for classification, regression (Quantile, Poisson, Tweedie) and ranking (YetiRank), class weights, monotone constraints, feature importance (PredictionValuesChange, LossFunctionChange) and SHAP values, GPU training and model export. Plans cat/text/numeric columns from a CSV and generates a runnable training script, and checks parameter sets for the conflicts CatBoost rejects (iterations vs n_estimators, Bayesian bootstrap with subsample, multiple class-weight options, depth > 16, use_best_model without eval_set).
categorical-encoding-selector
Choose and implement the right encoding for every categorical column given the model family and the data: one-hot (with infrequent-level grouping and handle_unknown), ordinal with an explicit order, cross-fitted target encoding (scikit-learn TargetEncoder, CatBoost ordered statistics, category_encoders inside CV), frequency/count encoding, feature hashing, native categorical support in LightGBM, HistGradientBoosting, XGBoost and CatBoost, prefix levels for codes (zip3, NAICS-2), and embeddings. Measures cardinality, rare-level share, unseen-level rate on a time-ordered holdout and target signal per column, emits the ColumnTransformer, and quantifies target-encoding leakage (in-sample vs out-of-fold vs holdout AUC).
clustering-algorithm-selector
Choose the right clustering algorithm and check that clustering makes sense at all: cluster tendency (Hopkins statistic with a correlated-features caveat, nearest-neighbour isolation), then k-means / MiniBatchKMeans, Gaussian mixtures with BIC, agglomerative (Ward, average, complete, single) with dendrograms, DBSCAN, HDBSCAN, OPTICS, spectral clustering, BIRCH, mean shift, k-prototypes and k-modes for mixed or categorical data, and Gower distance, matched to data size, expected shapes, noise, varying density, known k, soft memberships, hierarchy and streaming. Covers preprocessing (scaling, encoding, dimensionality, distance metrics), validation (internal indices, bootstrap stability, external labels, business profiling) and assigning new points.
collaborative-filtering-implementer
Implement collaborative filtering recommenders that beat popularity: item-based kNN with cosine, Jaccard or asymmetric similarity, shrinkage, neighbourhood size, positive-only or mean-centred rating signals and 'because you liked X' explanations; implicit ALS with confidence weighting, BPR pairwise ranking and the closed-form EASE model in plain NumPy, tuned on a temporal validation split by recall@k and NDCG@k and refit on all data; plus user-kNN, explicit matrix factorisation, the implicit/LightFM libraries, hyperparameter ranges, cold-start fallbacks, scaling with sparse matrices and approximate nearest neighbours. Produces top-k lists in the format of the recommendation evaluator.
counterfactual-explanation-generator
Generate counterfactual explanations and algorithmic recourse: the smallest realistic change to a row's features that changes the model's decision. Solves exact minimal-cost recourse for linear and logistic models (MAD-scaled L1 cost, immutable features, increase/decrease directions, bounds, integer and binary features, diverse alternatives, features every counterfactual must change, and a no-recourse finding when immutable features decide), generates DiCE counterfactuals for any scikit-learn model with constraints turned into permitted ranges and every result re-predicted, and validates counterfactuals from any method for constraint violations, implausible values, off-manifold rows, broken derived-feature relations and failure to flip the prediction.