For each task: what the label looks like, the loss to train with, the metric to report, a sensible first model, and the traps that make teams pick the wrong framing.
Binary classification
- Label: 0/1 event in a window (
defaulted_12m,churned_30d,clicked). - Loss: log loss (cross-entropy). Class weights only if you will not use calibrated probabilities.
- Metrics: ROC-AUC (ranking quality, insensitive to prevalence), PR-AUC / average precision (rare positives), recall or precision at capacity (top-K), expected cost at a threshold, Brier and calibration when the probability itself drives a decision (pricing, expected loss).
- First model: logistic regression with one-hot categoricals and scaled numerics; then gradient boosting.
- Traps: accuracy on imbalanced data; threshold 0.5 by default; probabilities distorted by resampling.
Multiclass classification
- Label: one of K mutually exclusive classes.
- Loss: softmax cross-entropy.
- Metrics: macro-F1 (every class counts), weighted-F1 / accuracy (frequent classes dominate), top-k accuracy (routing with a human picking from suggestions), confusion matrix for costly swaps.
- Traps: classes that are not mutually exclusive (use multilabel: one binary per label); long tail of rare classes (merge into "other" or a hierarchy).
Multilabel classification
- Label: any subset of labels (tags, symptoms).
- Loss: one binary cross-entropy per label.
- Metrics: micro/macro F1, Hamming loss, per-label PR-AUC.
Ordinal
- Label: ordered classes (1-5 stars, severity grades).
- Options: regression on the scale then round (often good enough), ordinal logistic (cumulative link), or K-1 binary "greater than k" models.
- Metrics: MAE on the scale, quadratic weighted kappa, Spearman.
- Traps: treating as nominal multiclass throws away the order.
Regression
- Label: continuous quantity in a window (
spend_next_90d,delivery_minutes). - Loss choice:
Target shape Loss Symmetric, few outliers squared error Outliers you do not want to chase Huber or absolute error Strictly positive, right-skewed log1p target + squared error, or Gamma Many exact zeros plus a positive tail Tweedie (power 1.1-1.9) or hurdle model Need an interval / a percentile quantile (pinball) loss - Metrics: MAE, RMSE, WAPE/MAPE (business), R² (comparison only), calibration by decile.
- Traps: log-target models predict the geometric mean (biased low) unless corrected; MAPE explodes near 0.
Count regression
- Label: non-negative integers (claims, visits, defects).
- Loss: Poisson; negative binomial when variance >> mean; Tweedie for zero-heavy counts with exposure.
- Exposure: model rate per unit exposure (
log(exposure)offset in GLMs,sample_weight/base margin in GBMs). - Metrics: Poisson deviance, MAE.
Forecasting
- Label: future values of a series at horizons h = 1..H.
- Framings: local statistical models per series (ETS, ARIMA), global ML models on lag features across many series (LightGBM), hierarchical reconciliation.
- Metrics: MASE (scale-free, vs seasonal naive), WAPE, sMAPE, pinball loss for quantiles, bias.
- Traps: random CV (must be rolling-origin); features known only in hindsight (actual weather); censored demand during stock-outs.
Ranking / learning to rank
- Label: graded relevance per (query, item), or implicit feedback (clicks, purchases).
- Loss: pointwise (regression/classification), pairwise (RankNet), listwise (LambdaMART on NDCG).
- Metrics: NDCG@k, MAP, MRR, recall@k.
- Traps: position bias in clicks; evaluating per row instead of per query.
Time-to-event (survival)
- Label:
(duration, event_observed); censored rows still carry information. - Models: Kaplan-Meier (descriptive), Cox PH, accelerated failure time, random survival forest, GBM with Cox or AFT objective.
- Metrics: concordance index, time-dependent AUC, integrated Brier score.
- Use when: windows cannot be fully observed, or "when" matters more than "whether".
Uplift / treatment effect
- Label: outcome plus treatment indicator, ideally from a randomised experiment.
- Models: two-model (T-learner), S-learner, X-learner, class transformation, causal forests.
- Metrics: Qini coefficient, AUUC, uplift at top-k.
- Use when: the decision is whether to intervene and the intervention changes the outcome.
Anomaly detection
- Label: none, or very few confirmed anomalies.
- Models: robust statistics (median/MAD), Isolation Forest, LOF, one-class SVM, autoencoders, forecasting residuals for time series.
- Metrics: precision@k on a reviewed sample, alert volume, time to detect.
- Reframe as classification once you have a few hundred labelled positives.
Clustering
- Label: none. An exploratory tool, not a decision.
- Make it actionable: a segment must map to a different action; validate stability and business meaning.
Recommendation
- Label: user-item interactions (explicit ratings or implicit events).
- Models: popularity, item-kNN, matrix factorisation (ALS, BPR), two-tower retrieval plus a ranker.
- Metrics: recall@k, NDCG@k, coverage, novelty; online CTR/conversion.
Choosing between neighbours
| Choice | Prefer the first when | Prefer the second when |
|---|---|---|
| Classification vs regression | the decision is a yes/no or a threshold on a quantity | the size of the outcome changes the action |
| Classification vs survival | the window is fully observed for all training rows | many rows are censored or timing matters |
| Classification vs ranking | each case is judged independently | only the order within a query/list matters |
| Classification vs uplift | the action does not change the outcome | the action is the intervention you are deciding on |
| Regression vs forecasting | rows are independent units | the same series continues over time |
| Anomaly vs classification | < ~100 confirmed positives | enough labelled positives to learn from |
| Clustering vs classification | no outcome to predict, exploring structure | a label exists (use it) |
anomaly-detection-model-builder
Build and evaluate anomaly detection for tabular, sensor and transaction data with few or no labels: define the anomaly type (point, contextual, collective), choose detectors (robust z-score/MAD and IQR rules, Mahalanobis with robust covariance, k-NN distance, LOF, Isolation Forest, one-class SVM, PCA reconstruction / autoencoders, forecasting residuals for time series), combine them by rank, explain each alert by its deviating features, set thresholds from an alert budget or the score distribution, and evaluate with precision@k, PR-AUC and recall on known incidents, or detector agreement when no labels exist. Includes pure-Python detectors (robust z, Mahalanobis, kNN, Isolation Forest, ensemble) and an evaluation script.
catboost-model-builder
Build CatBoost models that use its strengths correctly: raw categorical columns as cat_features (ordered target statistics, no manual encoding), text_features, Pool construction with NaN-safe categories, ordered boosting and has_time for temporal data, symmetric (oblivious) trees, overfitting detector with eval_set and use_best_model, losses for classification, regression (Quantile, Poisson, Tweedie) and ranking (YetiRank), class weights, monotone constraints, feature importance (PredictionValuesChange, LossFunctionChange) and SHAP values, GPU training and model export. Plans cat/text/numeric columns from a CSV and generates a runnable training script, and checks parameter sets for the conflicts CatBoost rejects (iterations vs n_estimators, Bayesian bootstrap with subsample, multiple class-weight options, depth > 16, use_best_model without eval_set).
categorical-encoding-selector
Choose and implement the right encoding for every categorical column given the model family and the data: one-hot (with infrequent-level grouping and handle_unknown), ordinal with an explicit order, cross-fitted target encoding (scikit-learn TargetEncoder, CatBoost ordered statistics, category_encoders inside CV), frequency/count encoding, feature hashing, native categorical support in LightGBM, HistGradientBoosting, XGBoost and CatBoost, prefix levels for codes (zip3, NAICS-2), and embeddings. Measures cardinality, rare-level share, unseen-level rate on a time-ordered holdout and target signal per column, emits the ColumnTransformer, and quantifies target-encoding leakage (in-sample vs out-of-fold vs holdout AUC).
classification-model-selector
Choose the right classification algorithm for tabular, text or mixed data and prove the choice with a fair bake-off: logistic regression (L2, L1, elastic net), linear SVM, naive Bayes, k-NN, RBF SVM, decision trees, random forest, extra trees, HistGradientBoosting, LightGBM, XGBoost, CatBoost, explainable boosting machines (EBM/GAM), points scorecards and MLPs, matched to data size, width, categorical and high-cardinality share, missingness, class imbalance, text, and constraints such as interpretability, monotonic constraints, calibrated probabilities, latency budgets and tiny deployable models. Ranks candidates with starting hyperparameters and cautions, then generates a bake-off script running every candidate on identical CV folds with per-fold metrics, fit time and prediction latency.
clustering-algorithm-selector
Choose the right clustering algorithm and check that clustering makes sense at all: cluster tendency (Hopkins statistic with a correlated-features caveat, nearest-neighbour isolation), then k-means / MiniBatchKMeans, Gaussian mixtures with BIC, agglomerative (Ward, average, complete, single) with dendrograms, DBSCAN, HDBSCAN, OPTICS, spectral clustering, BIRCH, mean shift, k-prototypes and k-modes for mixed or categorical data, and Gower distance, matched to data size, expected shapes, noise, varying density, known k, soft memberships, hierarchy and streaming. Covers preprocessing (scaling, encoding, dimensionality, distance metrics), validation (internal indices, bootstrap stability, external labels, business profiling) and assigning new points.
collaborative-filtering-implementer
Implement collaborative filtering recommenders that beat popularity: item-based kNN with cosine, Jaccard or asymmetric similarity, shrinkage, neighbourhood size, positive-only or mean-centred rating signals and 'because you liked X' explanations; implicit ALS with confidence weighting, BPR pairwise ranking and the closed-form EASE model in plain NumPy, tuned on a temporal validation split by recall@k and NDCG@k and refit on all data; plus user-kNN, explicit matrix factorisation, the implicit/LightFM libraries, hyperparameter ranges, cold-start fallbacks, scaling with sparse matrices and approximate nearest neighbours. Produces top-k lists in the format of the recommendation evaluator.