Each pitfall: how it shows up, how to detect it early, and what to change.
1. Unactionable predictions
Shows up as: a model with strong offline metrics that no one consumes, or that is consumed and the KPI does not move. Detect: ask "what will you do differently on Monday with this score?" If the answer is a dashboard, the framing is analytics. Fix: start from the decision and capacity; predict the input to that decision.
2. Horizon shorter than the reaction time
Shows up as: high recall offline, but flagged customers have already left when the call happens. Detect: compare horizon to the median time between scoring and action. Fix: add a gap window, lengthen the horizon, or act faster.
3. Leakage by framing
Shows up as: AUC 0.95+ offline, collapse in production. Detect: single-feature AUC near 1.0, features with names like status/reason/result, features computed from tables overwritten after the event, a model that is much better on old rows than recent ones. Fix: rebuild every feature as-of the prediction time; see target-leakage-detector.
4. Training population differs from the scoring population
Shows up as: offline metrics do not reproduce on live traffic. Examples: trained on all customers but scored only on new ones; trained on customers who were already contacted; trained after a dedupe step that production does not run. Fix: generate training rows with the exact eligibility logic used at scoring time.
5. Feedback loops
Shows up as: a model that keeps getting "worse" each retraining; outcomes shaped by past model actions. Mechanism: you call the high-risk customers, some are saved, so next time they look like non-churners and the model learns that high-risk signals mean "stays". Fix: keep a random untreated holdout; log score, action and outcome per row; train on holdout labels or correct with inverse propensity weights; reframe as uplift when the action is the decision.
6. Selection bias in labels
Shows up as: labels exist only for cases someone chose to look at. Examples: credit outcomes only for approved loans; fraud labels only for reviewed transactions; ticket resolution labels only for escalated tickets. Fix: randomised exploration (approve a small random sample), reject inference, or restrict the claim to the labelled population and say so.
7. Proxy labels that drift from the goal
Shows up as: the proxy metric improves while the business metric does not. Fix: document the gap, pair the proxy metric with a guardrail on the real goal, revisit the proxy after launch.
8. Goodhart's law
Shows up as: when a metric becomes a target people game it: tickets closed early to hit resolution time, leads marked qualified to hit quotas, and labels shift with them. Fix: use labels produced by processes not graded on the model; audit samples; multiple metrics.
9. One model for many decisions
Shows up as: a single "risk score" used for calls, emails, discounts and credit limits, each wanting a different horizon and population. Fix: one framing per decision; shared features are fine, shared labels usually are not.
10. Ignoring the cost asymmetry
Shows up as: threshold 0.5, F1 optimised, and an operating point nobody chose. Fix: write the cost of a false positive and a false negative (or the capacity) into the framing; pick the threshold from it (see ml-success-metric-selector and imbalanced-class-handler).
11. Fairness and regulation found late
Shows up as: a launch blocked by legal, or harm to a group discovered after deployment. Fix: list protected attributes and regulated decisions (credit, hiring, housing, insurance) at framing time; plan sliced metrics, explanation requirements and adverse-action reasons.
12. Predicting what is already known
Shows up as: a model that "predicts" renewals of customers who already signed, or demand for orders already placed. Fix: remove the already-decided population; the model should only score genuine uncertainty.
13. Static framing of a non-stationary world
Shows up as: performance decays month by month. Fix: decide the retraining cadence and drift monitors as part of the framing; prefer features that are stable in meaning (ratios, recency) over raw counts tied to a growing product.
Quick detection table
| Check | How | Red flag |
|---|---|---|
| Single-feature predictive power | AUC/R² of each feature alone | > 0.9 AUC from one column |
| Time stability | metric by month of prediction date | steady decline, or much better on old data |
| Population match | compare training eligibility SQL with scoring SQL | any difference |
| Action logging | are actions per row stored? | no -> feedback loop cannot be corrected |
| Label provenance | who/what writes the label, when | written by a process that sees model output |
anomaly-detection-model-builder
Build and evaluate anomaly detection for tabular, sensor and transaction data with few or no labels: define the anomaly type (point, contextual, collective), choose detectors (robust z-score/MAD and IQR rules, Mahalanobis with robust covariance, k-NN distance, LOF, Isolation Forest, one-class SVM, PCA reconstruction / autoencoders, forecasting residuals for time series), combine them by rank, explain each alert by its deviating features, set thresholds from an alert budget or the score distribution, and evaluate with precision@k, PR-AUC and recall on known incidents, or detector agreement when no labels exist. Includes pure-Python detectors (robust z, Mahalanobis, kNN, Isolation Forest, ensemble) and an evaluation script.
catboost-model-builder
Build CatBoost models that use its strengths correctly: raw categorical columns as cat_features (ordered target statistics, no manual encoding), text_features, Pool construction with NaN-safe categories, ordered boosting and has_time for temporal data, symmetric (oblivious) trees, overfitting detector with eval_set and use_best_model, losses for classification, regression (Quantile, Poisson, Tweedie) and ranking (YetiRank), class weights, monotone constraints, feature importance (PredictionValuesChange, LossFunctionChange) and SHAP values, GPU training and model export. Plans cat/text/numeric columns from a CSV and generates a runnable training script, and checks parameter sets for the conflicts CatBoost rejects (iterations vs n_estimators, Bayesian bootstrap with subsample, multiple class-weight options, depth > 16, use_best_model without eval_set).
categorical-encoding-selector
Choose and implement the right encoding for every categorical column given the model family and the data: one-hot (with infrequent-level grouping and handle_unknown), ordinal with an explicit order, cross-fitted target encoding (scikit-learn TargetEncoder, CatBoost ordered statistics, category_encoders inside CV), frequency/count encoding, feature hashing, native categorical support in LightGBM, HistGradientBoosting, XGBoost and CatBoost, prefix levels for codes (zip3, NAICS-2), and embeddings. Measures cardinality, rare-level share, unseen-level rate on a time-ordered holdout and target signal per column, emits the ColumnTransformer, and quantifies target-encoding leakage (in-sample vs out-of-fold vs holdout AUC).
classification-model-selector
Choose the right classification algorithm for tabular, text or mixed data and prove the choice with a fair bake-off: logistic regression (L2, L1, elastic net), linear SVM, naive Bayes, k-NN, RBF SVM, decision trees, random forest, extra trees, HistGradientBoosting, LightGBM, XGBoost, CatBoost, explainable boosting machines (EBM/GAM), points scorecards and MLPs, matched to data size, width, categorical and high-cardinality share, missingness, class imbalance, text, and constraints such as interpretability, monotonic constraints, calibrated probabilities, latency budgets and tiny deployable models. Ranks candidates with starting hyperparameters and cautions, then generates a bake-off script running every candidate on identical CV folds with per-fold metrics, fit time and prediction latency.
clustering-algorithm-selector
Choose the right clustering algorithm and check that clustering makes sense at all: cluster tendency (Hopkins statistic with a correlated-features caveat, nearest-neighbour isolation), then k-means / MiniBatchKMeans, Gaussian mixtures with BIC, agglomerative (Ward, average, complete, single) with dendrograms, DBSCAN, HDBSCAN, OPTICS, spectral clustering, BIRCH, mean shift, k-prototypes and k-modes for mixed or categorical data, and Gower distance, matched to data size, expected shapes, noise, varying density, known k, soft memberships, hierarchy and streaming. Covers preprocessing (scaling, encoding, dimensionality, distance metrics), validation (internal indices, bootstrap stability, external labels, business profiling) and assigning new points.
collaborative-filtering-implementer
Implement collaborative filtering recommenders that beat popularity: item-based kNN with cosine, Jaccard or asymmetric similarity, shrinkage, neighbourhood size, positive-only or mean-centred rating signals and 'because you liked X' explanations; implicit ALS with confidence weighting, BPR pairwise ranking and the closed-form EASE model in plain NumPy, tuned on a temporal validation split by recall@k and NDCG@k and refit on all data; plus user-kNN, explicit matrix factorisation, the implicit/LightFM libraries, hyperparameter ranges, cold-start fallbacks, scaling with sparse matrices and approximate nearest neighbours. Produces top-k lists in the format of the recommendation evaluator.