Single-feature screens
Before any model, score each column alone against the target:
| Target | Numeric feature | Categorical feature |
|---|---|---|
| Binary | ROC-AUC of the raw value (direction-free: max(AUC, 1-AUC)) | AUC of out-of-fold smoothed target mean per level |
| Numeric | Spearman | |
| Multiclass | one-vs-rest AUC for each class, take the max | same |
Why out-of-fold for categoricals
A high-cardinality column (zip code, user ID) gets a "target mean per level" that memorises the label in-sample,
producing a fake AUC near 1.0. Computing each row's level mean from the other fold, with smoothing toward
the global mean ((sum + m*prior) / (count + m)), removes that optimism. The same logic is why target
encoding must be done inside CV folds (see categorical-encoding-selector).
Missingness as a signal
Fields filled only when the outcome happens (cancellation_reason, chargeback_date, resolution_code) are
empty for negatives and filled for positives. Even if their values are never used, an "is missing" flag
created implicitly by imputation leaks the label. Score is_missing(col) as its own feature.
Reading the screen
- Best signal tiny: univariate signal is absent. Interactions can exist (XOR-like), but in tabular business data that is rare; get new data sources before building.
- Several moderate signals: the usual healthy case.
- One near-perfect feature: a leak until proven otherwise. Ask when the field is written.
- IDs or row numbers with signal: data is sorted by outcome or time; the split or the data pipeline leaks.
Leak signatures to look for
| Signature | Example | Check |
|---|---|---|
| Outcome-side fields | refund_issued, cancellation_reason | when is it written? |
| Post-event timestamps | last_activity_date read today | compare with the snapshot date |
| Aggregates over all history | lifetime_orders computed at extraction | recompute as-of |
| Missingness pattern | field null unless the event happened | is-missing AUC |
| ID or sequence | autoincrement IDs assigned at labelling | AUC of the ID |
| Duplicates across train/test | the same customer in both | grouped split |
| Target encoded on full data | category mean includes the row's own label | OOF encoding |
Label noise
Estimate before modelling when labels come from people or heuristics:
- Sample 200-500 rows stratified by label.
- Have two people label them independently with the written guideline.
- Compute agreement and Cohen's kappa:
text kappa = (p_observed - p_expected) / (1 - p_expected)
| Kappa | Agreement |
|---|---|
| < 0.4 | poor: guideline or concept is unclear |
| 0.4 - 0.6 | moderate |
| 0.6 - 0.8 | substantial |
| > 0.8 | almost perfect |
With noise rate e on a binary label, the best achievable accuracy against those labels is about 1 - e; a model "stuck" at 85% may be at the noise ceiling.
Label drift by period
Compute the label rate per month of prediction date. Compare each month with the overall rate using a z-score
(p_month - p) / sqrt(p (1 - p) / n_month); flag |z| > 3. Common causes: a change in the label definition
(new cancellation flow), a pipeline bug, a product change, seasonality. Decide which period is representative
of the future and validate there.
Selection bias
Labels exist only where someone looked:
- Credit: repayment observed only for approved loans.
- Fraud: confirmed only for reviewed or charged-back transactions.
- Marketing: conversion observed only for contacted leads.
Consequences: the model learns the reviewer's policy, and performance on the unreviewed population is unknown. Mitigations: a random exploration sample (approve/review a small random share), reject inference, or limit the model's scope to the population with labels and state it.
Feature availability audit
For each candidate source fill this table:
| Source | Owner | Update cadence | Written when | Available at prediction time? | Latency to fetch |
|---|---|---|---|---|---|
| billing.invoices | finance | hourly | at invoice creation | yes | 50 ms (cache) |
| crm.cases | support | real-time | at case open; status overwritten | only as-of snapshot | warehouse only |
| churn_survey | marketing | weekly | after cancellation | no | - |
Any "no" that carries most of the signal is a NO-GO for the current framing.
anomaly-detection-model-builder
Build and evaluate anomaly detection for tabular, sensor and transaction data with few or no labels: define the anomaly type (point, contextual, collective), choose detectors (robust z-score/MAD and IQR rules, Mahalanobis with robust covariance, k-NN distance, LOF, Isolation Forest, one-class SVM, PCA reconstruction / autoencoders, forecasting residuals for time series), combine them by rank, explain each alert by its deviating features, set thresholds from an alert budget or the score distribution, and evaluate with precision@k, PR-AUC and recall on known incidents, or detector agreement when no labels exist. Includes pure-Python detectors (robust z, Mahalanobis, kNN, Isolation Forest, ensemble) and an evaluation script.
catboost-model-builder
Build CatBoost models that use its strengths correctly: raw categorical columns as cat_features (ordered target statistics, no manual encoding), text_features, Pool construction with NaN-safe categories, ordered boosting and has_time for temporal data, symmetric (oblivious) trees, overfitting detector with eval_set and use_best_model, losses for classification, regression (Quantile, Poisson, Tweedie) and ranking (YetiRank), class weights, monotone constraints, feature importance (PredictionValuesChange, LossFunctionChange) and SHAP values, GPU training and model export. Plans cat/text/numeric columns from a CSV and generates a runnable training script, and checks parameter sets for the conflicts CatBoost rejects (iterations vs n_estimators, Bayesian bootstrap with subsample, multiple class-weight options, depth > 16, use_best_model without eval_set).
categorical-encoding-selector
Choose and implement the right encoding for every categorical column given the model family and the data: one-hot (with infrequent-level grouping and handle_unknown), ordinal with an explicit order, cross-fitted target encoding (scikit-learn TargetEncoder, CatBoost ordered statistics, category_encoders inside CV), frequency/count encoding, feature hashing, native categorical support in LightGBM, HistGradientBoosting, XGBoost and CatBoost, prefix levels for codes (zip3, NAICS-2), and embeddings. Measures cardinality, rare-level share, unseen-level rate on a time-ordered holdout and target signal per column, emits the ColumnTransformer, and quantifies target-encoding leakage (in-sample vs out-of-fold vs holdout AUC).
classification-model-selector
Choose the right classification algorithm for tabular, text or mixed data and prove the choice with a fair bake-off: logistic regression (L2, L1, elastic net), linear SVM, naive Bayes, k-NN, RBF SVM, decision trees, random forest, extra trees, HistGradientBoosting, LightGBM, XGBoost, CatBoost, explainable boosting machines (EBM/GAM), points scorecards and MLPs, matched to data size, width, categorical and high-cardinality share, missingness, class imbalance, text, and constraints such as interpretability, monotonic constraints, calibrated probabilities, latency budgets and tiny deployable models. Ranks candidates with starting hyperparameters and cautions, then generates a bake-off script running every candidate on identical CV folds with per-fold metrics, fit time and prediction latency.
clustering-algorithm-selector
Choose the right clustering algorithm and check that clustering makes sense at all: cluster tendency (Hopkins statistic with a correlated-features caveat, nearest-neighbour isolation), then k-means / MiniBatchKMeans, Gaussian mixtures with BIC, agglomerative (Ward, average, complete, single) with dendrograms, DBSCAN, HDBSCAN, OPTICS, spectral clustering, BIRCH, mean shift, k-prototypes and k-modes for mixed or categorical data, and Gower distance, matched to data size, expected shapes, noise, varying density, known k, soft memberships, hierarchy and streaming. Covers preprocessing (scaling, encoding, dimensionality, distance metrics), validation (internal indices, bootstrap stability, external labels, business profiling) and assigning new points.
collaborative-filtering-implementer
Implement collaborative filtering recommenders that beat popularity: item-based kNN with cosine, Jaccard or asymmetric similarity, shrinkage, neighbourhood size, positive-only or mean-centred rating signals and 'because you liked X' explanations; implicit ALS with confidence weighting, BPR pairwise ranking and the closed-form EASE model in plain NumPy, tuned on a temporal validation split by recall@k and NDCG@k and refit on all data; plus user-kNN, explicit matrix factorisation, the implicit/LightFM libraries, hyperparameter ranges, cold-start fallbacks, scaling with sparse matrices and approximate nearest neighbours. Produces top-k lists in the format of the recommendation evaluator.