A regression model estimates whatever its loss tells it to: squared error estimates the mean, absolute
error the median, pinball loss a quantile, Poisson/Gamma/Tweedie deviances a mean under a
variance that grows with the level. Picking the loss from the target's shape, and checking residuals for
bias by level and segment, matter more than the algorithm. scripts/target_diagnostics.py reads the target
and recommends the loss with the objective names for scikit-learn, LightGBM, XGBoost and CatBoost;
scripts/residual_report.py diagnoses predictions (decile calibration, heteroscedasticity, segment bias,
interval coverage) and can gate CI on bias.
Trigger Conditions
Activate this skill when the user:
- Trains a model to predict a quantity: price, demand, revenue, spend, LTV, duration, claim cost, counts
- Asks which loss/objective to use (MSE vs MAE vs Huber vs Poisson vs Tweedie vs log target)
- Sees large values systematically under-predicted, or errors that grow with the prediction
- Needs prediction intervals or a percentile (P90 demand, upper bound on delivery time)
- Has exposure (policy-years, store-days) or zero-heavy targets
- Wants a residual analysis or a bias gate for a regression model
Choosing the evaluation metric: ml-success-metric-selector; time-ordered targets: time-series-forecasting-model-builder; GBM tuning: xgboost-lightgbm-tuner.
Workflow
1. TARGET -> python scripts/target_diagnostics.py data.csv --target y [--need-quantile 0.9]
2. BASELINE -> mean/median and one-feature baselines (ml-baseline-model-builder)
3. LINEAR -> ridge / GLM with the recommended family (interpretable reference)
4. GBM -> HistGradientBoosting / LightGBM / XGBoost / CatBoost with the matching objective
5. PREDICT -> out-of-fold or holdout predictions to a CSV (y_true, y_pred, segment, intervals)
6. DIAGNOSE -> python scripts/residual_report.py preds.csv --segment region --max-abs-bias-pct 10
7. INTERVALS -> quantile models or conformal intervals; check coverage on the holdout
8. REPORT -> metric in business units, bias by decile/segment, coverageTarget Shape -> Loss
| Target | Loss / approach | scikit-learn | LightGBM | XGBoost | CatBoost |
|---|---|---|---|---|---|
| Symmetric, light tails | squared error | Ridge, HGB loss="squared_error" | regression | reg:squarederror | RMSE |
| Heavy-tailed noise / outliers | Huber or absolute | HuberRegressor, HGB loss="absolute_error" | huber, regression_l1 | reg:pseudohubererror, reg:absoluteerror | Huber:delta=, MAE |
| Positive, right-skewed, ~log-normal | log1p target (+ smearing) | TransformedTargetRegressor(np.log1p, np.expm1) | train on log1p | train on log1p | train on log1p |
| Positive, skewed, var ∝ mean² | Gamma deviance | GammaRegressor, HGB loss="gamma" | gamma | reg:gamma | Tweedie:variance_power=1.99 |
| Counts | Poisson deviance (+ exposure) | PoissonRegressor, HGB loss="poisson" | poisson | count:poisson | Poisson |
| Zeros + positive tail (loss cost, spend) | Tweedie, 1 < p < 2 | TweedieRegressor(power=1.5) | tweedie + tweedie_variance_power | reg:tweedie | Tweedie:variance_power=1.5 |
| Zeros from a separate process | two-part: classifier x positive regressor | two models | two models | two models | two models |
| Proportion in (0, 1) | logit transform or cross-entropy | TransformedTargetRegressor(logit, expit) | cross_entropy | reg:logistic | CrossEntropy |
| A percentile is the decision | quantile (pinball) | HGB loss="quantile", quantile=q | quantile, alpha=q | reg:quantileerror | Quantile:alpha=q |
On the fixtures: house prices -> log1p target (skew 4.6, log-skew 0.39); claim counts -> Poisson (variance/mean 1.01); claim amounts with 90% zeros -> Tweedie (two-part as the alternative); utilisation in (0, 1) -> logit. Loss mechanics, smearing and Tweedie power selection: losses-and-targets.md.
Model Recipes
import numpy as np
from sklearn.compose import TransformedTargetRegressor
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.linear_model import PoissonRegressor, TweedieRegressor
# Log target for prices (predicts ~median; multiply by a smearing factor when sums matter)
price_model = TransformedTargetRegressor(
regressor=make_pipeline(prep, HistGradientBoostingRegressor(max_leaf_nodes=31, learning_rate=0.05)),
func=np.log1p, inverse_func=np.expm1)
# Claim frequency with exposure: model the rate, weight by exposure
freq = HistGradientBoostingRegressor(loss="poisson")
freq.fit(X, y_counts / exposure, sample_weight=exposure) # predicts claims per unit exposure
# Pure premium (zeros + amounts)
pp = make_pipeline(prep, TweedieRegressor(power=1.6, link="log", alpha=0.01, max_iter=1000))LightGBM with an exposure offset (init_score=np.log(exposure)), monotone constraints, sample weights,
and CatBoost/XGBoost equivalents: model-recipes.md.
Reading Residuals
Calibration by prediction decile (mean pred / mean true):
D1:0.95 D2:1.00 D3:1.01 ... D8:0.84 D9:0.85 D10:0.87
|residual| vs prediction correlation +0.79: errors grow with the level (heteroscedastic)
By segment: large n=102 MAE 136.5 bias -134.2 (-14.8%)
Interval coverage 57.3% (nominal 80%) <- too narrow| Pattern | Cause | Fix |
|---|---|---|
| Top deciles under-predicted, bottom over-predicted | shrinkage / regularisation, squared error on skewed target, missing driver of large values | log target or Gamma/Tweedie; features for large cases; less regularisation |
| residual | grows with the prediction | |
| One segment biased | missing interaction or segment feature; different population | add the feature/interaction; separate model; sample weights |
| Bias overall after log target | retransformation bias | smearing correction or deviance loss |
| Intervals cover far less than nominal | model too confident; distribution shift | conformal calibration on a holdout; wider quantiles |
| Worst errors are data errors | outliers, unit mistakes | fix data; robust loss |
More diagnostics (partial residual plots, residuals vs time, influence): residual-diagnostics.md.
Prediction Intervals
| Method | How | Notes |
|---|---|---|
| Quantile models | fit q = 0.1 and q = 0.9 models (HGB / LightGBM quantile) | flexible; quantiles can cross; check coverage |
| Split conformal | holdout residuals -> add the (1 − α) quantile of | residual |
| Conformalised quantile regression (CQR) | quantile models + conformal correction | adaptive width with valid coverage |
| GLM / parametric | variance from the family | only if the family is right |
Code for each and coverage checks: prediction-intervals.md.
Output Format
When training or reviewing a regression model, deliver:
- Target diagnosis - shape statistics and the chosen loss with the reason (
target_diagnostics.py). - Models - baseline, linear/GLM and GBM with the matching objective; exposure/weights if relevant.
- Metrics - MAE/RMSE/WAPE (business units), against baselines, with CV spread.
- Residual report - decile calibration, heteroscedasticity, segment bias, worst rows (
residual_report.py). - Intervals - method and holdout coverage, if uncertainty matters.
- Fixes - what the residuals suggest next (features, loss, segmentation).
Scripts
python scripts/target_diagnostics.py data.csv --target price
python scripts/target_diagnostics.py data.csv --target claim_amount --json
python scripts/target_diagnostics.py data.csv --target units --need-quantile 0.9Reports quantiles, mean vs median, skew and skew of log, zero and negative shares, integer-ness, tail ratio, dispersion and outlier share; recommends squared, Huber/absolute, log1p, Gamma, Poisson (zero-inflation check against the Poisson zero share), Tweedie (two-part alternative when zeros > 50%), logit or quantile, with scikit-learn / LightGBM / XGBoost / CatBoost objective names, the matching metric and notes (smearing, tails, negatives). Exit codes: 0 diagnosed, 2 usage error or non-numeric target.
python scripts/residual_report.py preds.csv
python scripts/residual_report.py preds.csv --segment region --slice-by sqft year_built --top 5
python scripts/residual_report.py preds.csv --lower p10 --upper p90 --nominal 0.8
python scripts/residual_report.py preds.csv --max-abs-bias-pct 10 --jsonMAE, RMSE, MedAE, bias, R2, WAPE; mean prediction / mean actual per prediction decile; |residual| vs prediction correlation; MAE and bias per segment and per quartile of slice columns; interval coverage and width; the largest errors with CSV line numbers; a bias gate. Exit codes: 0 printed (gate passed), 1 gate failed, 2 usage or read error.
Additional Resources
| File | Content |
|---|---|
| losses-and-targets.md | What each loss estimates, deviance families, log-target retransformation bias and smearing, Tweedie power choice, hurdle models, proportions |
| model-recipes.md | Ridge/lasso/GLMs, HistGBM, LightGBM, XGBoost, CatBoost code per objective; exposure offsets; weights; monotone constraints; early stopping |
| residual-diagnostics.md | Decile calibration, heteroscedasticity, segment and time slices, partial residuals, influential points, what each pattern means |
| prediction-intervals.md | Quantile regression, split conformal, CQR, MAPIE, coverage evaluation, crossing quantiles |
anomaly-detection-model-builder
Build and evaluate anomaly detection for tabular, sensor and transaction data with few or no labels: define the anomaly type (point, contextual, collective), choose detectors (robust z-score/MAD and IQR rules, Mahalanobis with robust covariance, k-NN distance, LOF, Isolation Forest, one-class SVM, PCA reconstruction / autoencoders, forecasting residuals for time series), combine them by rank, explain each alert by its deviating features, set thresholds from an alert budget or the score distribution, and evaluate with precision@k, PR-AUC and recall on known incidents, or detector agreement when no labels exist. Includes pure-Python detectors (robust z, Mahalanobis, kNN, Isolation Forest, ensemble) and an evaluation script.
catboost-model-builder
Build CatBoost models that use its strengths correctly: raw categorical columns as cat_features (ordered target statistics, no manual encoding), text_features, Pool construction with NaN-safe categories, ordered boosting and has_time for temporal data, symmetric (oblivious) trees, overfitting detector with eval_set and use_best_model, losses for classification, regression (Quantile, Poisson, Tweedie) and ranking (YetiRank), class weights, monotone constraints, feature importance (PredictionValuesChange, LossFunctionChange) and SHAP values, GPU training and model export. Plans cat/text/numeric columns from a CSV and generates a runnable training script, and checks parameter sets for the conflicts CatBoost rejects (iterations vs n_estimators, Bayesian bootstrap with subsample, multiple class-weight options, depth > 16, use_best_model without eval_set).
categorical-encoding-selector
Choose and implement the right encoding for every categorical column given the model family and the data: one-hot (with infrequent-level grouping and handle_unknown), ordinal with an explicit order, cross-fitted target encoding (scikit-learn TargetEncoder, CatBoost ordered statistics, category_encoders inside CV), frequency/count encoding, feature hashing, native categorical support in LightGBM, HistGradientBoosting, XGBoost and CatBoost, prefix levels for codes (zip3, NAICS-2), and embeddings. Measures cardinality, rare-level share, unseen-level rate on a time-ordered holdout and target signal per column, emits the ColumnTransformer, and quantifies target-encoding leakage (in-sample vs out-of-fold vs holdout AUC).
classification-model-selector
Choose the right classification algorithm for tabular, text or mixed data and prove the choice with a fair bake-off: logistic regression (L2, L1, elastic net), linear SVM, naive Bayes, k-NN, RBF SVM, decision trees, random forest, extra trees, HistGradientBoosting, LightGBM, XGBoost, CatBoost, explainable boosting machines (EBM/GAM), points scorecards and MLPs, matched to data size, width, categorical and high-cardinality share, missingness, class imbalance, text, and constraints such as interpretability, monotonic constraints, calibrated probabilities, latency budgets and tiny deployable models. Ranks candidates with starting hyperparameters and cautions, then generates a bake-off script running every candidate on identical CV folds with per-fold metrics, fit time and prediction latency.
clustering-algorithm-selector
Choose the right clustering algorithm and check that clustering makes sense at all: cluster tendency (Hopkins statistic with a correlated-features caveat, nearest-neighbour isolation), then k-means / MiniBatchKMeans, Gaussian mixtures with BIC, agglomerative (Ward, average, complete, single) with dendrograms, DBSCAN, HDBSCAN, OPTICS, spectral clustering, BIRCH, mean shift, k-prototypes and k-modes for mixed or categorical data, and Gower distance, matched to data size, expected shapes, noise, varying density, known k, soft memberships, hierarchy and streaming. Covers preprocessing (scaling, encoding, dimensionality, distance metrics), validation (internal indices, bootstrap stability, external labels, business profiling) and assigning new points.
collaborative-filtering-implementer
Implement collaborative filtering recommenders that beat popularity: item-based kNN with cosine, Jaccard or asymmetric similarity, shrinkage, neighbourhood size, positive-only or mean-centred rating signals and 'because you liked X' explanations; implicit ALS with confidence weighting, BPR pairwise ranking and the closed-form EASE model in plain NumPy, tuned on a temporal validation split by recall@k and NDCG@k and refit on all data; plus user-kNN, explicit matrix factorisation, the implicit/LightFM libraries, hyperparameter ranges, cold-start fallbacks, scaling with sparse matrices and approximate nearest neighbours. Produces top-k lists in the format of the recommendation evaluator.