target-leakage-detector

Leakage is information in training that will not exist when the model makes a real prediction. It comes in two kinds: data leakage (a column that knows the answer: written after the event, derived from the label, or aggregated over the future) and pipeline leakage (the evaluation lets test rows influence training: preprocessing, selection or resampling fitted before the split). Both inflate offline scores and collapse in production. scripts/leakage_scan.py scans a training table for eight leakage signatures; scripts/code_leak_lint.py lints training code for ten pipeline-leakage bugs.

Trigger Conditions

Activate this skill when the user:

Gets an implausibly good result (AUC > 0.95, R² > 0.95 on a hard business problem, a first baseline that beats the expected ceiling)
Sees offline metrics far better than shadow/online metrics, or a model that decays immediately after launch
Has one feature dominating importance or SHAP, especially a status, date, reason, flag or ID
Is building a new feature table, label query or training script and wants a pre-flight check
Uses target encoding, SMOTE, feature selection or scalers and is unsure whether CV is honest
Asks "is this feature leaking?" or "why is my CV score so much higher than my test score?"

Split-level problems (duplicates and entity overlap between train and test, time overlap) are covered in depth by train-test-split-auditor; point-in-time feature building by time-based-feature-builder.

Workflow

text

1. PREDICTION TIME -> write down when the model scores (the leakage line) and the label window
2. DATA SCAN       -> python scripts/leakage_scan.py table.csv --target y --time-col score_date --drop id
3. CODE SCAN       -> python scripts/code_leak_lint.py train.py
4. INVESTIGATE     -> for each high finding: when is the field written? by which process? overwritten?
5. FIX             -> drop, rebuild as-of prediction time, or move the step inside the Pipeline
6. CONFIRM         -> time-ordered holdout; the score should drop to a plausible level and stay stable by period
7. GUARD           -> keep both scans in CI for the feature table and the training code

Data Leakage Signatures (leakage_scan.py)

Code	Signature	Typical cause	Fix
LD001	one feature alone gives AUC > 0.9 (or	Spearman	> 0.95)
LD002	is-missing predicts the target	field filled only when the event happens (reason, resolution date)	drop; never impute it
LD003	dates later than the prediction time	`last_login` read today for an old snapshot; event dates after scoring	as-of snapshot tables
LD004	outcome-like column name	status, reason, refund, chargeback, closed, settled, approved	confirm write time with the owner
LD005	row order / ID predicts the target	file sorted by outcome; IDs assigned at labelling	never use IDs; split deliberately
LD006	feature is a deterministic function of the target	label encoded in another column, joins on the label table	drop
LD007	identical features with different labels; duplicates	label noise, bad joins, duplicated events	dedupe; group-aware splits
LD008	much stronger signal in old rows than new rows	backfilled or overwritten history	snapshot history; time-based validation

text

LD001 [high] account_status: single feature AUC 0.939 > 0.9: almost certainly leakage or a proxy of the label
LD003 [high] last_login_date: 10% of values are later than score_date: information from the future
LD005 [medium] (row order): row position predicts the target (AUC 1.00): the file is sorted by outcome

Full taxonomy with real-world cases (healthcare, credit, churn, fraud, competitions): leakage-taxonomy.md.

Pipeline Leakage (code_leak_lint.py)

Code	Bug	Correct pattern
LK001	scaler / imputer / encoder / PCA fitted before the split or CV	all preprocessing inside a `Pipeline` that is cross-validated
LK002	SMOTE / over- / under-sampling before the split	`imblearn.pipeline.Pipeline([... ("smote", SMOTE()), ("model", ...)])`
LK003	feature selection on all data before CV	selector inside the Pipeline; nested CV if tuned
LK004	target column left in X	`X = df.drop(columns=[target, ...])`
LK005	`.fit()` on test / validation data	fit on train only; `transform` elsewhere
LK006	`groupby(...)[target].transform("mean")` on the training table	`TargetEncoder` (cross-fitted) or expanding means over past rows
LK007	random split / shuffled KFold on temporal data	`TimeSeriesSplit`, cutoff dates, gap for label windows
LK008	`fillna(df[c].mean())`, manual standardisation before the split	`SimpleImputer` / scalers in the Pipeline
LK009	early stopping on the test set	separate validation split for `eval_set`
LK010	CV of a bare estimator while preprocessing exists	cross-validate the whole Pipeline

python

from imblearn.pipeline import Pipeline            # supports samplers; behaves like sklearn's Pipeline
pipe = Pipeline([
    ("prep", ColumnTransformer([...])),           # fitted per training fold
    ("select", SelectFromModel(LogisticRegression(penalty="l1", solver="liblinear", C=0.1))),
    ("smote", SMOTE(random_state=0)),             # resamples training folds only
    ("model", LogisticRegression(max_iter=2000)),
])
scores = cross_val_score(pipe, X, y, cv=TimeSeriesSplit(n_splits=5, gap=4))

Before/after code for every rule: pipeline-leakage-patterns.md.

Point-in-Time Correctness

For each row, every feature must be computable from data that existed at that row's prediction time:

Join slowly changing tables "as of" the prediction time (pd.merge_asof, SQL valid_from <= t < valid_to).
Aggregates use only events strictly before t (shift() before rolling; closed="left" windows).
Respect data latency: if yesterday's events land at 06:00, a 02:00 score cannot use them.
Respect label latency for target-derived features: only labels known by t.

Patterns and SQL: point-in-time-correctness.md.

Investigating a Suspicious Feature

Who writes it, and when? Ask the data owner; read the ETL. A field "updated nightly" overwrites history.
Look at its timeline: value distribution by days before the event. A jump just before the label is suspicious; a jump after is leakage.
Rebuild it as-of and compare single-feature AUC: a leak collapses, a real signal survives weaker.
Ablate: train without it on a time-ordered holdout. A small loss means it carried little real signal.
Shadow test: compute it live at prediction time and compare with the training value.

Playbook with queries and decision rules: investigation-playbook.md.

Plausibility Check

Problem	Typical honest ROC-AUC	Treat as suspicious above
Customer churn (30-day)	0.70-0.85	0.92
Card fraud (rich velocity features)	0.90-0.97	0.99
Credit default (12-month)	0.70-0.80	0.88
Lead conversion	0.70-0.85	0.92
Hospital readmission (30-day)	0.65-0.75	0.85
Click-through	0.70-0.80	0.90

Ranges are rough; an unusually strong result needs a leakage review before celebration, not after launch.

Output Format

When checking for leakage, answer with:

Prediction time and label window used as the leakage line.
Data findings - leakage_scan.py output, each high finding with the likely mechanism.
Code findings - code_leak_lint.py output with the corrected code.
Investigation results - write times confirmed, as-of rebuilds, ablation deltas.
Fixed evaluation - score on a time-ordered holdout after fixes, stability by period.
Guardrails - scans added to CI, data contracts for the risky sources.

Scripts

bash

python scripts/leakage_scan.py table.csv --target churned
python scripts/leakage_scan.py table.csv --target churned --time-col score_date --drop customer_id
python scripts/leakage_scan.py table.csv --target price --auc 0.85 --json

Checks LD001-LD008: out-of-fold single-feature AUC (binary) or |Spearman| (numeric) above --auc (default 0.9), is-missing AUC > 0.85, datetime columns later than --time-col, outcome-like names, row order and numeric IDs that predict the target, deterministic level-to-class mappings, conflicting or duplicated feature rows, and signal decay from oldest to newest third. Exit codes: 0 no high findings, 1 high findings, 2 usage or read error.

bash

python scripts/code_leak_lint.py train.py
python scripts/code_leak_lint.py notebooks/*.py --json
cat train.py | python scripts/code_leak_lint.py -

AST rules LK001-LK010: preprocessing, selection and resampling fitted before the split; target column left in X; fitting on test/validation data; in-sample groupby target means; shuffled splits in code that handles dates; full-data fillna/standardisation; early stopping on the test set; CV of bare estimators while preprocessing exists. Heuristic by design: export notebooks with jupyter nbconvert --to script first. Exit codes: 0 no high findings, 1 high findings, 2 read or syntax error.

Additional Resources

File	Content
leakage-taxonomy.md	Every leakage type with real cases: outcome fields, future data, proxies, IDs, duplicates, target encoding, temporal and group leakage, competition leaks
point-in-time-correctness.md	As-of joins (pandas merge_asof, SQL), snapshot tables, event-time vs processing-time, data and label latency
pipeline-leakage-patterns.md	Before/after code for LK001-LK010, imblearn pipelines, nested CV, early stopping splits
investigation-playbook.md	Step-by-step investigation, timeline queries, as-of rebuild comparison, ablation, CI guards

anomaly-detection-model-builder

Build and evaluate anomaly detection for tabular, sensor and transaction data with few or no labels: define the anomaly type (point, contextual, collective), choose detectors (robust z-score/MAD and IQR rules, Mahalanobis with robust covariance, k-NN distance, LOF, Isolation Forest, one-class SVM, PCA reconstruction / autoencoders, forecasting residuals for time series), combine them by rank, explain each alert by its deviating features, set thresholds from an alert budget or the score distribution, and evaluate with precision@k, PR-AUC and recall on known incidents, or detector agreement when no labels exist. Includes pure-Python detectors (robust z, Mahalanobis, kNN, Isolation Forest, ensemble) and an evaluation script.

catboost-model-builder

Build CatBoost models that use its strengths correctly: raw categorical columns as cat_features (ordered target statistics, no manual encoding), text_features, Pool construction with NaN-safe categories, ordered boosting and has_time for temporal data, symmetric (oblivious) trees, overfitting detector with eval_set and use_best_model, losses for classification, regression (Quantile, Poisson, Tweedie) and ranking (YetiRank), class weights, monotone constraints, feature importance (PredictionValuesChange, LossFunctionChange) and SHAP values, GPU training and model export. Plans cat/text/numeric columns from a CSV and generates a runnable training script, and checks parameter sets for the conflicts CatBoost rejects (iterations vs n_estimators, Bayesian bootstrap with subsample, multiple class-weight options, depth > 16, use_best_model without eval_set).

categorical-encoding-selector

Choose and implement the right encoding for every categorical column given the model family and the data: one-hot (with infrequent-level grouping and handle_unknown), ordinal with an explicit order, cross-fitted target encoding (scikit-learn TargetEncoder, CatBoost ordered statistics, category_encoders inside CV), frequency/count encoding, feature hashing, native categorical support in LightGBM, HistGradientBoosting, XGBoost and CatBoost, prefix levels for codes (zip3, NAICS-2), and embeddings. Measures cardinality, rare-level share, unseen-level rate on a time-ordered holdout and target signal per column, emits the ColumnTransformer, and quantifies target-encoding leakage (in-sample vs out-of-fold vs holdout AUC).

classification-model-selector

Choose the right classification algorithm for tabular, text or mixed data and prove the choice with a fair bake-off: logistic regression (L2, L1, elastic net), linear SVM, naive Bayes, k-NN, RBF SVM, decision trees, random forest, extra trees, HistGradientBoosting, LightGBM, XGBoost, CatBoost, explainable boosting machines (EBM/GAM), points scorecards and MLPs, matched to data size, width, categorical and high-cardinality share, missingness, class imbalance, text, and constraints such as interpretability, monotonic constraints, calibrated probabilities, latency budgets and tiny deployable models. Ranks candidates with starting hyperparameters and cautions, then generates a bake-off script running every candidate on identical CV folds with per-fold metrics, fit time and prediction latency.

clustering-algorithm-selector

Choose the right clustering algorithm and check that clustering makes sense at all: cluster tendency (Hopkins statistic with a correlated-features caveat, nearest-neighbour isolation), then k-means / MiniBatchKMeans, Gaussian mixtures with BIC, agglomerative (Ward, average, complete, single) with dendrograms, DBSCAN, HDBSCAN, OPTICS, spectral clustering, BIRCH, mean shift, k-prototypes and k-modes for mixed or categorical data, and Gower distance, matched to data size, expected shapes, noise, varying density, known k, soft memberships, hierarchy and streaming. Covers preprocessing (scaling, encoding, dimensionality, distance metrics), validation (internal indices, bootstrap stability, external labels, business profiling) and assigning new points.

collaborative-filtering-implementer

Implement collaborative filtering recommenders that beat popularity: item-based kNN with cosine, Jaccard or asymmetric similarity, shrinkage, neighbourhood size, positive-only or mean-centred rating signals and 'because you liked X' explanations; implicit ALS with confidence weighting, BPR pairwise ranking and the closed-form EASE model in plain NumPy, tuned on a temporal validation split by recall@k and NDCG@k and refit on all data; plus user-kNN, explicit matrix factorisation, the implicit/LightFM libraries, hyperparameter ranges, cold-start fallbacks, scaling with sparse matrices and approximate nearest neighbours. Produces top-k lists in the format of the recommendation evaluator.

target-leakage-detector

#Trigger Conditions

#Workflow

#Data Leakage Signatures (leakage_scan.py)

#Pipeline Leakage (code_leak_lint.py)

#Point-in-Time Correctness

#Investigating a Suspicious Feature

#Plausibility Check

#Output Format

#Scripts

#Additional Resources