Skip to content
v0.1.3latest release121 skills4 categories6 sub-categories247 tested scripts397 reference docsContact us
Vector Agentic Skills
Submit a skill

train-test-split-auditor

Audit a train/test (or train/validation, train/production) split before trusting test scores: exact duplicate rows across sets, entity leakage (the same customer, patient, device or session on both sides), time overlap in forward splits, near-duplicates (rounded numerics, case/whitespace variants, edited text copies via MinHash and LSH), feature drift (PSI, KS), single features that separate train from test (univariate adversarial AUC), target-distribution shift, unseen categories and missing-value differences; plus how to build a correct split (random, stratified, grouped, time-based, out-of-distribution) and how to run full adversarial validation. Use when someone asks whether their train/test split is valid, why the test score is too good or collapses in production, how to check for duplicates or leakage between splits, or how different the test or production data is from training.

AI & Machine Learning / Classical Machine Learningv1.0.0Updated OCT 25 20263 references2 scripts

A test score is only as honest as the split behind it. Three failures inflate it: the same thing on both sides (duplicate rows, near-duplicates, the same entity), the future in the past (training rows later than test rows when the model predicts forward), and an easy test set (test data that differs from production). Shift is not inflation but matters just as much: a test set that looks unlike production measures the wrong thing. scripts/split_audit.py runs nine checks on two CSVs; scripts/near_duplicates.py finds the duplicates that exact matching misses, including edited text copies.

Trigger Conditions

Activate this skill when the user:

  • Asks "is my train/test split OK?" or "why is my test score so high?"
  • Sees a model collapse in production after a great test score
  • Has repeated entities, dumps from several systems, scraped or user-generated text (likely duplicates)
  • Wants to compare a training set with a production sample (drift before deployment)
  • Builds a holdout set and wants to be sure it is clean

CV design (folds rather than one split): cross-validation-strategy-designer; feature-level leakage: target-leakage-detector.

Workflow

text
1. DESCRIBE  -> how was the split made? what will production look like (new entities, later time)?
2. AUDIT     -> python scripts/split_audit.py train.csv test.csv --target y --id-col id [--time-col ts --expect forward]
3. NEAR-DUPS -> python scripts/near_duplicates.py train.csv test.csv --drop id [--text-col body]
4. FIX       -> dedupe / group split / time split; rebuild the test set
5. SHIFT     -> python scripts/split_audit.py train.csv prod_sample.csv --drop id   (is test like production?)
6. RECHECK   -> re-run both scripts; re-evaluate the model on the cleaned split

Checks

CodeCheckWhy it mattersFix
SA001identical feature rows in both setsthe model has memorised test rowsdedupe before splitting, or group duplicates
SA002ID / entity on both sidesentity memorisation (customer, patient)grouped split by entity
SA003test rows not after the last training time (forward expectation)future information in trainingtime-based split, with a gap for label windows
SA004feature drift (PSI > 0.25; KS)test unlike traininginvestigate; maybe the split is right and the world changed
SA005one feature separates the sets (AUC > 0.7)split encoded in a feature (time, source system)remove/rebuild the feature or accept and document a time split
SA006target distribution shiftmetrics not comparable; prior shiftstratify, or recalibrate
SA007unseen categories in testencoders must handle unknownshandle_unknown, monitor rate
SA008missing-share difference > 10 pointspipeline / source differencesfix extraction
SA009sizes, ratio, forward/backwardcontext-

On the churn fixture the audit finds the three planted problems (12 duplicated customers, 10.7% of test):

text
SA001 [high] (rows): 12 test row(s) (10.7%) have an identical feature row in train
SA002 [high] customer_id: 12 customer_id value(s) in both sets, covering 12 test rows (10.7%)
SA003 [high] signup_date: 12 test row(s) (10.7%) are not after the last training time 2024-02-02

Near-Duplicates

Exact matching misses: floats re-exported with noise (3 vs 3.0000000001), case and whitespace changes, and edited copies of text (tickets re-opened, reviews posted twice, product titles with a suffix). near_duplicates.py rounds numerics to significant digits, normalises text, and uses MinHash with LSH banding on character shingles to find text pairs above a Jaccard threshold. On the tickets fixture it finds all 20 planted copies (15 upper-cased bodies, 5 float-noise rows). Methods, thresholds and image/audio equivalents: duplicates-and-entity-leakage.md.

Building a Correct Split

ProductionSplit
i.i.d. rowsstratified random (train_test_split(stratify=y)) after deduplication
New entitiesGroupShuffleSplit by entity
Next periodtrain on dates < cutoff, test on dates >= cutoff (+ gap for label windows)
New regions / segmentshold out whole segments (out-of-distribution test)
Several of the abovetime split first, then drop test entities from training if new entities matter
python
from sklearn.model_selection import GroupShuffleSplit
df = df.drop_duplicates(subset=feature_cols)                          # before splitting
tr_idx, te_idx = next(GroupShuffleSplit(test_size=0.2, random_state=0).split(df, groups=df.customer_id))

Size guidance, holdout policy and multi-way splits: split-design.md.

Drift and Adversarial Validation

PSI and KS describe per-feature shift; adversarial validation answers the joint question: train a classifier to tell train rows from test (or production) rows.

Adversarial AUCMeaning
~0.5indistinguishable: the test set represents training conditions
0.6-0.7mild shift; check the top features
> 0.8strong shift or an encoded split (time, source); investigate before trusting test scores

Code, PSI/KS/Wasserstein details and what to do about shift: drift-and-adversarial-validation.md.

Output Format

When auditing a split, deliver:

  1. Split description - how it was made and what production looks like.
  2. Findings - split_audit.py and near_duplicates.py output, high findings first, with the affected share.
  3. Impact - re-evaluated test score after fixing (when possible), or the expected direction of bias.
  4. Fixes - dedupe, grouped or time split, rebuilt holdout; code.
  5. Shift - drift and adversarial AUC between test and production samples.
  6. Holdout policy - how the cleaned test set will be protected and used once.

Scripts

bash
python scripts/split_audit.py train.csv test.csv --target churned --id-col customer_id
python scripts/split_audit.py train.csv test.csv --target churned --time-col signup_date --expect forward
python scripts/split_audit.py train.csv prod_sample.csv --drop customer_id --psi-max 0.2 --json

SA001-SA009: duplicate feature rows across sets, ID/group overlap, forward-time violations, PSI (train deciles or level shares) with KS for numerics (ISO dates compared as day numbers), univariate train-vs-test AUC, target shift (PSI or binary rate outside 3 SE), unseen levels, missing-share gaps and set sizes. Exit codes: 0 no high findings, 1 high findings, 2 usage error.

bash
python scripts/near_duplicates.py train.csv test.csv --drop id --decimals 3
python scripts/near_duplicates.py train.csv test.csv --text-col body --threshold 0.8 --shingle 5
python scripts/near_duplicates.py data.csv --text-col title --threshold 0.9          # within one file

Fuzzy row keys (numerics rounded to --decimals significant digits, normalised text) and MinHash/LSH text similarity (--perms, --shingle, --threshold); counts of affected test rows and example pairs with line numbers. Exit codes: 0 none found, 1 near-duplicates found, 2 usage error.

Additional Resources

FileContent
split-design.mdRandom, stratified, grouped, time and OOD splits; sizes; train/validation/test roles; holdout protection
duplicates-and-entity-leakage.mdExact and near-duplicate detection (hashing, MinHash, embeddings, perceptual hashes), entity resolution, building group ids
drift-and-adversarial-validation.mdPSI, KS, chi-square, Wasserstein, JS divergence; adversarial validation code; handling covariate, prior and concept shift
More in Classical Machine Learning

anomaly-detection-model-builder

Build and evaluate anomaly detection for tabular, sensor and transaction data with few or no labels: define the anomaly type (point, contextual, collective), choose detectors (robust z-score/MAD and IQR rules, Mahalanobis with robust covariance, k-NN distance, LOF, Isolation Forest, one-class SVM, PCA reconstruction / autoencoders, forecasting residuals for time series), combine them by rank, explain each alert by its deviating features, set thresholds from an alert budget or the score distribution, and evaluate with precision@k, PR-AUC and recall on known incidents, or detector agreement when no labels exist. Includes pure-Python detectors (robust z, Mahalanobis, kNN, Isolation Forest, ensemble) and an evaluation script.

catboost-model-builder

Build CatBoost models that use its strengths correctly: raw categorical columns as cat_features (ordered target statistics, no manual encoding), text_features, Pool construction with NaN-safe categories, ordered boosting and has_time for temporal data, symmetric (oblivious) trees, overfitting detector with eval_set and use_best_model, losses for classification, regression (Quantile, Poisson, Tweedie) and ranking (YetiRank), class weights, monotone constraints, feature importance (PredictionValuesChange, LossFunctionChange) and SHAP values, GPU training and model export. Plans cat/text/numeric columns from a CSV and generates a runnable training script, and checks parameter sets for the conflicts CatBoost rejects (iterations vs n_estimators, Bayesian bootstrap with subsample, multiple class-weight options, depth > 16, use_best_model without eval_set).

categorical-encoding-selector

Choose and implement the right encoding for every categorical column given the model family and the data: one-hot (with infrequent-level grouping and handle_unknown), ordinal with an explicit order, cross-fitted target encoding (scikit-learn TargetEncoder, CatBoost ordered statistics, category_encoders inside CV), frequency/count encoding, feature hashing, native categorical support in LightGBM, HistGradientBoosting, XGBoost and CatBoost, prefix levels for codes (zip3, NAICS-2), and embeddings. Measures cardinality, rare-level share, unseen-level rate on a time-ordered holdout and target signal per column, emits the ColumnTransformer, and quantifies target-encoding leakage (in-sample vs out-of-fold vs holdout AUC).

classification-model-selector

Choose the right classification algorithm for tabular, text or mixed data and prove the choice with a fair bake-off: logistic regression (L2, L1, elastic net), linear SVM, naive Bayes, k-NN, RBF SVM, decision trees, random forest, extra trees, HistGradientBoosting, LightGBM, XGBoost, CatBoost, explainable boosting machines (EBM/GAM), points scorecards and MLPs, matched to data size, width, categorical and high-cardinality share, missingness, class imbalance, text, and constraints such as interpretability, monotonic constraints, calibrated probabilities, latency budgets and tiny deployable models. Ranks candidates with starting hyperparameters and cautions, then generates a bake-off script running every candidate on identical CV folds with per-fold metrics, fit time and prediction latency.

clustering-algorithm-selector

Choose the right clustering algorithm and check that clustering makes sense at all: cluster tendency (Hopkins statistic with a correlated-features caveat, nearest-neighbour isolation), then k-means / MiniBatchKMeans, Gaussian mixtures with BIC, agglomerative (Ward, average, complete, single) with dendrograms, DBSCAN, HDBSCAN, OPTICS, spectral clustering, BIRCH, mean shift, k-prototypes and k-modes for mixed or categorical data, and Gower distance, matched to data size, expected shapes, noise, varying density, known k, soft memberships, hierarchy and streaming. Covers preprocessing (scaling, encoding, dimensionality, distance metrics), validation (internal indices, bootstrap stability, external labels, business profiling) and assigning new points.

collaborative-filtering-implementer

Implement collaborative filtering recommenders that beat popularity: item-based kNN with cosine, Jaccard or asymmetric similarity, shrinkage, neighbourhood size, positive-only or mean-centred rating signals and 'because you liked X' explanations; implicit ALS with confidence weighting, BPR pairwise ranking and the closed-form EASE model in plain NumPy, tuned on a temporal validation split by recall@k and NDCG@k and refit on all data; plus user-kNN, explicit matrix factorisation, the implicit/LightFM libraries, hyperparameter ranges, cold-start fallbacks, scaling with sparse matrices and approximate nearest neighbours. Produces top-k lists in the format of the recommendation evaluator.