Skip to content
v0.1.3latest release121 skills4 categories6 sub-categories247 tested scripts397 reference docsContact us
Vector Agentic Skills
Submit a skill

overfitting-underfitting-diagnoser

Diagnose overfitting and underfitting from the evidence a model leaves behind: training histories (metric per boosting round or epoch from LightGBM, XGBoost, CatBoost, Keras or PyTorch) and validation curves (train and validation score across one hyperparameter such as max_depth, C, min_samples_leaf or alpha). Finds the best iteration on a smoothed curve, the overfitting onset, degradation and noise, and suggests early-stopping patience; flags under-training, plateaus, divergence and models that are not learning. For validation curves it labels underfit, sweet-spot and overfit zones, applies the one-standard-error rule to pick the simplest adequate value, infers complexity direction, and warns when the best value sits at the edge of the sweep. Includes the fix for each diagnosis per model family. Use when someone asks whether a model overfits or underfits, why validation loss rises, how many rounds or epochs to train, or how to read a train/validation gap.

AI & Machine Learning / Classical Machine Learningv1.0.0Updated OCT 25 20263 references2 scripts

"Is it overfitting?" deserves a measured answer, not a guess from one train/test pair. Two curves answer it: the training history (score per boosting round or epoch) shows when the model starts memorising, and the validation curve (score across one hyperparameter) shows how much flexibility the data supports. scripts/training_history.py reads the first and scripts/validation_curve_report.py the second. Both return a verdict and the concrete setting to change.

Trigger Conditions

Activate this skill when the user:

  • Asks whether a model overfits or underfits, or why training is much better than validation
  • Shows a loss curve where validation rises, flattens or never improves
  • Needs the number of boosting rounds / epochs, or an early-stopping patience
  • Sweeps a hyperparameter (max_depth, C, alpha, min_samples_leaf, k) and asks which value to use
  • Sees a model that was great in development and worse in production (also check leakage and drift)

Curves over training-set size: learning-curve-diagnoser. Multi-parameter search results: hyperparameter-search-planner. Too-good scores: target-leakage-detector.

Workflow

text
1. EVIDENCE   -> export the history (evals_result_, history.history) and/or a validation curve with fold scores
2. HISTORY    -> python scripts/training_history.py history.csv
3. SWEEP      -> python scripts/validation_curve_report.py validation_curve.csv --current <value in use>
4. CROSS-CHECK-> gap size vs fold noise; same splitter as model selection; leakage if validation is too good
5. FIX        -> one change from the tables below; re-run the same curve
6. REPORT     -> verdict, the numbers behind it, the change, the before/after curve

Diagnosis Table

EvidenceDiagnosisTypical fix
Validation loss bottoms out, then rises while training keeps fallingOVERFITTING (after the best iteration)early stopping at the best round; regularise if the gap is already large there
Best validation at the last roundSTILL IMPROVING (under-trained)more rounds / epochs, or a higher learning rate
Validation flat for the last thirdPLATEAUstop earlier; gains need features or capacity
Training loss risesTRAINING DIVERGESlower learning rate; check custom objectives, exploding values
Validation never beats its first valueNOT LEARNINGdata/label bug, wrong metric, learning rate far off
Validation curve: score falls as complexity grows past the best value, gap widensoverfit zonemove to the 1-SE pick
Validation curve: both train and validation low at low complexityunderfit zonemore flexible setting
Best value at the edge of the sweeprange too narrowextend the sweep beyond that edge

Thresholds are relative to the curve's own noise (std of the metric around its moving average), not to fixed numbers. Detail on reading both curves: reading-the-curves.md.

Key Rules

  • A gap alone is not overfitting. Boosted trees and forests always score higher on training rows. Overfitting means validation gets worse as flexibility grows, or that a simpler setting validates as well.
  • Decide on validation folds, confirm on a holdout. The best iteration chosen on the validation set is optimistic for that set.
  • Smooth before picking the best round. A single lucky round should not decide; the script uses a centred moving average (--smooth) and sets patience to max(10, 3 x window, 5% of the rounds).
  • Prefer the 1-SE pick. The least complex value within one standard error of the best generalises at least as well and is cheaper.
  • Know which direction is "more complex". Larger max_depth / num_leaves / C means more flexible; larger min_samples_leaf / alpha / reg_lambda / n_neighbors means less flexible. XGBoost gamma is a regulariser, but SVM gamma adds flexibility, so the script infers it from the training-score trend.
  • Check the evaluation before the model. Leakage, duplicates across folds or a non-grouped split make validation too good. A distribution shift makes production worse than validation, and no regulariser fixes either.

Fixes per model family and how to confirm them: remedies.md. Exporting histories and validation curves from each library: exporting-curves.md.

Output Format

When diagnosing over- or underfitting, deliver:

  1. Evidence - which curves, metric, splitter, folds.
  2. Verdict - the diagnosis with best iteration/value, onset, gap and noise (history) or zones, best and 1-SE pick (validation curve).
  3. Change - the setting to change (rounds, patience, parameter value) and why.
  4. Confirmation - the curve after the change and the holdout score against the previous model.

Scripts

bash
python scripts/training_history.py history.csv
python scripts/training_history.py history.csv --train-col training_auc --val-col valid_1_auc --smooth 9
python scripts/training_history.py history.csv --json

Detects train/validation column pairs (train_*/training_* with valid_*/val_*/eval_*/test_*, including LightGBM's valid_1_*) and the metric direction from its name (override with --lower-better / --higher-better). Reports best iteration, smoothed best, degradation, onset, gap at start/best/end, noise, suggested patience and the verdict. Exit codes: 0 diagnosed, 2 usage or read error.

bash
python scripts/validation_curve_report.py validation_curve.csv
python scripts/validation_curve_report.py vc.csv --param-col param_C --current 100
python scripts/validation_curve_report.py vc.csv --complexity decreasing --lower-better --json

Accepts one row per (value, fold) or per value; numeric parameters are sorted, categorical ones keep file order. Reports zones, best, 1-SE pick, boundary warnings and, with --current, which way to move. Exit codes: 0 reported, 2 usage or read error.

Additional Resources

FileContent
reading-the-curves.mdhistory shapes, validation-curve zones, noise and smoothing, the 1-SE rule, misreadings
remedies.mdregularisation and capacity levers per model family, early stopping set-ups, confirmation
exporting-curves.mdgetting histories from LightGBM, XGBoost, CatBoost, Keras, PyTorch; validation_curve with fold scores
More in Classical Machine Learning

anomaly-detection-model-builder

Build and evaluate anomaly detection for tabular, sensor and transaction data with few or no labels: define the anomaly type (point, contextual, collective), choose detectors (robust z-score/MAD and IQR rules, Mahalanobis with robust covariance, k-NN distance, LOF, Isolation Forest, one-class SVM, PCA reconstruction / autoencoders, forecasting residuals for time series), combine them by rank, explain each alert by its deviating features, set thresholds from an alert budget or the score distribution, and evaluate with precision@k, PR-AUC and recall on known incidents, or detector agreement when no labels exist. Includes pure-Python detectors (robust z, Mahalanobis, kNN, Isolation Forest, ensemble) and an evaluation script.

catboost-model-builder

Build CatBoost models that use its strengths correctly: raw categorical columns as cat_features (ordered target statistics, no manual encoding), text_features, Pool construction with NaN-safe categories, ordered boosting and has_time for temporal data, symmetric (oblivious) trees, overfitting detector with eval_set and use_best_model, losses for classification, regression (Quantile, Poisson, Tweedie) and ranking (YetiRank), class weights, monotone constraints, feature importance (PredictionValuesChange, LossFunctionChange) and SHAP values, GPU training and model export. Plans cat/text/numeric columns from a CSV and generates a runnable training script, and checks parameter sets for the conflicts CatBoost rejects (iterations vs n_estimators, Bayesian bootstrap with subsample, multiple class-weight options, depth > 16, use_best_model without eval_set).

categorical-encoding-selector

Choose and implement the right encoding for every categorical column given the model family and the data: one-hot (with infrequent-level grouping and handle_unknown), ordinal with an explicit order, cross-fitted target encoding (scikit-learn TargetEncoder, CatBoost ordered statistics, category_encoders inside CV), frequency/count encoding, feature hashing, native categorical support in LightGBM, HistGradientBoosting, XGBoost and CatBoost, prefix levels for codes (zip3, NAICS-2), and embeddings. Measures cardinality, rare-level share, unseen-level rate on a time-ordered holdout and target signal per column, emits the ColumnTransformer, and quantifies target-encoding leakage (in-sample vs out-of-fold vs holdout AUC).

classification-model-selector

Choose the right classification algorithm for tabular, text or mixed data and prove the choice with a fair bake-off: logistic regression (L2, L1, elastic net), linear SVM, naive Bayes, k-NN, RBF SVM, decision trees, random forest, extra trees, HistGradientBoosting, LightGBM, XGBoost, CatBoost, explainable boosting machines (EBM/GAM), points scorecards and MLPs, matched to data size, width, categorical and high-cardinality share, missingness, class imbalance, text, and constraints such as interpretability, monotonic constraints, calibrated probabilities, latency budgets and tiny deployable models. Ranks candidates with starting hyperparameters and cautions, then generates a bake-off script running every candidate on identical CV folds with per-fold metrics, fit time and prediction latency.

clustering-algorithm-selector

Choose the right clustering algorithm and check that clustering makes sense at all: cluster tendency (Hopkins statistic with a correlated-features caveat, nearest-neighbour isolation), then k-means / MiniBatchKMeans, Gaussian mixtures with BIC, agglomerative (Ward, average, complete, single) with dendrograms, DBSCAN, HDBSCAN, OPTICS, spectral clustering, BIRCH, mean shift, k-prototypes and k-modes for mixed or categorical data, and Gower distance, matched to data size, expected shapes, noise, varying density, known k, soft memberships, hierarchy and streaming. Covers preprocessing (scaling, encoding, dimensionality, distance metrics), validation (internal indices, bootstrap stability, external labels, business profiling) and assigning new points.

collaborative-filtering-implementer

Implement collaborative filtering recommenders that beat popularity: item-based kNN with cosine, Jaccard or asymmetric similarity, shrinkage, neighbourhood size, positive-only or mean-centred rating signals and 'because you liked X' explanations; implicit ALS with confidence weighting, BPR pairwise ranking and the closed-form EASE model in plain NumPy, tuned on a temporal validation split by recall@k and NDCG@k and refit on all data; plus user-kNN, explicit matrix factorisation, the implicit/LightFM libraries, hyperparameter ranges, cold-start fallbacks, scaling with sparse matrices and approximate nearest neighbours. Produces top-k lists in the format of the recommendation evaluator.