Skip to content
v0.1.3latest release121 skills4 categories6 sub-categories247 tested scripts397 reference docsContact us
Vector Agentic Skills
Submit a skill

xgboost-lightgbm-tuner

Tune XGBoost and LightGBM models efficiently and read their training curves: validation design and early stopping inside CV, fixed learning rate for search then lower rate with more trees for the final fit, capacity knobs (num_leaves, max_depth, min_child_samples, min_child_weight) sized to the data, subsampling, L1/L2 and split-gain regularisation, categorical handling, imbalance (scale_pos_weight vs is_unbalance vs no weighting), monotone and interaction constraints, Optuna/TPE search with fold-level pruning and parameter importances, and the LightGBM <-> XGBoost <-> CatBoost parameter map. Parses LightGBM, XGBoost and CatBoost logs or CSV histories to find the best iteration, overfitting onset, train/validation gap and whether more rounds or a lower learning rate are needed. Use when someone tunes a gradient-boosted model, asks which hyperparameters matter, sees overfitting or underfitting in boosting curves, hits XGBoost 2 / LightGBM 4 API changes, or wants a search space for a GBM.

AI & Machine Learning / Classical Machine Learningv1.0.0Updated OCT 25 20264 references2 scripts

Gradient boosting rewards a disciplined order: honest validation -> early stopping -> capacity -> sampling -> regularisation -> lower learning rate for the final fit. Most of the gain comes from the first three; the last few percent rarely justify a huge search. scripts/gbm_param_advisor.py produces starting parameters, a search space scaled to the data (rows, features, positives) and an early-stopping plan for LightGBM or XGBoost, plus the cross-library parameter map; scripts/parse_gbm_log.py reads training logs from LightGBM, XGBoost or CatBoost (or a CSV history) and says whether you are overfitting, under-trained or using too high a learning rate.

Trigger Conditions

Activate this skill when the user:

  • Tunes XGBoost or LightGBM, or asks which hyperparameters matter and what ranges to search
  • Shows a training log or learning curve and asks whether the model is overfitting
  • Has a GBM that peaks after a handful of rounds, never stops improving, or has a big train/valid gap
  • Needs imbalance handling, monotone constraints or categorical settings for a GBM
  • Migrates between LightGBM, XGBoost and CatBoost and needs equivalent parameters
  • Hits errors from API changes (XGBoost 2+ early_stopping_rounds in the constructor; LightGBM callbacks)

CatBoost-specific work: catboost-model-builder; general search planning: hyperparameter-search-planner; full Optuna study scripts: optuna-study-writer; diagnosing curves beyond boosting: overfitting-underfitting-diagnoser.

Workflow

text
1. VALIDATE  -> folds that mirror deployment; fix them for every trial
2. ADVISE    -> python scripts/gbm_param_advisor.py --library lightgbm --rows N --features P --positives K
3. BASELINE  -> defaults + early stopping; log train and valid metrics
4. READ      -> python scripts/parse_gbm_log.py train.log   (best round, gap, still improving?)
5. SEARCH    -> 30-200 TPE/random trials on capacity, sampling, regularisation at a fixed learning rate
6. NARROW    -> parameter importances; widen ranges whose best sits at a bound; fix unimportant ones
7. FINAL     -> lower learning rate, more trees (early stopping again or scaled best_iteration), all training data
8. CONFIRM   -> holdout vs the untuned baseline; keep the gain only if it is beyond noise

What Matters Most

RankKnobLightGBMXGBoostRule
1trees x learning raten_estimators + early stopping, learning_ratesamesearch at 0.05-0.1; final at half
2tree capacitynum_leaves (7-255, log)max_depth (3-10)larger for big data / many interactions
3minimum leaf sizemin_child_samples (~0.1-1% of rows)min_child_weightthe regulariser that matters on small or noisy data
4row samplingsubsample + subsample_freq=1subsample0.6-0.9
5column samplingcolsample_bytreecolsample_bytree0.3-0.9; lower with many correlated features
6L2 / L1reg_lambda, reg_alphareg_lambda, reg_alphalog ranges; second-order
7split gainmin_split_gaingammaloss-scale dependent

LightGBM grows leaf-wise: num_leaves controls capacity, max_depth=-1 by default. XGBoost grows depth-wise: max_depth controls capacity (grow_policy="lossguide" + max_leaves mimics LightGBM). gbm_param_advisor.py --map prints the full LightGBM / XGBoost / CatBoost name map.

Reading Training Curves

text
valid_1 binary_logloss (lower is better), 30 logged rounds
best 0.13503 at iteration 10; last 0.27694 at 300; train/valid gap at best 0.1227
- best at iteration 10: the learning rate is high for this data; lower it (x0.3-0.5)
- validation got 100% of its range worse after the best round: early stopping is essential
- large train/valid gap at the best round: increase min_child_samples, lower num_leaves, raise reg_lambda
CurveMeaningAction
Valid best in the first ~10-50 rounds, then riseslearning rate too high and/or capacity too highlower LR, raise min leaf size, lower leaves/depth
Valid still improving at the last roundunder-trainedmore rounds or higher LR; early stopping with patience
Large train/valid gap, valid flatoverfitting capacitystronger leaf-size limits, subsampling, L2
Train ≈ valid and both mediocreunder-fitting / weak featuresmore capacity, features, interactions
Valid noisy round to roundsmall validation setlarger fold, repeated CV, patience ≥ 100

Early Stopping Done Right

  • Early stopping chooses the number of trees on the validation fold; reporting that fold's score is mildly optimistic. For a strict number use an inner validation split or confirm on a holdout.
  • XGBoost >= 2: early_stopping_rounds and eval_metric go in the constructor.
  • LightGBM >= 4: callbacks=[lgb.early_stopping(200), lgb.log_evaluation(100)]; early_stopping_rounds in fit was removed; LightGBM >= 4.7 prefers eval_X/eval_y to eval_set in the scikit-learn API.
  • Never early-stop on the test set (target-leakage-detector flags it as LK009).
python
import lightgbm as lgb
m = lgb.LGBMClassifier(learning_rate=0.05, n_estimators=10000, num_leaves=31, min_child_samples=50,
                       subsample=0.8, subsample_freq=1, colsample_bytree=0.8, verbose=-1, random_state=0)
m.fit(X_tr, y_tr, eval_set=[(X_tr, y_tr), (X_va, y_va)], eval_metric="auc",
      callbacks=[lgb.early_stopping(200, verbose=False), lgb.log_evaluation(50)])

Playbook with ranges for small and large data: tuning-playbook.md.

Imbalance in GBMs

OptionEffectUse when
No weighting (default)calibrated probabilities; ranking unaffectedyou evaluate with PR-AUC / top-K and use probabilities
scale_pos_weight = neg/pos (or is_unbalance=True in LightGBM)more weight on positives; probabilities inflatedrecall at a fixed default threshold matters; recalibrate if probabilities are used
Larger min_child_samples + max_delta_step (XGBoost)stabilises leaf values with very rare positives< 1% positives

Ranking metrics (ROC-AUC, PR-AUC) usually change little with weighting. See imbalanced-class-handler.

Library Notes

  • LightGBM parameters, native API, categoricals, DART/linear trees, common warnings: lightgbm-reference.md.
  • XGBoost 2/3 API, DMatrix, categoricals, constraints, multi-quantile, GPU: xgboost-reference.md.
  • Optuna objective with fixed folds, fold-level pruning, parameter importances and the final refit (verified with Optuna 5.0 and LightGBM 4.7): optuna-for-gbms.md.

Output Format

When tuning or reviewing a GBM, deliver:

  1. Validation design - folds, metric, early-stopping placement.
  2. Starting point - parameters and search space (gbm_param_advisor.py), with the reason for each range.
  3. Curve reading - parse_gbm_log.py output for the baseline and the tuned model.
  4. Search results - best parameters, parameter importances, trials used, best iteration.
  5. Final model - learning rate and tree count for the refit, holdout score vs untuned baseline (with CI).
  6. Next lever - if tuning gained little: features, labels or validation, not more trials.

Scripts

bash
python scripts/gbm_param_advisor.py --library lightgbm --rows 200000 --features 80 --task binary --positives 4000
python scripts/gbm_param_advisor.py --library xgboost --rows 3000 --features 25 --task regression --budget low --json
python scripts/gbm_param_advisor.py --map

Starting parameters (learning rate by data size, leaf size ~0.2% of rows clamped and capped by the positive count, leaves/depth by size, feature sampling by width, early-stopping patience), a search space with types and log scales, imbalance advice with the scale_pos_weight value, a trial budget and a 5-step plan; --map prints the LightGBM / XGBoost / CatBoost parameter equivalences. Exit codes: 0 printed, 2 usage error.

bash
python scripts/parse_gbm_log.py train.log                 # LightGBM / XGBoost / CatBoost text logs
python scripts/parse_gbm_log.py history.csv --json        # iteration + train/valid metric columns
python scripts/parse_gbm_log.py xgb.log --train-name validation_0
python train.py 2>&1 | python scripts/parse_gbm_log.py -

Detects the format, infers metric direction (AUC/NDCG/MAP/accuracy higher; loss/error/RMSE lower; --higher-better overrides), treats validation_0/valid_0 as the training curve when it scores better than _1, and reports the best iteration and value, rounds after best, the train/valid gap and degradation, with advice (more rounds, lower learning rate, early-stopping patience, regularisation, more capacity). Exit codes: 0 parsed, 2 nothing parseable.

Additional Resources

FileContent
tuning-playbook.mdTuning order, ranges, learning-rate/trees trade-off, early stopping inside CV, small- and large-data settings, when tuning cannot help
lightgbm-reference.mdscikit-learn and native APIs, key parameters, categoricals, objectives and metrics, common problems
xgboost-reference.mdXGBoost 2/3 API, DMatrix, parameters, categoricals, objectives, GPU, common errors
optuna-for-gbms.mdCV objective with fixed folds and pruning, importances, final refit with a lower learning rate
More in Classical Machine Learning

anomaly-detection-model-builder

Build and evaluate anomaly detection for tabular, sensor and transaction data with few or no labels: define the anomaly type (point, contextual, collective), choose detectors (robust z-score/MAD and IQR rules, Mahalanobis with robust covariance, k-NN distance, LOF, Isolation Forest, one-class SVM, PCA reconstruction / autoencoders, forecasting residuals for time series), combine them by rank, explain each alert by its deviating features, set thresholds from an alert budget or the score distribution, and evaluate with precision@k, PR-AUC and recall on known incidents, or detector agreement when no labels exist. Includes pure-Python detectors (robust z, Mahalanobis, kNN, Isolation Forest, ensemble) and an evaluation script.

catboost-model-builder

Build CatBoost models that use its strengths correctly: raw categorical columns as cat_features (ordered target statistics, no manual encoding), text_features, Pool construction with NaN-safe categories, ordered boosting and has_time for temporal data, symmetric (oblivious) trees, overfitting detector with eval_set and use_best_model, losses for classification, regression (Quantile, Poisson, Tweedie) and ranking (YetiRank), class weights, monotone constraints, feature importance (PredictionValuesChange, LossFunctionChange) and SHAP values, GPU training and model export. Plans cat/text/numeric columns from a CSV and generates a runnable training script, and checks parameter sets for the conflicts CatBoost rejects (iterations vs n_estimators, Bayesian bootstrap with subsample, multiple class-weight options, depth > 16, use_best_model without eval_set).

categorical-encoding-selector

Choose and implement the right encoding for every categorical column given the model family and the data: one-hot (with infrequent-level grouping and handle_unknown), ordinal with an explicit order, cross-fitted target encoding (scikit-learn TargetEncoder, CatBoost ordered statistics, category_encoders inside CV), frequency/count encoding, feature hashing, native categorical support in LightGBM, HistGradientBoosting, XGBoost and CatBoost, prefix levels for codes (zip3, NAICS-2), and embeddings. Measures cardinality, rare-level share, unseen-level rate on a time-ordered holdout and target signal per column, emits the ColumnTransformer, and quantifies target-encoding leakage (in-sample vs out-of-fold vs holdout AUC).

classification-model-selector

Choose the right classification algorithm for tabular, text or mixed data and prove the choice with a fair bake-off: logistic regression (L2, L1, elastic net), linear SVM, naive Bayes, k-NN, RBF SVM, decision trees, random forest, extra trees, HistGradientBoosting, LightGBM, XGBoost, CatBoost, explainable boosting machines (EBM/GAM), points scorecards and MLPs, matched to data size, width, categorical and high-cardinality share, missingness, class imbalance, text, and constraints such as interpretability, monotonic constraints, calibrated probabilities, latency budgets and tiny deployable models. Ranks candidates with starting hyperparameters and cautions, then generates a bake-off script running every candidate on identical CV folds with per-fold metrics, fit time and prediction latency.

clustering-algorithm-selector

Choose the right clustering algorithm and check that clustering makes sense at all: cluster tendency (Hopkins statistic with a correlated-features caveat, nearest-neighbour isolation), then k-means / MiniBatchKMeans, Gaussian mixtures with BIC, agglomerative (Ward, average, complete, single) with dendrograms, DBSCAN, HDBSCAN, OPTICS, spectral clustering, BIRCH, mean shift, k-prototypes and k-modes for mixed or categorical data, and Gower distance, matched to data size, expected shapes, noise, varying density, known k, soft memberships, hierarchy and streaming. Covers preprocessing (scaling, encoding, dimensionality, distance metrics), validation (internal indices, bootstrap stability, external labels, business profiling) and assigning new points.

collaborative-filtering-implementer

Implement collaborative filtering recommenders that beat popularity: item-based kNN with cosine, Jaccard or asymmetric similarity, shrinkage, neighbourhood size, positive-only or mean-centred rating signals and 'because you liked X' explanations; implicit ALS with confidence weighting, BPR pairwise ranking and the closed-form EASE model in plain NumPy, tuned on a temporal validation split by recall@k and NDCG@k and refit on all data; plus user-kNN, explicit matrix factorisation, the implicit/LightFM libraries, hyperparameter ranges, cold-start fallbacks, scaling with sparse matrices and approximate nearest neighbours. Produces top-k lists in the format of the recommendation evaluator.