Skip to content
v0.1.3latest release121 skills4 categories6 sub-categories247 tested scripts397 reference docsContact us
Vector Agentic Skills
Submit a skill

learning-to-rank-model-builder

Build learning-to-rank models for search, recommendations re-ranking and marketplace sorting: validate query-document data (contiguous query groups, single-label and no-relevant queries, duplicates, query-level features, single-feature baselines and leakage-like features), train LightGBM lambdarank and XGBoost rank:ndcg rankers with grouped cross-validation, early stopping on NDCG@k, label gains and truncation, and evaluate per query with NDCG (exponential or linear gain), MAP, MRR, ERR and precision@k, with tie handling made explicit and paired bootstrap / randomisation tests that tell whether a ranker really beats BM25 or the current system. Covers pointwise, pairwise and listwise losses, judgement and click labels, position bias, feature design and online interleaving. Use when someone builds a ranker, re-ranks search results or recommendation candidates, uses LambdaMART, or compares rankings offline.

AI & Machine Learning / Classical Machine Learningv1.0.0Updated OCT 25 20263 references4 scripts

A ranker orders the documents of each query, so everything is per query: grouping, splitting, losses and metrics. The common failures are silent ones:

  • unsorted rows that mix queries into the wrong groups
  • queries leaking across folds
  • NDCG averaged over queries with no relevant documents
  • tied scores evaluated optimistically
  • a "better" model with no significance test over queries

The scripts check the data, train the rankers and evaluate them per query:

  • scripts/ltr_data_check.py validates the data
  • scripts/gen_ltr_script.py trains LightGBM / XGBoost rankers with grouped CV
  • scripts/rank_eval.py evaluates and compares rankings per query

Trigger Conditions

Activate this skill when the user:

  • Builds or tunes a ranker (LambdaMART, LightGBM LGBMRanker, XGBoost XGBRanker) for search or re-ranking
  • Re-ranks recommendation candidates with features
  • Compares rankings offline (BM25 vs a model, model A vs B) or needs NDCG/MAP/MRR computed correctly
  • Has relevance judgements or click logs and asks how to turn them into training labels

Candidate generation for recommendations: recommendation-system-designer. GBM tuning detail: xgboost-lightgbm-tuner.

Workflow

text
1. CHECK     -> python scripts/ltr_data_check.py data.csv --qid-col qid --label-col label --exclude doc_id
2. BASELINE  -> single-feature NDCG (bm25, current production score) from the check, or rank_eval.py
3. TRAIN     -> python scripts/gen_ltr_script.py --data data.csv --features f1 f2 f3 --models lgbm xgb --k 10 -o ltr.py
4. EVALUATE  -> python ltr.py; python scripts/rank_eval.py oof_scores.csv --scores lgbm xgb --baseline bm25 --k 10
5. DECIDE    -> significant per-query gain over the baseline? Then plan an online interleaving / A/B test

Data Rules

RuleWhyCheck
Rows sorted by query id before building groupsLightGBM group= and XGBoost qid= assume contiguous blocksBLOCKING in ltr_data_check.py
Split by query (GroupKFold)documents of one query in train and test leakthe generated script
Single-label queries out of trainingno pairs, so no gradient for pairwise/listwise lossesWARNING, excluded
Queries with no relevant document reported separatelyNDCG is undefined (LightGBM counts them as 1)rank_eval.py --empty
Single-feature baselines firstmany rankers barely beat BM25the check's feature table
Features that rank almost perfectlylabels leaking into features (clicks derived from judgements)BLOCKING at NDCG >= 0.98
Query-level features (constant in a query)cannot reorder documents on their ownWARNING

Labels from judgements, clicks or purchases, position bias and feature design: ltr-data-and-labels.md.

Models and Losses

ApproachLossLibrary
Pointwiseregression / classification of the labelany GBM
PairwiseRankNet / BPR-style pairsXGBoost rank:pairwise
Listwise (LambdaMART)lambdas weighted by the NDCG changeLightGBM lambdarank, XGBoost rank:ndcg
Neurallistwise softmax / ApproxNDCGTF-Ranking, allRank

LambdaMART (GBDT with lambdarank) is the strong default for tabular features. Set label_gain = 2^label - 1 (LightGBM's default is the same; pass it explicitly when labels exceed 30), truncate the lambdas at the evaluation depth (lambdarank_truncation_level / lambdarank_num_pair_per_sample), and early-stop on NDCG@k of a query-grouped validation split. Parameters and code: ltr-models.md.

Evaluation

MetricUse
NDCG@kgraded relevance, position-discounted; the main metric
MAP@k, P@kbinary relevance (label >= threshold)
MRRnavigational queries: the first relevant result matters
ERR@kcascade user model: stops at the first satisfying result
  • Gain convention: exponential (2^label - 1; LightGBM, XGBoost, TREC) by default. --gain linear matches sklearn.metrics.ndcg_score (verified: 0.9322 on the fixture).
  • Ties: pessimistic by default (a constant score cannot look good). --ties input reproduces LightGBM's own NDCG@k exactly (verified 0.9019/0.9263). With coarse scores the two differed by 0.07 on the fixture.
  • Significance: queries are the unit. --baseline reports the mean per-query difference with a bootstrap CI and a sign-flip randomisation p-value. On the fixture, LightGBM and XGBoost improved NDCG@10 by about 0.02 over BM25 but were not significant (p 0.25-0.35, 40 queries).

Metric formulas, pitfalls and online evaluation: ltr-evaluation.md.

Output Format

When building a ranker, deliver:

  1. Data check - queries, group sizes, labels, blocking issues fixed, single-feature baselines.
  2. Model - loss, features, label gains, CV scheme, best iterations, feature importance.
  3. Evaluation - NDCG@k/MAP/MRR per model and baseline, the gain and tie conventions, excluded queries.
  4. Significance - per-query differences with CI and p-value against the baseline.
  5. Next steps - an online interleaving or A/B test, label improvements, feature ideas.

Scripts

bash
python scripts/ltr_data_check.py ltr.csv --qid-col qid --label-col label --exclude doc_id score
python scripts/ltr_data_check.py train.csv --qid-col query --label-col rel --features bm25 pagerank ctr --k 5 --json

Exit codes: 0 no blocking issue, 1 blocking issue, 2 usage or read error.

bash
python scripts/gen_ltr_script.py --data ltr.csv --qid-col qid --label-col label --features bm25 freshness clicks -o ltr.py
python scripts/gen_ltr_script.py --data train.csv --qid-col query --label-col rel --exclude doc_id --models lgbm xgb --k 5

Writes oof_scores.csv (out-of-fold scores plus raw features for baselines) and ranker_<model>.joblib. Exit codes: 0 written, 2 usage error.

bash
python scripts/rank_eval.py preds.csv --qid-col qid --label-col label --scores lgbm xgb --baseline bm25 --k 5 10
python scripts/rank_eval.py preds.csv --scores model --k 10 --gain linear --ties input --per-query per_query.csv

Exit codes: 0 evaluated, 1 with --baseline when no model is significantly better, 2 usage error. scripts/_rank_metrics.py holds the shared metric functions.

Additional Resources

FileContent
ltr-data-and-labels.mdjudgements, click models and position bias, sampling, feature families
ltr-models.mdpointwise/pairwise/listwise, LambdaMART, LightGBM and XGBoost parameters, neural rankers
ltr-evaluation.mdmetric definitions, gain and tie conventions, significance, interleaving and A/B tests
More in Classical Machine Learning

anomaly-detection-model-builder

Build and evaluate anomaly detection for tabular, sensor and transaction data with few or no labels: define the anomaly type (point, contextual, collective), choose detectors (robust z-score/MAD and IQR rules, Mahalanobis with robust covariance, k-NN distance, LOF, Isolation Forest, one-class SVM, PCA reconstruction / autoencoders, forecasting residuals for time series), combine them by rank, explain each alert by its deviating features, set thresholds from an alert budget or the score distribution, and evaluate with precision@k, PR-AUC and recall on known incidents, or detector agreement when no labels exist. Includes pure-Python detectors (robust z, Mahalanobis, kNN, Isolation Forest, ensemble) and an evaluation script.

catboost-model-builder

Build CatBoost models that use its strengths correctly: raw categorical columns as cat_features (ordered target statistics, no manual encoding), text_features, Pool construction with NaN-safe categories, ordered boosting and has_time for temporal data, symmetric (oblivious) trees, overfitting detector with eval_set and use_best_model, losses for classification, regression (Quantile, Poisson, Tweedie) and ranking (YetiRank), class weights, monotone constraints, feature importance (PredictionValuesChange, LossFunctionChange) and SHAP values, GPU training and model export. Plans cat/text/numeric columns from a CSV and generates a runnable training script, and checks parameter sets for the conflicts CatBoost rejects (iterations vs n_estimators, Bayesian bootstrap with subsample, multiple class-weight options, depth > 16, use_best_model without eval_set).

categorical-encoding-selector

Choose and implement the right encoding for every categorical column given the model family and the data: one-hot (with infrequent-level grouping and handle_unknown), ordinal with an explicit order, cross-fitted target encoding (scikit-learn TargetEncoder, CatBoost ordered statistics, category_encoders inside CV), frequency/count encoding, feature hashing, native categorical support in LightGBM, HistGradientBoosting, XGBoost and CatBoost, prefix levels for codes (zip3, NAICS-2), and embeddings. Measures cardinality, rare-level share, unseen-level rate on a time-ordered holdout and target signal per column, emits the ColumnTransformer, and quantifies target-encoding leakage (in-sample vs out-of-fold vs holdout AUC).

classification-model-selector

Choose the right classification algorithm for tabular, text or mixed data and prove the choice with a fair bake-off: logistic regression (L2, L1, elastic net), linear SVM, naive Bayes, k-NN, RBF SVM, decision trees, random forest, extra trees, HistGradientBoosting, LightGBM, XGBoost, CatBoost, explainable boosting machines (EBM/GAM), points scorecards and MLPs, matched to data size, width, categorical and high-cardinality share, missingness, class imbalance, text, and constraints such as interpretability, monotonic constraints, calibrated probabilities, latency budgets and tiny deployable models. Ranks candidates with starting hyperparameters and cautions, then generates a bake-off script running every candidate on identical CV folds with per-fold metrics, fit time and prediction latency.

clustering-algorithm-selector

Choose the right clustering algorithm and check that clustering makes sense at all: cluster tendency (Hopkins statistic with a correlated-features caveat, nearest-neighbour isolation), then k-means / MiniBatchKMeans, Gaussian mixtures with BIC, agglomerative (Ward, average, complete, single) with dendrograms, DBSCAN, HDBSCAN, OPTICS, spectral clustering, BIRCH, mean shift, k-prototypes and k-modes for mixed or categorical data, and Gower distance, matched to data size, expected shapes, noise, varying density, known k, soft memberships, hierarchy and streaming. Covers preprocessing (scaling, encoding, dimensionality, distance metrics), validation (internal indices, bootstrap stability, external labels, business profiling) and assigning new points.

collaborative-filtering-implementer

Implement collaborative filtering recommenders that beat popularity: item-based kNN with cosine, Jaccard or asymmetric similarity, shrinkage, neighbourhood size, positive-only or mean-centred rating signals and 'because you liked X' explanations; implicit ALS with confidence weighting, BPR pairwise ranking and the closed-form EASE model in plain NumPy, tuned on a temporal validation split by recall@k and NDCG@k and refit on all data; plus user-kNN, explicit matrix factorisation, the implicit/LightFM libraries, hyperparameter ranges, cold-start fallbacks, scaling with sparse matrices and approximate nearest neighbours. Produces top-k lists in the format of the recommendation evaluator.