Skip to content
v0.1.3latest release121 skills4 categories6 sub-categories247 tested scripts397 reference docsContact us
Vector Agentic Skills
Submit a skill

ml-baseline-model-builder

Build the baselines every ML model must beat before anyone trusts it: dummy majority/prior/mean/median predictors, the current business rule, a one-feature model (OneR), seasonal-naive and last-value forecasts, popularity recommenders, and a first honest linear model and HistGradientBoosting inside a leak-free scikit-learn Pipeline with a time-ordered, grouped or stratified holdout. Reports ROC-AUC, PR-AUC, log loss, Brier, balanced accuracy, MAE, RMSE, R2 and WAPE side by side, and warns when a first baseline looks too good (leakage). Use when starting a modelling project, asked for a baseline or a quick first model, asked whether a model is actually better than a simple rule, or when results need a reference point for a comparison report.

AI & Machine Learning / Classical Machine Learningv1.0.0Updated OCT 25 20263 references2 scripts

A model's score means nothing without a reference point. Baselines answer three questions before any tuning: Is there anything to learn (dummy), is ML better than what we do today (the business rule, one-feature model), and how much does a real model add (linear model and gradient boosting, untuned, in a leak-free pipeline). scripts/baseline_scores.py scores dummy, rule and one-feature baselines on an honest holdout with no dependencies; scripts/scaffold_baseline.py writes a runnable scikit-learn script with the right split, preprocessing and metrics for the dataset's columns.

Trigger Conditions

Activate this skill when the user:

  • Starts a new tabular ML project and asks for a baseline, a first model or "something quick"
  • Asks "is my model actually good?", "is 0.82 AUC good?", "does ML beat our rule?"
  • Needs the reference rows for a model comparison report or a go/no-go decision
  • Has a forecasting, ranking or recommendation problem and needs the naive reference (last value, seasonal naive, popularity)
  • Got a suspiciously strong first result and wants a sanity check against trivial predictors

Statistical comparison of candidate models goes to ml-model-comparison-reporter; full pipeline design to scikit-learn-pipeline-builder; choosing the metric to ml-success-metric-selector.

Workflow

text
1. SPLIT      -> the holdout that mirrors deployment: time-ordered > grouped > stratified random
2. TRIVIAL    -> python scripts/baseline_scores.py data.csv --target y --time-col ts --drop id
3. RULE       -> add today's process: --rule "support_calls >= 3 or last_login_days > 30"
4. FIRST ML   -> python scripts/scaffold_baseline.py data.csv --target y --time-col ts -o baseline.py
5. RUN        -> python baseline.py  (dummy vs linear vs HistGradientBoosting, same split)
6. SANITY     -> too good (AUC > 0.97, R2 > 0.98)? leakage. No better than dummy? framing or data.
7. RECORD     -> baseline table with split, metric, numbers and noise estimate; every later model is compared to it

The Baseline Ladder

Report every rung; each answers a different question.

RungClassificationRegressionAnswers
0. Constantmajority class; prior probabilitymean; median (MAE)the floor; any metric at this level means no learning
1. Randomstratified random labelspermuted targetwhat chance looks like for rank metrics
2. Current processbusiness rule, human queue order, vendor scorelast year's value, planner's estimatedoes ML beat today?
3. One featureOneR: best single column, binnedper-bin mean of best columnis the signal just one obvious column?
4. Simple modellogistic regression, one-hot + scalingridge / linear on log targetthe honest ML baseline
5. Default GBMHistGradientBoosting, untunedHistGradientBoostingwhat non-linearity adds before tuning

Tuned and fancy models are compared against rung 5 and rung 2; the business only cares about rung 2.

Baselines for Other Tasks

TaskBaselineNotes
Forecastingnaive (last value), seasonal naive (same day last week/year), moving average, driftMASE uses seasonal naive as its denominator; a model with MASE > 1 loses to it
Intermittent demandzero forecast, Croston/SBA, mean of non-zero x ratemany series are best left at 0
Ranking / searchBM25 or current ranking; popularity; randomreport NDCG@k per query
Recommendationmost popular overall, most popular in the user's segment, recently viewedpopularity is notoriously hard to beat on accuracy metrics
Anomaly detectionrobust z-score / IQR per feature, static thresholdsanalysts' current alerts are rung 2
SurvivalKaplan-Meier (no covariates)concordance 0.5
Uplifttreat everyone, treat nobody, random targeting, response modelQini of random = 0
Multiclassmajority class; class priors; keyword rulesmacro-F1 of majority is tiny
Clusteringrandom assignment with the same sizes; one clustersilhouette and stability vs random

Code for each: baseline-catalog.md.

Split First, Then Baseline

The baseline is only comparable if every model uses the same holdout:

DataSplitWhy
Model predicts the future (churn, demand, fraud)time-ordered: train on older rows, test on newestrandom splits leak future patterns and overstate scores
Several rows per entity (customer snapshots, patient visits)grouped: an entity is only in train or testthe model memorises entities otherwise
i.i.d. rows, classificationstratified randomkeeps the class rate stable
Small data (< ~2k rows)repeated stratified K-foldone split is too noisy to compare baselines

Both scripts take --time-col and the scaffold also --group-col. Strategy detail: cross-validation-strategy-designer.

Reading the Baseline Table

text
baseline       accuracy balanced_acc  roc_auc   pr_auc  log_loss   brier  detail
majority         0.8300       0.5000   0.5000   0.1700    5.8716  0.1700  always not 1
prior            0.8300       0.5000   0.5000   0.1700    0.4627  0.1433  p = 0.217 for every row
one_r            0.8300       0.5493   0.6311   0.2856    0.4379  0.1360  positive rate by 10 bins of last_login_days
rule             0.8500       0.7225   0.7225   0.3778    5.1808  0.1500  support_calls >= 3 or last_login_days > 30
  • Accuracy 0.83 from "always no": with 17% positives accuracy is useless; use PR-AUC, recall@capacity or balanced accuracy.
  • PR-AUC of a constant = prevalence (0.17): that is the PR-AUC floor, not 0.
  • Log loss of the prior (0.46) is the floor for probabilistic models; a model above it is worse than predicting the base rate.
  • The rule beats one feature here: the business rule is the bar ML must clear, with margin.
  • A hard 0/1 rule gets terrible log loss; compare rules to models on ranking or threshold metrics, not log loss.

When the Baseline Is Already Hard to Beat

SituationWhat it meansDo
Linear ≈ GBM ≈ rulesignal is simple, or features are weakship the rule or the linear model; invest in features
GBM >> linearinteractions / non-linearity mattertune the GBM; check it is not exploiting a leak
Everything ≈ dummyno learnable signal with these featuresrevisit framing and data sources (feasibility)
First model AUC > 0.97almost certainly leakagerun target-leakage-detector
Big gap between CV and holdoutsplit mismatch or driftuse a time-based holdout everywhere

Making the Comparison Fair

  1. Same rows, same split, same metric, same preprocessing budget for every baseline.
  2. Preprocessing inside a Pipeline so scalers, imputers and encoders are fit on training rows only.
  3. Threshold-dependent metrics at a stated threshold (train prevalence or capacity), not silently 0.5.
  4. Noise estimate: repeated splits or bootstrap CI; a 0.005 AUC gain over the baseline is often noise.
  5. Rules are scored as-is (0/1); give them the same population and holdout.

Reporting templates and the noise estimate: baseline-reporting.md.

Output Format

When building baselines, answer with:

  1. Split used and why (time, group, stratified), with train/test sizes and class rates.
  2. Baseline table - every rung with the primary metric first, then secondary metrics.
  3. Reading - which baseline is the bar, what the gap between rungs says, any leakage warning.
  4. First-model script - the scaffold_baseline.py output (or equivalent code) with the same split.
  5. Next step - feature work, tuning, or stop, based on the gaps.

Scripts

bash
python scripts/baseline_scores.py data.csv --target churned --time-col signup_date --drop customer_id
python scripts/baseline_scores.py data.csv --target churned --rule "support_calls >= 3 or plan == basic"
python scripts/baseline_scores.py data.csv --target price --drop house_id --test-frac 0.2 --json

Binary targets: majority, prior, one-feature (chosen on an inner split of the training rows, never on the test set; smoothed positive rate per level or per 10 quantile bins) and an optional rule (col op value terms with and / or, ops >= <= > < == !=). Metrics: accuracy at 0.5, balanced accuracy at the training rate, ROC-AUC, tie-aware PR-AUC, log loss, Brier. Numeric targets: mean, median and one-feature, with MAE, RMSE, R2, WAPE. Holdout: latest rows by --time-col, else seeded random. Exit codes: 0 scored, 2 usage error or unusable target.

bash
python scripts/scaffold_baseline.py data.csv --target churned --time-col signup_date -o baseline.py
python scripts/scaffold_baseline.py data.csv --target churned --group-col customer_id > baseline.py
python scripts/scaffold_baseline.py data.csv --target price --task regression --drop house_id

Infers numeric, categorical (including numeric-looking codes such as zip), date (expanded to year/month/day of week) and dropped columns (IDs, near-unique text), then writes a pandas + scikit-learn script: honest split, Dummy vs LogisticRegression/Ridge vs HistGradientBoosting, each in a Pipeline with ColumnTransformer (median imputation with missing indicators, scaling for the linear model, one-hot with infrequent-category grouping), a metric table, and a leakage warning when AUC > 0.97 or R2 > 0.98. Exit codes: 0 written, 2 usage error.

Additional Resources

FileContent
baseline-catalog.mdCode for every baseline: dummy, rules, OneR, linear, GBM, forecasting naive/seasonal/Croston, popularity recommenders, ranking, survival, uplift
baseline-reporting.mdBaseline table template, noise estimates (repeated CV, bootstrap), how to state "beats the baseline", common reading mistakes
first-model-recipes.mdLeak-free first models in scikit-learn: preprocessing defaults, time/group splits, log targets, class weights, what not to tune yet
More in Classical Machine Learning

anomaly-detection-model-builder

Build and evaluate anomaly detection for tabular, sensor and transaction data with few or no labels: define the anomaly type (point, contextual, collective), choose detectors (robust z-score/MAD and IQR rules, Mahalanobis with robust covariance, k-NN distance, LOF, Isolation Forest, one-class SVM, PCA reconstruction / autoencoders, forecasting residuals for time series), combine them by rank, explain each alert by its deviating features, set thresholds from an alert budget or the score distribution, and evaluate with precision@k, PR-AUC and recall on known incidents, or detector agreement when no labels exist. Includes pure-Python detectors (robust z, Mahalanobis, kNN, Isolation Forest, ensemble) and an evaluation script.

catboost-model-builder

Build CatBoost models that use its strengths correctly: raw categorical columns as cat_features (ordered target statistics, no manual encoding), text_features, Pool construction with NaN-safe categories, ordered boosting and has_time for temporal data, symmetric (oblivious) trees, overfitting detector with eval_set and use_best_model, losses for classification, regression (Quantile, Poisson, Tweedie) and ranking (YetiRank), class weights, monotone constraints, feature importance (PredictionValuesChange, LossFunctionChange) and SHAP values, GPU training and model export. Plans cat/text/numeric columns from a CSV and generates a runnable training script, and checks parameter sets for the conflicts CatBoost rejects (iterations vs n_estimators, Bayesian bootstrap with subsample, multiple class-weight options, depth > 16, use_best_model without eval_set).

categorical-encoding-selector

Choose and implement the right encoding for every categorical column given the model family and the data: one-hot (with infrequent-level grouping and handle_unknown), ordinal with an explicit order, cross-fitted target encoding (scikit-learn TargetEncoder, CatBoost ordered statistics, category_encoders inside CV), frequency/count encoding, feature hashing, native categorical support in LightGBM, HistGradientBoosting, XGBoost and CatBoost, prefix levels for codes (zip3, NAICS-2), and embeddings. Measures cardinality, rare-level share, unseen-level rate on a time-ordered holdout and target signal per column, emits the ColumnTransformer, and quantifies target-encoding leakage (in-sample vs out-of-fold vs holdout AUC).

classification-model-selector

Choose the right classification algorithm for tabular, text or mixed data and prove the choice with a fair bake-off: logistic regression (L2, L1, elastic net), linear SVM, naive Bayes, k-NN, RBF SVM, decision trees, random forest, extra trees, HistGradientBoosting, LightGBM, XGBoost, CatBoost, explainable boosting machines (EBM/GAM), points scorecards and MLPs, matched to data size, width, categorical and high-cardinality share, missingness, class imbalance, text, and constraints such as interpretability, monotonic constraints, calibrated probabilities, latency budgets and tiny deployable models. Ranks candidates with starting hyperparameters and cautions, then generates a bake-off script running every candidate on identical CV folds with per-fold metrics, fit time and prediction latency.

clustering-algorithm-selector

Choose the right clustering algorithm and check that clustering makes sense at all: cluster tendency (Hopkins statistic with a correlated-features caveat, nearest-neighbour isolation), then k-means / MiniBatchKMeans, Gaussian mixtures with BIC, agglomerative (Ward, average, complete, single) with dendrograms, DBSCAN, HDBSCAN, OPTICS, spectral clustering, BIRCH, mean shift, k-prototypes and k-modes for mixed or categorical data, and Gower distance, matched to data size, expected shapes, noise, varying density, known k, soft memberships, hierarchy and streaming. Covers preprocessing (scaling, encoding, dimensionality, distance metrics), validation (internal indices, bootstrap stability, external labels, business profiling) and assigning new points.

collaborative-filtering-implementer

Implement collaborative filtering recommenders that beat popularity: item-based kNN with cosine, Jaccard or asymmetric similarity, shrinkage, neighbourhood size, positive-only or mean-centred rating signals and 'because you liked X' explanations; implicit ALS with confidence weighting, BPR pairwise ranking and the closed-form EASE model in plain NumPy, tuned on a temporal validation split by recall@k and NDCG@k and refit on all data; plus user-kNN, explicit matrix factorisation, the implicit/LightFM libraries, hyperparameter ranges, cold-start fallbacks, scaling with sparse matrices and approximate nearest neighbours. Produces top-k lists in the format of the recommendation evaluator.