Skip to content
v0.1.3latest release121 skills4 categories6 sub-categories247 tested scripts397 reference docsContact us
Vector Agentic Skills
Submit a skill

model-card-writer

Write model cards that hold up to review: model details, intended and out-of-scope uses, training and evaluation data with periods and label definitions, overall metrics with confidence intervals, the baseline and decision threshold, performance disaggregated by group or segment with bootstrap intervals, fairness gaps (TPR/equal opportunity, FPR, positive rate and the four-fifths ratio), explainability, limitations, ethical considerations, risks with mitigations, and monitoring. Computes slice metrics from a predictions file, generates the card from a JSON spec with visible TODOs for missing sections, and lints existing cards for missing sections, placeholders, metrics without numbers or uncertainty, missing subgroup results and overclaiming language. Follows Mitchell et al. (2019) Model Cards and the Hugging Face card layout. Use when someone documents a model, prepares a model for review or release, or needs a model card, model documentation or a fairness and performance report.

AI & Machine Learning / Classical Machine Learningv1.0.0Updated OCT 25 20263 references3 scripts

A model card is the model's label: what it does, what it was trained and tested on, how well it works (overall and for each group that matters), where it fails, and who to call. Cards fail review for predictable reasons: no out-of-scope uses, metrics without uncertainty, no breakdown by group, "unbiased" claims without evidence, and TODOs left in. scripts/slice_metrics.py computes the disaggregated evaluation, scripts/model_card.py writes the card from a JSON spec, and scripts/card_lint.py reviews any card.

Trigger Conditions

Activate this skill when the user:

  • Asks for a model card, model documentation, or a model release/review document
  • Needs performance by group, segment or demographic, or a fairness section
  • Prepares for model risk review, an audit, or publishing a model (e.g. on Hugging Face)
  • Shares a card or README and asks whether it is complete

Inputs often come from other skills: metrics from ml-success-metric-selector, comparisons from ml-model-comparison-reporter, drivers from feature-importance-reporter and shap-value-explainer.

Workflow

text
1. PREDICT   -> evaluation predictions: y_true, y_score (or y_pred) and the slice columns
2. SLICES    -> python scripts/slice_metrics.py preds.csv --slices group age_band --threshold 0.4 --json > slices.json
3. SPEC      -> write spec.json (schema: references/card-spec.md), with facts from the owners
4. CARD      -> python scripts/model_card.py spec.json --slices slices.json -o MODEL_CARD.md --strict
5. LINT      -> python scripts/card_lint.py MODEL_CARD.md
6. REVIEW    -> owners confirm intended use, limitations and risks; version the card with the model

What a Card Must Contain

SectionContentCommon failure
Model detailswhat it predicts, type, version, date, ownersno owner or version
Intended usedecisions supported, usersvague ("predict churn")
Out-of-scopeuses that are not allowedmissing entirely
Datasources, periods, rows, label definition, known gapsno dates; label not defined
Evaluationmetrics on held-out (preferably out-of-time) data with CIs, baseline, thresholdtest-set point estimates only
Performance across groupsthe same metrics per slice with CIs; small slices markedabsent, or only overall
Fairnessattributes assessed, gaps, the decision on them"the model is fair"
Limitationswhere it fails, populations not coveredgeneric disclaimers
Ethical considerationsaffected people, sensitive features and proxies, harmsmissing
Risks and mitigationsconcrete risk, concrete mitigation, ownerrisks without mitigation
Monitoringmetrics, drift checks, retraining trigger, owner"will be monitored"

Section-by-section guidance with examples: writing-model-cards.md.

Disaggregated Evaluation

slice_metrics.py reports, overall and per slice level: n, positives, ROC AUC and TPR with 95% bootstrap CIs, average precision, FPR, positive rate, Brier score (or MAE/RMSE/bias for regression). Per slice column it reports:

GapFairness notionFlag
TPR max - minequal opportunity difference> --max-gap (0.1)
FPR max - minpart of equalised odds> 0.1
positive rate max - mindemographic parity difference> 0.1
positive rate min / maxdisparate impact ratio< 0.8 (four-fifths rule)
AUC CI below overall AUCslice underperformsUNDERPERFORMS
n < 50 or < 10 positives / negativesunreliable estimateLOW-N (excluded from gaps)

Metrics match scikit-learn (AUC, average precision, recall, Brier verified). Which notion applies depends on the decision; they cannot all hold at once when base rates differ. See fairness-evaluation.md.

Writing Rules

  • Numbers with uncertainty, on the right data: out-of-time evaluation where the model will face the future; CIs or fold ranges.
  • State the threshold for classifiers. Rates (TPR, FPR, positive rate) depend on it.
  • Compare to a baseline: the current process, a simple rule, or the previous model.
  • Say who is not covered: segments missing from training, low-n slices, new markets.
  • No blanket claims: write "TPR differs by 0.33 between groups A and B (0.84 vs 0.51)", not "the model is fair/unbiased".
  • Version the card with the model: the card describes one model version and one evaluation.

Output Format

When writing a model card, deliver:

  1. MODEL_CARD.md - all sections above, generated from the spec, with no TODOs left.
  2. Slice report - the disaggregated table with CIs and the flagged gaps, plus how they are addressed.
  3. Lint result - card_lint.py with no high findings.
  4. Open questions - facts only the owners can confirm (intended use, out-of-scope, risks).

Scripts

bash
python scripts/slice_metrics.py preds.csv --slices group age_band --threshold 0.4
python scripts/slice_metrics.py preds.csv --slices region --markdown
python scripts/slice_metrics.py preds.csv --regression --pred-col y_pred --slices segment --json > slices.json

Exit codes: 0 no flags, 1 a slice underperforms or a gap exceeds the limits, 2 usage or read error.

bash
python scripts/model_card.py spec.json -o MODEL_CARD.md
python scripts/model_card.py spec.json --slices slices.json -o MODEL_CARD.md --strict

Writes YAML metadata and every section. Missing required sections become visible **TODO:** lines. Exit codes: 0 written, 1 TODOs remain with --strict, 2 usage or read error.

bash
python scripts/card_lint.py MODEL_CARD.md
python scripts/card_lint.py README.md --json

Rules MC001-MC011: missing core sections, placeholders, metrics without numbers, no uncertainty, no slices, overclaiming, no data dates, no baseline, no threshold, no owner/version, no monitoring. Exit codes: 0 no high findings, 1 high findings, 2 read error.

Additional Resources

FileContent
card-spec.mdthe JSON spec schema with a complete example
writing-model-cards.mdsection-by-section guidance, good and bad examples, review checklist
fairness-evaluation.mdchoosing slices, fairness metrics and their trade-offs, small groups, mitigation options
More in Classical Machine Learning

anomaly-detection-model-builder

Build and evaluate anomaly detection for tabular, sensor and transaction data with few or no labels: define the anomaly type (point, contextual, collective), choose detectors (robust z-score/MAD and IQR rules, Mahalanobis with robust covariance, k-NN distance, LOF, Isolation Forest, one-class SVM, PCA reconstruction / autoencoders, forecasting residuals for time series), combine them by rank, explain each alert by its deviating features, set thresholds from an alert budget or the score distribution, and evaluate with precision@k, PR-AUC and recall on known incidents, or detector agreement when no labels exist. Includes pure-Python detectors (robust z, Mahalanobis, kNN, Isolation Forest, ensemble) and an evaluation script.

catboost-model-builder

Build CatBoost models that use its strengths correctly: raw categorical columns as cat_features (ordered target statistics, no manual encoding), text_features, Pool construction with NaN-safe categories, ordered boosting and has_time for temporal data, symmetric (oblivious) trees, overfitting detector with eval_set and use_best_model, losses for classification, regression (Quantile, Poisson, Tweedie) and ranking (YetiRank), class weights, monotone constraints, feature importance (PredictionValuesChange, LossFunctionChange) and SHAP values, GPU training and model export. Plans cat/text/numeric columns from a CSV and generates a runnable training script, and checks parameter sets for the conflicts CatBoost rejects (iterations vs n_estimators, Bayesian bootstrap with subsample, multiple class-weight options, depth > 16, use_best_model without eval_set).

categorical-encoding-selector

Choose and implement the right encoding for every categorical column given the model family and the data: one-hot (with infrequent-level grouping and handle_unknown), ordinal with an explicit order, cross-fitted target encoding (scikit-learn TargetEncoder, CatBoost ordered statistics, category_encoders inside CV), frequency/count encoding, feature hashing, native categorical support in LightGBM, HistGradientBoosting, XGBoost and CatBoost, prefix levels for codes (zip3, NAICS-2), and embeddings. Measures cardinality, rare-level share, unseen-level rate on a time-ordered holdout and target signal per column, emits the ColumnTransformer, and quantifies target-encoding leakage (in-sample vs out-of-fold vs holdout AUC).

classification-model-selector

Choose the right classification algorithm for tabular, text or mixed data and prove the choice with a fair bake-off: logistic regression (L2, L1, elastic net), linear SVM, naive Bayes, k-NN, RBF SVM, decision trees, random forest, extra trees, HistGradientBoosting, LightGBM, XGBoost, CatBoost, explainable boosting machines (EBM/GAM), points scorecards and MLPs, matched to data size, width, categorical and high-cardinality share, missingness, class imbalance, text, and constraints such as interpretability, monotonic constraints, calibrated probabilities, latency budgets and tiny deployable models. Ranks candidates with starting hyperparameters and cautions, then generates a bake-off script running every candidate on identical CV folds with per-fold metrics, fit time and prediction latency.

clustering-algorithm-selector

Choose the right clustering algorithm and check that clustering makes sense at all: cluster tendency (Hopkins statistic with a correlated-features caveat, nearest-neighbour isolation), then k-means / MiniBatchKMeans, Gaussian mixtures with BIC, agglomerative (Ward, average, complete, single) with dendrograms, DBSCAN, HDBSCAN, OPTICS, spectral clustering, BIRCH, mean shift, k-prototypes and k-modes for mixed or categorical data, and Gower distance, matched to data size, expected shapes, noise, varying density, known k, soft memberships, hierarchy and streaming. Covers preprocessing (scaling, encoding, dimensionality, distance metrics), validation (internal indices, bootstrap stability, external labels, business profiling) and assigning new points.

collaborative-filtering-implementer

Implement collaborative filtering recommenders that beat popularity: item-based kNN with cosine, Jaccard or asymmetric similarity, shrinkage, neighbourhood size, positive-only or mean-centred rating signals and 'because you liked X' explanations; implicit ALS with confidence weighting, BPR pairwise ranking and the closed-form EASE model in plain NumPy, tuned on a temporal validation split by recall@k and NDCG@k and refit on all data; plus user-kNN, explicit matrix factorisation, the implicit/LightFM libraries, hyperparameter ranges, cold-start fallbacks, scaling with sparse matrices and approximate nearest neighbours. Produces top-k lists in the format of the recommendation evaluator.