There is no accuracy for clustering. Validate on four fronts.
1. Internal indices (diagnostics, not truth)
| Index | Better | Bias |
|---|---|---|
| Silhouette | higher (−1..1) | favours convex, well-separated clusters |
| Calinski-Harabasz | higher | favours convex clusters, grows with k on large data |
| Davies-Bouldin | lower | favours convex clusters |
| DBCV (density-based validation) | higher | for density clusters (HDBSCAN relative_validity_ in the hdbscan package) |
from sklearn.metrics import silhouette_score, calinski_harabasz_score, davies_bouldin_score
mask = labels != -1 # exclude noise for density methods
print(silhouette_score(X[mask], labels[mask]), calinski_harabasz_score(X[mask], labels[mask]),
davies_bouldin_score(X[mask], labels[mask]))Compare algorithms with the index that matches their cluster model; never compare k-means and HDBSCAN by silhouette alone.
2. Stability
Clusters that matter reappear on resampled data:
import numpy as np
from sklearn.metrics import adjusted_rand_score
rng = np.random.default_rng(0)
scores = []
for _ in range(20):
idx = rng.choice(len(X), int(0.8 * len(X)), replace=False)
a = KMeans(4, n_init="auto", random_state=int(rng.integers(1e6))).fit(X[idx])
b = KMeans(4, n_init="auto", random_state=int(rng.integers(1e6))).fit(X[idx])
scores.append(adjusted_rand_score(a.labels_, b.labels_)) # seed stability; also compare across subsamples on shared rows
print(np.mean(scores)) # > 0.8 stable, < 0.5 arbitraryFor subsample stability, fit on two overlapping subsamples and compare labels on the shared rows.
3. External validation (when some labels exist)
Adjusted Rand Index, normalised mutual information, homogeneity/completeness against known groups. A segmentation that is unrelated to every known variable is suspicious.
4. Business validation
- Profile each cluster: size, mean/median of each feature relative to the population (lift), top categories.
- Name them in plain language; if you cannot, the cluster may be an artefact.
- Outcome differences: do clusters differ in churn rate, spend, response? Segments that behave the same do not need different treatment.
- Actionability: each cluster maps to a different action or message.
- Size floor: tiny clusters (< 1-2% of rows) are rarely actionable.
- Stability over time: re-run next month; do segments persist and do customers move between them sensibly?
Profiling snippet
prof = df.assign(cluster=labels).groupby("cluster")[feature_cols].mean()
lift = prof / df[feature_cols].mean()
sizes = df.assign(cluster=labels).cluster.value_counts(normalize=True)anomaly-detection-model-builder
Build and evaluate anomaly detection for tabular, sensor and transaction data with few or no labels: define the anomaly type (point, contextual, collective), choose detectors (robust z-score/MAD and IQR rules, Mahalanobis with robust covariance, k-NN distance, LOF, Isolation Forest, one-class SVM, PCA reconstruction / autoencoders, forecasting residuals for time series), combine them by rank, explain each alert by its deviating features, set thresholds from an alert budget or the score distribution, and evaluate with precision@k, PR-AUC and recall on known incidents, or detector agreement when no labels exist. Includes pure-Python detectors (robust z, Mahalanobis, kNN, Isolation Forest, ensemble) and an evaluation script.
catboost-model-builder
Build CatBoost models that use its strengths correctly: raw categorical columns as cat_features (ordered target statistics, no manual encoding), text_features, Pool construction with NaN-safe categories, ordered boosting and has_time for temporal data, symmetric (oblivious) trees, overfitting detector with eval_set and use_best_model, losses for classification, regression (Quantile, Poisson, Tweedie) and ranking (YetiRank), class weights, monotone constraints, feature importance (PredictionValuesChange, LossFunctionChange) and SHAP values, GPU training and model export. Plans cat/text/numeric columns from a CSV and generates a runnable training script, and checks parameter sets for the conflicts CatBoost rejects (iterations vs n_estimators, Bayesian bootstrap with subsample, multiple class-weight options, depth > 16, use_best_model without eval_set).
categorical-encoding-selector
Choose and implement the right encoding for every categorical column given the model family and the data: one-hot (with infrequent-level grouping and handle_unknown), ordinal with an explicit order, cross-fitted target encoding (scikit-learn TargetEncoder, CatBoost ordered statistics, category_encoders inside CV), frequency/count encoding, feature hashing, native categorical support in LightGBM, HistGradientBoosting, XGBoost and CatBoost, prefix levels for codes (zip3, NAICS-2), and embeddings. Measures cardinality, rare-level share, unseen-level rate on a time-ordered holdout and target signal per column, emits the ColumnTransformer, and quantifies target-encoding leakage (in-sample vs out-of-fold vs holdout AUC).
classification-model-selector
Choose the right classification algorithm for tabular, text or mixed data and prove the choice with a fair bake-off: logistic regression (L2, L1, elastic net), linear SVM, naive Bayes, k-NN, RBF SVM, decision trees, random forest, extra trees, HistGradientBoosting, LightGBM, XGBoost, CatBoost, explainable boosting machines (EBM/GAM), points scorecards and MLPs, matched to data size, width, categorical and high-cardinality share, missingness, class imbalance, text, and constraints such as interpretability, monotonic constraints, calibrated probabilities, latency budgets and tiny deployable models. Ranks candidates with starting hyperparameters and cautions, then generates a bake-off script running every candidate on identical CV folds with per-fold metrics, fit time and prediction latency.
collaborative-filtering-implementer
Implement collaborative filtering recommenders that beat popularity: item-based kNN with cosine, Jaccard or asymmetric similarity, shrinkage, neighbourhood size, positive-only or mean-centred rating signals and 'because you liked X' explanations; implicit ALS with confidence weighting, BPR pairwise ranking and the closed-form EASE model in plain NumPy, tuned on a temporal validation split by recall@k and NDCG@k and refit on all data; plus user-kNN, explicit matrix factorisation, the implicit/LightFM libraries, hyperparameter ranges, cold-start fallbacks, scaling with sparse matrices and approximate nearest neighbours. Produces top-k lists in the format of the recommendation evaluator.
counterfactual-explanation-generator
Generate counterfactual explanations and algorithmic recourse: the smallest realistic change to a row's features that changes the model's decision. Solves exact minimal-cost recourse for linear and logistic models (MAD-scaled L1 cost, immutable features, increase/decrease directions, bounds, integer and binary features, diverse alternatives, features every counterfactual must change, and a no-recourse finding when immutable features decide), generates DiCE counterfactuals for any scikit-learn model with constraints turned into permitted ranges and every result re-predicted, and validates counterfactuals from any method for constraint violations, implausible values, off-manifold rows, broken derived-feature relations and failure to flip the prediction.