What each algorithm assumes
| Algorithm | Cluster model | Needs k | Handles noise | Shapes | Scales to | Soft |
|---|---|---|---|---|---|---|
| k-means | nearest centroid, equal-variance spheres | yes | no | convex | millions (MiniBatch) | no |
| Gaussian mixture | ellipsoidal Gaussians | yes (BIC) | partly | elliptical | ~10^5 | yes |
| Agglomerative Ward | merge minimising variance | yes (or cut height) | no | convex | ~2-3 x 10^4 | no |
| Agglomerative average/complete | merge by linkage on any metric | yes | weak | moderate | ~2-3 x 10^4 | no |
| Single linkage | chaining | yes | no | elongated | ~10^4 | no |
| DBSCAN | dense regions of fixed density | no | yes | arbitrary | ~10^6 (index-dependent) | no |
| HDBSCAN | density hierarchy, varying density | no | yes | arbitrary | ~10^6 | via hdbscan package |
| OPTICS | density ordering | no | yes | arbitrary | ~10^5 | no |
| Spectral | graph cuts on a similarity graph | yes | no | non-convex | ~10^4 | no |
| BIRCH | CF-tree micro-clusters | optional | weak | convex | streaming | no |
| Mean shift | density modes | no | weak | blobs | ~10^4 | no |
| Affinity propagation | exemplars via message passing | no | no | moderate | ~10^4 | no |
| k-prototypes / k-modes | centroids / modes with mixed dissimilarity | yes | no | convex-ish | ~10^6 | no |
Decision questions
- Is there structure? Run cluster_tendency.py. If H ~ 0.5, any partition is arbitrary.
- Is k given by the business? (one segment per sales team) -> k-means / GMM / Ward with that k.
- Must every point belong to a cluster? If no (outliers, background), density methods.
- Shapes? Convex segments -> centroid methods; manifolds, rings, elongated groups -> HDBSCAN / spectral.
- Different densities? HDBSCAN / OPTICS, not DBSCAN.
- Soft memberships? GMM (or HDBSCAN soft clustering).
- Hierarchy / drill-down? Agglomerative with a dendrogram.
- Mixed types? k-prototypes, Gower + hierarchical/HDBSCAN, or encode + scale carefully.
- Size? > 10^5 rows: MiniBatchKMeans, BIRCH, HDBSCAN on a sample or a UMAP embedding.
Typical combinations
| Goal | Pipeline |
|---|---|
| Customer segments for marketing | features (RFM, behaviour) -> scale -> k-means (k = 4-8) -> profile -> stability check |
| Discover unusual groups / noise | scale -> UMAP (10-20 dims) -> HDBSCAN |
| Geographic hotspots | lat/lon (haversine) -> DBSCAN / HDBSCAN with metric="haversine" on radians |
| Document / embedding topics | embeddings -> UMAP -> HDBSCAN (BERTopic-style) |
| Product taxonomy | agglomerative (average, cosine) -> dendrogram cut |
| Overlapping personas | GMM with probabilities |
What clustering cannot do
- Find the "right" number of clusters without a goal; different k answer different questions.
- Be validated by accuracy: there is no ground truth. Use stability, internal indices as diagnostics, and business usefulness (do segments behave differently and lead to different actions?).
- Fix bad features: clusters reflect the features and their scaling more than the algorithm.
anomaly-detection-model-builder
Build and evaluate anomaly detection for tabular, sensor and transaction data with few or no labels: define the anomaly type (point, contextual, collective), choose detectors (robust z-score/MAD and IQR rules, Mahalanobis with robust covariance, k-NN distance, LOF, Isolation Forest, one-class SVM, PCA reconstruction / autoencoders, forecasting residuals for time series), combine them by rank, explain each alert by its deviating features, set thresholds from an alert budget or the score distribution, and evaluate with precision@k, PR-AUC and recall on known incidents, or detector agreement when no labels exist. Includes pure-Python detectors (robust z, Mahalanobis, kNN, Isolation Forest, ensemble) and an evaluation script.
catboost-model-builder
Build CatBoost models that use its strengths correctly: raw categorical columns as cat_features (ordered target statistics, no manual encoding), text_features, Pool construction with NaN-safe categories, ordered boosting and has_time for temporal data, symmetric (oblivious) trees, overfitting detector with eval_set and use_best_model, losses for classification, regression (Quantile, Poisson, Tweedie) and ranking (YetiRank), class weights, monotone constraints, feature importance (PredictionValuesChange, LossFunctionChange) and SHAP values, GPU training and model export. Plans cat/text/numeric columns from a CSV and generates a runnable training script, and checks parameter sets for the conflicts CatBoost rejects (iterations vs n_estimators, Bayesian bootstrap with subsample, multiple class-weight options, depth > 16, use_best_model without eval_set).
categorical-encoding-selector
Choose and implement the right encoding for every categorical column given the model family and the data: one-hot (with infrequent-level grouping and handle_unknown), ordinal with an explicit order, cross-fitted target encoding (scikit-learn TargetEncoder, CatBoost ordered statistics, category_encoders inside CV), frequency/count encoding, feature hashing, native categorical support in LightGBM, HistGradientBoosting, XGBoost and CatBoost, prefix levels for codes (zip3, NAICS-2), and embeddings. Measures cardinality, rare-level share, unseen-level rate on a time-ordered holdout and target signal per column, emits the ColumnTransformer, and quantifies target-encoding leakage (in-sample vs out-of-fold vs holdout AUC).
classification-model-selector
Choose the right classification algorithm for tabular, text or mixed data and prove the choice with a fair bake-off: logistic regression (L2, L1, elastic net), linear SVM, naive Bayes, k-NN, RBF SVM, decision trees, random forest, extra trees, HistGradientBoosting, LightGBM, XGBoost, CatBoost, explainable boosting machines (EBM/GAM), points scorecards and MLPs, matched to data size, width, categorical and high-cardinality share, missingness, class imbalance, text, and constraints such as interpretability, monotonic constraints, calibrated probabilities, latency budgets and tiny deployable models. Ranks candidates with starting hyperparameters and cautions, then generates a bake-off script running every candidate on identical CV folds with per-fold metrics, fit time and prediction latency.
collaborative-filtering-implementer
Implement collaborative filtering recommenders that beat popularity: item-based kNN with cosine, Jaccard or asymmetric similarity, shrinkage, neighbourhood size, positive-only or mean-centred rating signals and 'because you liked X' explanations; implicit ALS with confidence weighting, BPR pairwise ranking and the closed-form EASE model in plain NumPy, tuned on a temporal validation split by recall@k and NDCG@k and refit on all data; plus user-kNN, explicit matrix factorisation, the implicit/LightFM libraries, hyperparameter ranges, cold-start fallbacks, scaling with sparse matrices and approximate nearest neighbours. Produces top-k lists in the format of the recommendation evaluator.
counterfactual-explanation-generator
Generate counterfactual explanations and algorithmic recourse: the smallest realistic change to a row's features that changes the model's decision. Solves exact minimal-cost recourse for linear and logistic models (MAD-scaled L1 cost, immutable features, increase/decrease directions, bounds, integer and binary features, diverse alternatives, features every counterfactual must change, and a no-recourse finding when immutable features decide), generates DiCE counterfactuals for any scikit-learn model with constraints turned into permitted ranges and every result re-predicted, and validates counterfactuals from any method for constraint violations, implausible values, off-manifold rows, broken derived-feature relations and failure to flip the prediction.