Clusters are defined by distances, so preprocessing is the model.
Feature choice
- Use features that describe the behaviour the segments should differ on (for customer segments: recency, frequency, monetary, product mix, channel), not IDs, dates or outcome labels.
- Drop near-duplicates: two copies of the same signal double its weight.
- Fewer, meaningful features beat many weak ones (distances concentrate in high dimensions).
Scaling
| Situation | Transform |
|---|---|
| Mixed units | StandardScaler (or RobustScaler with outliers) |
| Skewed amounts / counts | log1p, then standardise |
| Proportions | keep in [0, 1] or logit |
| Features of different importance | multiply scaled columns by weights deliberately |
Unscaled, the widest column decides the clusters (feature-scaling-advisor's scale_impact.py shows the
dominance).
Categorical features
| Option | Notes |
|---|---|
| k-prototypes / k-modes | native mixed / categorical objective |
| Gower distance | proper mixed dissimilarity; O(n²) |
| One-hot + scaling | a binary column's variance depends on its frequency; down-weight one-hot blocks (e.g. divide by sqrt(levels)) or the categorical features dominate |
| Target-free embeddings | e.g. from an autoencoder or entity embeddings |
Dimensionality
- PCA to retain ~80-90% variance removes noise and speeds k-means/GMM.
- UMAP (n_components 5-20,
min_dist=0.0) before HDBSCAN works well for complex manifolds; do not cluster on a 2-D t-SNE plot (it distorts densities and distances). - Check that clusters found in the reduced space are visible in the original feature profiles.
Distance metrics
| Data | Metric |
|---|---|
| Continuous, scaled | Euclidean |
| High-dimensional embeddings / text | cosine (or L2-normalise then Euclidean) |
| Counts / profiles | cosine, Bray-Curtis |
| Lat/lon | haversine (radians) |
| Mixed | Gower |
| Binary | Jaccard, Hamming |
Outliers
Extreme rows pull centroids (k-means) and steal mixture components (GMM). Either cap features (winsorise), remove obvious errors, or use a density method that labels noise.
Checklist
- IDs, dates, labels removed
- Skewed features transformed
- Features scaled; weights deliberate
- Categoricals handled with a suitable dissimilarity
- Redundant features removed
- Dimensionality reduced if > ~20-50 features
anomaly-detection-model-builder
Build and evaluate anomaly detection for tabular, sensor and transaction data with few or no labels: define the anomaly type (point, contextual, collective), choose detectors (robust z-score/MAD and IQR rules, Mahalanobis with robust covariance, k-NN distance, LOF, Isolation Forest, one-class SVM, PCA reconstruction / autoencoders, forecasting residuals for time series), combine them by rank, explain each alert by its deviating features, set thresholds from an alert budget or the score distribution, and evaluate with precision@k, PR-AUC and recall on known incidents, or detector agreement when no labels exist. Includes pure-Python detectors (robust z, Mahalanobis, kNN, Isolation Forest, ensemble) and an evaluation script.
catboost-model-builder
Build CatBoost models that use its strengths correctly: raw categorical columns as cat_features (ordered target statistics, no manual encoding), text_features, Pool construction with NaN-safe categories, ordered boosting and has_time for temporal data, symmetric (oblivious) trees, overfitting detector with eval_set and use_best_model, losses for classification, regression (Quantile, Poisson, Tweedie) and ranking (YetiRank), class weights, monotone constraints, feature importance (PredictionValuesChange, LossFunctionChange) and SHAP values, GPU training and model export. Plans cat/text/numeric columns from a CSV and generates a runnable training script, and checks parameter sets for the conflicts CatBoost rejects (iterations vs n_estimators, Bayesian bootstrap with subsample, multiple class-weight options, depth > 16, use_best_model without eval_set).
categorical-encoding-selector
Choose and implement the right encoding for every categorical column given the model family and the data: one-hot (with infrequent-level grouping and handle_unknown), ordinal with an explicit order, cross-fitted target encoding (scikit-learn TargetEncoder, CatBoost ordered statistics, category_encoders inside CV), frequency/count encoding, feature hashing, native categorical support in LightGBM, HistGradientBoosting, XGBoost and CatBoost, prefix levels for codes (zip3, NAICS-2), and embeddings. Measures cardinality, rare-level share, unseen-level rate on a time-ordered holdout and target signal per column, emits the ColumnTransformer, and quantifies target-encoding leakage (in-sample vs out-of-fold vs holdout AUC).
classification-model-selector
Choose the right classification algorithm for tabular, text or mixed data and prove the choice with a fair bake-off: logistic regression (L2, L1, elastic net), linear SVM, naive Bayes, k-NN, RBF SVM, decision trees, random forest, extra trees, HistGradientBoosting, LightGBM, XGBoost, CatBoost, explainable boosting machines (EBM/GAM), points scorecards and MLPs, matched to data size, width, categorical and high-cardinality share, missingness, class imbalance, text, and constraints such as interpretability, monotonic constraints, calibrated probabilities, latency budgets and tiny deployable models. Ranks candidates with starting hyperparameters and cautions, then generates a bake-off script running every candidate on identical CV folds with per-fold metrics, fit time and prediction latency.
collaborative-filtering-implementer
Implement collaborative filtering recommenders that beat popularity: item-based kNN with cosine, Jaccard or asymmetric similarity, shrinkage, neighbourhood size, positive-only or mean-centred rating signals and 'because you liked X' explanations; implicit ALS with confidence weighting, BPR pairwise ranking and the closed-form EASE model in plain NumPy, tuned on a temporal validation split by recall@k and NDCG@k and refit on all data; plus user-kNN, explicit matrix factorisation, the implicit/LightFM libraries, hyperparameter ranges, cold-start fallbacks, scaling with sparse matrices and approximate nearest neighbours. Produces top-k lists in the format of the recommendation evaluator.
counterfactual-explanation-generator
Generate counterfactual explanations and algorithmic recourse: the smallest realistic change to a row's features that changes the model's decision. Solves exact minimal-cost recourse for linear and logistic models (MAD-scaled L1 cost, immutable features, increase/decrease directions, bounds, integer and binary features, diverse alternatives, features every counterfactual must change, and a no-recourse finding when immutable features decide), generates DiCE counterfactuals for any scikit-learn model with constraints turned into permitted ranges and every result re-predicted, and validates counterfactuals from any method for constraint violations, implausible values, off-manifold rows, broken derived-feature relations and failure to flip the prediction.