Skip to content
v0.1.3latest release121 skills4 categories6 sub-categories247 tested scripts397 reference docsContact us
Vector Agentic Skills
Submit a skill

kmeans-cluster-count-finder

Choose the number of clusters k for k-means (and k-means-like methods) with evidence instead of a single elbow plot: inertia elbow via the kneedle method, silhouette, Calinski-Harabasz, Davies-Bouldin, the gap statistic against uniform reference data (which also detects that there is no structure at all), bootstrap stability (adjusted Rand index on overlapping subsamples), BIC for Gaussian mixtures, cluster-size floors and business usefulness, then profile the chosen clusters (sizes, standardised feature differences, over-represented categories, outcome by cluster) so they can be named and acted on. Pure-Python k-means++ sweep validated against scikit-learn. Use when someone asks how many clusters to use, gets an ambiguous elbow plot, wants silhouette or gap statistic analysis, or needs to name and sanity-check k-means segments.

AI & Machine Learning / Classical Machine Learningv1.0.0Updated OCT 25 20263 references2 scripts

There is no true k; there is a k that the data supports and a k the business can use. Look at several criteria, because each fails differently: the elbow is often vague, silhouette favours few well-separated groups, Calinski-Harabasz grows with k on large data, and the gap statistic is the one that notices when the data is no more clustered than noise. Then check stability (does the same partition come back?) and profiles (can each cluster be named and acted on?). scripts/k_sweep.py runs the sweep and votes; scripts/cluster_profile.py profiles the clusters you keep.

Trigger Conditions

Activate this skill when the user:

  • Asks "how many clusters should I use?" or shows an elbow plot without a clear elbow
  • Wants silhouette, gap statistic, Calinski-Harabasz, Davies-Bouldin or BIC analysis
  • Must pick between, say, 4 and 6 customer segments
  • Has k-means clusters and needs to describe, name or validate them
  • Suspects the clusters are arbitrary

Choosing the algorithm itself: clustering-algorithm-selector; density-based alternatives without k: dbscan-parameter-tuner.

Workflow

text
1. PREPARE  -> same features and scaling you will use in production (see clustering-algorithm-selector)
2. SWEEP    -> python scripts/k_sweep.py data.csv --drop id --k-min 2 --k-max 10 --b 5
3. READ     -> votes + table; stability >= 0.8 at the chosen k; no tiny clusters; gap > 0.25
4. BUSINESS -> which k gives segments that need different actions? capacity for k strategies?
5. FIT      -> KMeans(n_clusters=k, n_init="auto", random_state=0) on all data
6. PROFILE  -> python scripts/cluster_profile.py segments.csv --label cluster --outcome churned
7. NAME     -> one plain-language name per cluster from its top distinguishing features

The Criteria and How They Fail

CriterionChoose k whereFails when
Inertia elbow (kneedle)curvature is largestcurve is smooth (no elbow); always decreases
Silhouettemean silhouette is highestprefers k = 2 for elongated or nested structure; expensive (O(n²))
Calinski-Harabaszratio is highestgrows with k on large datasets
Davies-Bouldinindex is lowestfavours compact spherical clusters
Gap statisticfirst k with gap(k) >= gap(k+1) − s(k+1)slower; uniform reference in the bounding box is crude for correlated data
Stability (ARI across subsamples)partitions reproduce (ARI > 0.8)stable is necessary, not sufficient (uniform data splits stably)
BIC (Gaussian mixtures)BIC minimum / elbowspends components on outliers
Businesssegments map to distinct actionsk chosen before looking at data

Reading k_sweep.py

text
  k     inertia silhouette        CH     DB stability     gap  sizes
  3       188.9      0.676     559.2  0.461      0.98   1.923  [102, 100, 98]
  4       155.6      0.689     471.9  0.789      0.99   1.884  [100, 98, 98, 4]
  6       111.1      0.703     417.4  0.528      0.80   1.908  [98, 98, 96, 3, 3, 2]
votes: elbow=3, silhouette=6, calinski_harabasz=3, davies_bouldin=3, gap=3
RECOMMENDED k = 3 (4 of 5 criteria; stability ARI 0.98)

Silhouette prefers k = 6 only by splitting off 2-3-point noise clusters (see sizes): always read cluster sizes next to the indices. On the uniform fixture every index votes k = 4, yet the gap statistic is <= 0.05 for every k: the script reports NO CLEAR k, because k-means cuts uniform data into stable but meaningless quadrants.

Formulas, computational cost and scikit-learn / yellowbrick equivalents: choosing-k-methods.md. The script's inertia, silhouette, CH and DB values match scikit-learn on the blobs fixture.

Business Rules for k

  • Actionability: k <= the number of distinct treatments the team can run.
  • Minimum size: each cluster >= 2-5% of rows (or a minimum count) unless tiny clusters are the point (anomalies).
  • Interpretability: every cluster has a name from 1-3 distinguishing features.
  • Stability over time: re-fit next month; segments persist and membership changes are explainable.
  • When two k values are close on the criteria, choose the one with the clearer profiles.

Profiling and Naming

text
cluster 3: 58 rows (14.5%), churned mean 0.379
  last_login_days   mean 35.26 vs 8.041 elsewhere  (d +3.45, higher)      -> "dormant"
cluster 0: 45 rows (11.2%), churned mean 0.467
  support_calls     mean 4.756 vs 0.707 elsewhere  (d +3.85, higher)      -> "support-heavy"
cluster 1: 103 rows (25.8%), churned mean 0.136
  plan=premium      97% vs 32% overall (lift 3.0x)                         -> "premium"

cluster_profile.py ranks features by the standardised difference (Cohen's d) between a cluster and the rest, lists over-represented categories, compares a business outcome across clusters, and warns about tiny or featureless clusters. Naming and reporting guidance: interpreting-segments.md.

k-Means Practicalities

IssuePractice
Initialisationk-means++ with n_init="auto" (or 10+); fix random_state
Scalingstandardise; log1p skewed features first
Outlierscap or remove; they become singleton clusters or drag centroids
Large dataMiniBatchKMeans; choose k on a sample
Non-spherical clustersGMM, HDBSCAN or spectral instead
Categorical featuresk-prototypes / k-modes, not one-hot k-means
Medoids needed (actual rows as centres, robust)k-medoids (sklearn_extra / kmedoids)
Hierarchy of segmentsbisecting k-means (BisectingKMeans) or Ward

More in kmeans-practice.md.

Output Format

When choosing k, deliver:

  1. Sweep table - inertia, silhouette, CH, DB, stability, gap and sizes per k (k_sweep.py).
  2. Votes and recommendation - or "no clear k" with the reason.
  3. Business check - actionability, size floor, interpretability for the candidate k values.
  4. Profiles - per cluster: size, top distinguishing features, categories, outcome (cluster_profile.py).
  5. Names and actions - a name and the intended action per cluster.
  6. Stability plan - refit cadence and how membership changes will be monitored.

Scripts

bash
python scripts/k_sweep.py data.csv --drop customer_id --k-max 10
python scripts/k_sweep.py data.csv --columns x1 x2 x3 --k-min 2 --k-max 8 --b 10 --json
python scripts/k_sweep.py data.csv --no-scale --n-init 10 --max-rows 3000

Pure-Python k-means++ (best of --n-init) on standardised numeric columns for each k: inertia, sampled silhouette, Calinski-Harabasz, Davies-Bouldin, stability ARI on overlapping 80% subsamples, gap statistic with --b uniform references (0 = skip), cluster sizes; votes from elbow (kneedle), silhouette, CH, DB and gap; recommendation or "no clear k" (criteria spread, weak silhouette, unstable winner, gap < 0.25). Exit codes: 0 clear recommendation, 1 no clear k, 2 usage error.

bash
python scripts/cluster_profile.py segments.csv --label cluster
python scripts/cluster_profile.py segments.csv --label cluster --outcome churned --drop customer_id --top 4 --json

Per cluster (noise label -1 shown last): size and share, numeric features ranked by Cohen's d vs the rest, categorical levels with the highest lift, outcome mean; warnings for clusters below --min-share or without a distinguishing feature, and for outcomes that barely differ. Exit codes: 0 profiled, 2 usage error.

Additional Resources

FileContent
choosing-k-methods.mdElbow/kneedle, silhouette, CH, DB, gap statistic, stability, BIC/AIC, information criteria for k-means, scikit-learn and yellowbrick code
kmeans-practice.mdInitialisation, scaling, outliers, MiniBatch, empty clusters, k-medoids, bisecting k-means, assigning new rows, refreshing segments
interpreting-segments.mdProfiling, Cohen's d and lift, naming, segment cards, outcome validation, communicating segments
More in Classical Machine Learning

anomaly-detection-model-builder

Build and evaluate anomaly detection for tabular, sensor and transaction data with few or no labels: define the anomaly type (point, contextual, collective), choose detectors (robust z-score/MAD and IQR rules, Mahalanobis with robust covariance, k-NN distance, LOF, Isolation Forest, one-class SVM, PCA reconstruction / autoencoders, forecasting residuals for time series), combine them by rank, explain each alert by its deviating features, set thresholds from an alert budget or the score distribution, and evaluate with precision@k, PR-AUC and recall on known incidents, or detector agreement when no labels exist. Includes pure-Python detectors (robust z, Mahalanobis, kNN, Isolation Forest, ensemble) and an evaluation script.

catboost-model-builder

Build CatBoost models that use its strengths correctly: raw categorical columns as cat_features (ordered target statistics, no manual encoding), text_features, Pool construction with NaN-safe categories, ordered boosting and has_time for temporal data, symmetric (oblivious) trees, overfitting detector with eval_set and use_best_model, losses for classification, regression (Quantile, Poisson, Tweedie) and ranking (YetiRank), class weights, monotone constraints, feature importance (PredictionValuesChange, LossFunctionChange) and SHAP values, GPU training and model export. Plans cat/text/numeric columns from a CSV and generates a runnable training script, and checks parameter sets for the conflicts CatBoost rejects (iterations vs n_estimators, Bayesian bootstrap with subsample, multiple class-weight options, depth > 16, use_best_model without eval_set).

categorical-encoding-selector

Choose and implement the right encoding for every categorical column given the model family and the data: one-hot (with infrequent-level grouping and handle_unknown), ordinal with an explicit order, cross-fitted target encoding (scikit-learn TargetEncoder, CatBoost ordered statistics, category_encoders inside CV), frequency/count encoding, feature hashing, native categorical support in LightGBM, HistGradientBoosting, XGBoost and CatBoost, prefix levels for codes (zip3, NAICS-2), and embeddings. Measures cardinality, rare-level share, unseen-level rate on a time-ordered holdout and target signal per column, emits the ColumnTransformer, and quantifies target-encoding leakage (in-sample vs out-of-fold vs holdout AUC).

classification-model-selector

Choose the right classification algorithm for tabular, text or mixed data and prove the choice with a fair bake-off: logistic regression (L2, L1, elastic net), linear SVM, naive Bayes, k-NN, RBF SVM, decision trees, random forest, extra trees, HistGradientBoosting, LightGBM, XGBoost, CatBoost, explainable boosting machines (EBM/GAM), points scorecards and MLPs, matched to data size, width, categorical and high-cardinality share, missingness, class imbalance, text, and constraints such as interpretability, monotonic constraints, calibrated probabilities, latency budgets and tiny deployable models. Ranks candidates with starting hyperparameters and cautions, then generates a bake-off script running every candidate on identical CV folds with per-fold metrics, fit time and prediction latency.

clustering-algorithm-selector

Choose the right clustering algorithm and check that clustering makes sense at all: cluster tendency (Hopkins statistic with a correlated-features caveat, nearest-neighbour isolation), then k-means / MiniBatchKMeans, Gaussian mixtures with BIC, agglomerative (Ward, average, complete, single) with dendrograms, DBSCAN, HDBSCAN, OPTICS, spectral clustering, BIRCH, mean shift, k-prototypes and k-modes for mixed or categorical data, and Gower distance, matched to data size, expected shapes, noise, varying density, known k, soft memberships, hierarchy and streaming. Covers preprocessing (scaling, encoding, dimensionality, distance metrics), validation (internal indices, bootstrap stability, external labels, business profiling) and assigning new points.

collaborative-filtering-implementer

Implement collaborative filtering recommenders that beat popularity: item-based kNN with cosine, Jaccard or asymmetric similarity, shrinkage, neighbourhood size, positive-only or mean-centred rating signals and 'because you liked X' explanations; implicit ALS with confidence weighting, BPR pairwise ranking and the closed-form EASE model in plain NumPy, tuned on a temporal validation split by recall@k and NDCG@k and refit on all data; plus user-kNN, explicit matrix factorisation, the implicit/LightFM libraries, hyperparameter ranges, cold-start fallbacks, scaling with sparse matrices and approximate nearest neighbours. Produces top-k lists in the format of the recommendation evaluator.