DBSCAN has two parameters with physical meaning: min_samples (how many points make a dense
neighbourhood, i.e. the smallest group you care about, and how strict noise labelling is) and eps (the
neighbourhood radius in the scaled feature space). Choose min_samples first, read eps off the k-distance curve,
then confirm on a small grid that the result is stable: the same cluster count over neighbouring settings.
If no single eps works because clusters have different densities, DBSCAN is the wrong tool: use HDBSCAN.
scripts/kdist.py finds the knee and a noise-share bracket; scripts/dbscan_grid.py runs the grid and
recommends a stable setting (results identical to scikit-learn's DBSCAN on the fixture).
Trigger Conditions
Activate this skill when the user:
- Asks how to choose eps / min_samples for DBSCAN
- Gets one giant cluster, everything labelled noise, or wildly different results from small eps changes
- Clusters GPS / lat-lon points, embeddings, or sensor readings with outliers
- Needs density-based clusters with noise and wonders about HDBSCAN or OPTICS
- Must assign new points to DBSCAN clusters
Choosing between algorithms: clustering-algorithm-selector; scaling: feature-scaling-advisor; anomaly scoring instead of clustering: anomaly-detection-model-builder.
Workflow
1. SCALE -> standardise (or a meaningful metric: haversine in radians, cosine for embeddings)
2. DIMENSIONS -> > ~10-20 features: reduce first (PCA / UMAP); distances concentrate in high dimensions
3. MIN_SAMPLES-> start at 2 x dims (>= 4); raise for noisy data or to ignore small groups
4. EPS -> python scripts/kdist.py data.csv --drop id --min-samples M (knee + bracket)
5. GRID -> python scripts/dbscan_grid.py data.csv --drop id --eps ... --min-samples ...
6. CHECK -> stable cluster count, noise 2-15% (domain-dependent), no cluster > 90%, profiles make sense
7. FALLBACK -> no stable setting or varying densities -> HDBSCAN(min_cluster_size=...)min_samples
| Consideration | Guidance |
|---|---|
| Default | 2 x number of dimensions (Sander et al. 1998), at least 4 |
| Noisy data | larger (fewer spurious core points, more noise labels) |
| Smallest meaningful group | min_samples <= that group size |
| Large datasets | larger values (10-50) stabilise results |
| Duplicates / many identical points | larger, or deduplicate with sample weights |
min_samples counts the point itself in scikit-learn. A point is core if at least min_samples points (including itself) lie within eps; border if within eps of a core point; otherwise noise.
eps from the k-distance curve
300 rows x 3 columns (standardised); min_samples=6 -> distance to the 5-th nearest neighbour
quantiles: p10=0.1848 p25=0.2167 p50=0.2594 p75=0.3509 p90=0.4516 p95=0.7473 p99=3.0747
knee at rank 277/300: eps ~ 0.5133 (points beyond it: 7.3% -> likely noise)
eps for ~15% .. ~2% noise: 0.4046 .. 2.178Sort every point's distance to its (min_samples − 1)-th nearest neighbour; cluster points sit on the flat part, noise on the steep tail. The knee is the starting eps (the fixture has 5% planted noise; the knee flags 7%).
Reading the grid
min_s eps clusters noise largest silh stable ARI
6 0.4 3 5.7% 31.7% 0.733 4 0.989 <- recommended
6 1 2 4.7% 63.7% 0.535 2 0.590 (two clusters merged)
10 0.3 3 28.0% 25.7% 0.786 2 0.581 (too strict: real points become noise)- Stable region: the cluster count repeats across neighbouring eps/min_samples cells (
stablecolumn); a count that appears in one cell only is an accident. The script requires >= 2 agreeing neighbours. - Merging:
largestjumping toward 100% means eps bridged two clusters. - Fragmentation: very small eps makes many tiny clusters and lots of noise.
- On the uniform fixture no setting is stable: the script says so instead of recommending 10 "clusters".
Parameter interplay, metrics and HDBSCAN translation: dbscan-in-depth.md.
Common Failure Modes
| Symptom | Cause | Fix |
|---|---|---|
| One giant cluster | eps too large, or chains of border points bridge clusters | lower eps; raise min_samples |
| Almost everything is noise | eps too small for the scaled space; high dimensions | raise eps (k-distance), reduce dimensions |
| Results jump with tiny eps changes | no density gap / varying densities | HDBSCAN or OPTICS |
| Dense core found, sparse groups lost | single global density | HDBSCAN |
| Clusters dominated by one feature | unscaled features | standardise; weight features deliberately |
| Geographic clusters wrong size | Euclidean on degrees | haversine on radians, eps = km / 6371 |
| Slow on large data | O(n²) neighbourhoods without an index | algorithm="ball_tree", sample, or HDBSCAN |
Geographic and Other Metrics
import numpy as np
from sklearn.cluster import DBSCAN
coords = np.radians(df[["lat", "lon"]].to_numpy())
db = DBSCAN(eps=0.5 / 6371.0, min_samples=10, metric="haversine", algorithm="ball_tree").fit(coords) # 500 mEmbeddings: L2-normalise and use Euclidean (equivalent ordering to cosine), or metric="cosine" with
algorithm="brute". Details: metrics-and-scaling.md.
When to Use HDBSCAN Instead
HDBSCAN removes eps: it builds the density hierarchy and keeps the most persistent clusters, so it handles
varying densities and gives membership strengths. Translate: min_cluster_size ≈ the smallest meaningful
group; min_samples as in DBSCAN (noise strictness). cluster_selection_epsilon merges clusters closer than
a distance (DBSCAN-like behaviour at the bottom).
Output Format
When tuning DBSCAN, deliver:
- Preprocessing - scaling/metric, dimensionality reduction if any.
- min_samples rationale and the k-distance summary (
kdist.py): knee eps and bracket. - Grid - clusters, noise, largest share, silhouette, stability (and ARI if labels exist) (
dbscan_grid.py). - Choice - eps and min_samples from the stable region, with cluster sizes and noise share.
- Sanity checks - cluster profiles; noise points inspected; HDBSCAN comparison if unstable.
- Operational note - how new points are assigned (nearest core point / classifier).
Scripts
python scripts/kdist.py data.csv --drop id label
python scripts/kdist.py data.csv --min-samples 10 --columns x y --no-scale --jsonDistance from every point to its (min_samples − 1)-th nearest neighbour (standardised features, default min_samples = 2 x dims, at least 4), quantiles, kneedle knee as the eps suggestion with the share of points beyond it, an eps bracket for ~15% to ~2% noise, and an ASCII plot. Exit codes: 0 printed, 2 usage or read error.
python scripts/dbscan_grid.py data.csv --drop id --eps 0.3 0.4 0.5 0.6 0.8 --min-samples 4 6 10
python scripts/dbscan_grid.py data.csv --labels true_cluster
python scripts/dbscan_grid.py data.csv --noise-max 0.1 --jsonPure-Python DBSCAN over the grid (eps defaults to multiples of the k-distance 90th percentile): cluster count,
noise share, largest-cluster share, sampled silhouette of non-noise points, stable neighbour count and optional
ARI vs --labels; recommends the most stable setting meeting the noise window and merge limit. Exit codes: 0
recommendation, 1 no stable setting, 2 usage error.
Additional Resources
| File | Content |
|---|---|
| dbscan-in-depth.md | Core/border/noise mechanics, parameter interplay, determinism, complexity, HDBSCAN/OPTICS parameters, assigning new points |
| metrics-and-scaling.md | Scaling, feature weighting, haversine for coordinates, cosine for embeddings, high dimensions and UMAP, precomputed distances |
| evaluating-density-clusters.md | Noise interpretation, DBCV, silhouette on non-noise points, stability across settings, profiling noise vs clusters |
anomaly-detection-model-builder
Build and evaluate anomaly detection for tabular, sensor and transaction data with few or no labels: define the anomaly type (point, contextual, collective), choose detectors (robust z-score/MAD and IQR rules, Mahalanobis with robust covariance, k-NN distance, LOF, Isolation Forest, one-class SVM, PCA reconstruction / autoencoders, forecasting residuals for time series), combine them by rank, explain each alert by its deviating features, set thresholds from an alert budget or the score distribution, and evaluate with precision@k, PR-AUC and recall on known incidents, or detector agreement when no labels exist. Includes pure-Python detectors (robust z, Mahalanobis, kNN, Isolation Forest, ensemble) and an evaluation script.
catboost-model-builder
Build CatBoost models that use its strengths correctly: raw categorical columns as cat_features (ordered target statistics, no manual encoding), text_features, Pool construction with NaN-safe categories, ordered boosting and has_time for temporal data, symmetric (oblivious) trees, overfitting detector with eval_set and use_best_model, losses for classification, regression (Quantile, Poisson, Tweedie) and ranking (YetiRank), class weights, monotone constraints, feature importance (PredictionValuesChange, LossFunctionChange) and SHAP values, GPU training and model export. Plans cat/text/numeric columns from a CSV and generates a runnable training script, and checks parameter sets for the conflicts CatBoost rejects (iterations vs n_estimators, Bayesian bootstrap with subsample, multiple class-weight options, depth > 16, use_best_model without eval_set).
categorical-encoding-selector
Choose and implement the right encoding for every categorical column given the model family and the data: one-hot (with infrequent-level grouping and handle_unknown), ordinal with an explicit order, cross-fitted target encoding (scikit-learn TargetEncoder, CatBoost ordered statistics, category_encoders inside CV), frequency/count encoding, feature hashing, native categorical support in LightGBM, HistGradientBoosting, XGBoost and CatBoost, prefix levels for codes (zip3, NAICS-2), and embeddings. Measures cardinality, rare-level share, unseen-level rate on a time-ordered holdout and target signal per column, emits the ColumnTransformer, and quantifies target-encoding leakage (in-sample vs out-of-fold vs holdout AUC).
classification-model-selector
Choose the right classification algorithm for tabular, text or mixed data and prove the choice with a fair bake-off: logistic regression (L2, L1, elastic net), linear SVM, naive Bayes, k-NN, RBF SVM, decision trees, random forest, extra trees, HistGradientBoosting, LightGBM, XGBoost, CatBoost, explainable boosting machines (EBM/GAM), points scorecards and MLPs, matched to data size, width, categorical and high-cardinality share, missingness, class imbalance, text, and constraints such as interpretability, monotonic constraints, calibrated probabilities, latency budgets and tiny deployable models. Ranks candidates with starting hyperparameters and cautions, then generates a bake-off script running every candidate on identical CV folds with per-fold metrics, fit time and prediction latency.
clustering-algorithm-selector
Choose the right clustering algorithm and check that clustering makes sense at all: cluster tendency (Hopkins statistic with a correlated-features caveat, nearest-neighbour isolation), then k-means / MiniBatchKMeans, Gaussian mixtures with BIC, agglomerative (Ward, average, complete, single) with dendrograms, DBSCAN, HDBSCAN, OPTICS, spectral clustering, BIRCH, mean shift, k-prototypes and k-modes for mixed or categorical data, and Gower distance, matched to data size, expected shapes, noise, varying density, known k, soft memberships, hierarchy and streaming. Covers preprocessing (scaling, encoding, dimensionality, distance metrics), validation (internal indices, bootstrap stability, external labels, business profiling) and assigning new points.
collaborative-filtering-implementer
Implement collaborative filtering recommenders that beat popularity: item-based kNN with cosine, Jaccard or asymmetric similarity, shrinkage, neighbourhood size, positive-only or mean-centred rating signals and 'because you liked X' explanations; implicit ALS with confidence weighting, BPR pairwise ranking and the closed-form EASE model in plain NumPy, tuned on a temporal validation split by recall@k and NDCG@k and refit on all data; plus user-kNN, explicit matrix factorisation, the implicit/LightFM libraries, hyperparameter ranges, cold-start fallbacks, scaling with sparse matrices and approximate nearest neighbours. Produces top-k lists in the format of the recommendation evaluator.