Why Models Are (or Are Not) Scale-Sensitive

#Distances (k-NN, k-means, DBSCAN, SVM-RBF, hierarchical clustering)

Euclidean distance sums squared differences across features:

d(a, b)² = Σ_j (a_j − b_j)²

A feature with standard deviation 1,000 contributes ~10⁶ times more than a feature with standard deviation 1. The model effectively ignores everything but the widest columns. On the churn fixture total_charges holds 99.9% of the raw distance variance; scaling lifts 15-NN ROC-AUC from 0.58 to ~0.78 (scale_impact.py).

For DBSCAN, eps is in the scaled units: rescaling changes which points are neighbours, so tune eps after scaling.

#Penalties (ridge, lasso, elastic net, regularised logistic regression, SVM)

The penalty λ Σ β_j² (or Σ |β_j|) is unit-dependent: a feature measured in cents needs a coefficient 100x smaller than the same feature in dollars and is penalised 10,000x less under L2. Without scaling, the penalty decides importance by units. Standardise numeric features; for one-hot columns scaling is a choice (it changes how rare levels are regularised).

#Gradients (neural networks, SGD, logistic regression solvers)

Badly scaled inputs give elongated loss surfaces: gradient descent zig-zags and needs tiny learning rates. Standardised inputs make optimisation faster and more stable. lbfgs in LogisticRegression may emit convergence warnings on unscaled data.

#Variance (PCA)

PCA finds directions of maximum variance. Unscaled, the first component is simply the widest column. Standardise (= PCA on the correlation matrix) unless all features share units and their variances are meaningful (e.g. pixel intensities).

#Trees and gradient boosting: invariant

A split x <= t partitions rows the same way as log(x) <= log(t) or (x − μ)/σ <= (t − μ)/σ. Histogram- based GBMs bin by quantiles, also invariant. Scaling changes nothing except, occasionally, tie-breaking and floating-point details. Monotone transforms of the target do change tree models (different loss).

#Linear regression without penalties

Ordinary least squares predictions are invariant to scaling features; coefficients rescale inversely. Scaling helps interpretation (coefficients per standard deviation) and numerical conditioning.

#Naive Bayes, LDA/QDA

Gaussian NB models each feature separately: scale-invariant. LDA is invariant to linear rescaling of features (the classifier adapts); regularised variants are not.

#Quick demonstration (ridge)

import numpy as np
from sklearn.linear_model import Ridge
rng = np.random.default_rng(0)
x1 = rng.normal(0, 1, 500)                # same importance...
x2 = rng.normal(0, 1, 500)
y = x1 + x2 + rng.normal(0, 0.5, 500)
X = np.column_stack([x1, x2 * 1000])      # ...but x2 recorded in different units
print(Ridge(alpha=100).fit(X, y).coef_ * [1, 1000])   # ~[0.85, 0.99]: x1 shrunk, x2 untouched - units drive the penalty