feature-scaling-advisor

Scaling matters for any model that uses distances (k-NN, k-means, SVM with RBF, DBSCAN), penalties (ridge, lasso, elastic net, regularised logistic regression), gradients (neural nets, SGD) or variance (PCA). It does nothing for trees and gradient boosting, which split on thresholds and are invariant to monotone transforms. The choice of scaler follows the column's shape: symmetric -> standard; outliers -> robust; right-skewed -> log/Box-Cox/Yeo-Johnson; heavy tails -> quantile; sparse -> max-abs. scripts/scaling_plan.py recommends a transform per column and emits the ColumnTransformer; scripts/scale_impact.py shows the effect on a k-NN model with cross-validation.

#Trigger Conditions

Activate this skill when the user:

• Asks "should I normalise / standardise?", "which scaler?", "MinMax or Standard?"
• Uses k-NN, k-means, SVM, PCA, regularised linear models or neural nets and gets odd results
• Has skewed, heavy-tailed or outlier-heavy numeric features, or a skewed regression target
• Wonders whether to scale for XGBoost / LightGBM / random forest (no)
• Suspects scaling leakage (scaler fitted on the full dataset before CV)

#Workflow

1. MODEL     -> tree-based? stop: no scaling (keep the raw units for interpretation)
2. MEASURE   -> python scripts/scaling_plan.py data.csv --model linear --target y --drop id
3. CHOOSE    -> per column: standard / robust / log1p+standard / power / quantile / passthrough
4. BUILD     -> python scripts/scaling_plan.py ... --code   (ColumnTransformer inside a Pipeline)
5. EVIDENCE  -> python scripts/scale_impact.py data.csv --target y   (raw vs scaled k-NN, CV)
6. TARGET    -> skewed regression target? TransformedTargetRegressor(log1p) or a Tweedie/Gamma loss
7. VERIFY    -> scaler fitted on training folds only; same transform at serving time

#Which Models Need Scaling

scale_impact.py on the churn fixture: raw 15-NN ROC-AUC 0.58 (total_charges holds 99.9% of raw distance variance) vs 0.76-0.78 scaled.

#Scaler Selection by Column Shape

PowerTransformer standardises its output by default (standardize=True). QuantileTransformer is non-linear and distorts distances between far-apart values; prefer it for heavy-tailed inputs to neural nets or linear models, not when relative magnitudes matter.

Transforms, formulas and their failure modes: transforms-and-outliers.md.

#Scaling Without Leakage

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, PowerTransformer
from sklearn.linear_model import LogisticRegression

prep = ColumnTransformer([
    ("std", StandardScaler(), ["tenure_months", "support_calls"]),
    ("pow", PowerTransformer(method="yeo-johnson"), ["total_charges"]),
], remainder="passthrough")
model = Pipeline([("prep", prep), ("clf", LogisticRegression(C=1.0, max_iter=2000))])
# cross_val_score(model, X, y, cv=cv) refits the scalers on every training fold.

• Never scaler.fit(X_all) before splitting or before cross-validation: test-fold means and variances leak into training. The effect is usually small for standard scaling and larger for quantile transforms, target-dependent steps and small datasets, but it is always avoidable.
• Fit once on training data, persist the fitted pipeline, apply the same object at serving time.
• Time series: fit on the past window only; refit at each retrain.

More leakage cases and serving issues: scaling-pitfalls.md.

#Regression Targets

Log-target models predict something closer to the median; when sums matter (total revenue), correct the retransformation bias (smearing) or use a Gamma/Tweedie loss. Details: transforms-and-outliers.md.

#Output Format

When advising on scaling, answer with:

1. Does this model need scaling? - yes/no with the reason (distance, penalty, gradient, variance, or tree invariance).
2. Per-column plan - the scaling_plan.py table: shape statistics and the chosen transform.
3. Code - the ColumnTransformer inside a Pipeline.
4. Evidence - scale_impact.py result (raw vs scaled), or CV comparison of two scalers.
5. Target handling - for regression with a skewed target.
6. Serving notes - persisted fitted transformer, behaviour on out-of-range values.

#Scripts

python scripts/scaling_plan.py data.csv --model linear --target price --drop house_id
python scripts/scaling_plan.py data.csv --model tree
python scripts/scaling_plan.py data.csv --model nn --target y --json
python scripts/scaling_plan.py data.csv --model linear --target y --code > prep.py

Models: tree, linear (also SVM, k-NN, k-means, PCA), nn, sparse. Per numeric column: min, max, mean, std, median, IQR, skew, robust outlier share (|x - median| / (1.4826 MAD) > 3.5), zero share, negatives, distinct count; the transform with the reason; and the largest/smallest standard-deviation ratio across columns. --code prints a ColumnTransformer of median imputation + scaler per group with unique step names. Exit codes: 0 printed, 2 usage error or no numeric columns.

python scripts/scale_impact.py data.csv --target churned --drop customer_id
python scripts/scale_impact.py data.csv --target price --k 10 --folds 5 --max-rows 1500 --json

Pure-Python k-NN with K-fold CV on the numeric columns under raw, standard, robust, log1p+standard and min-max scaling (fitted on each training fold): ROC-AUC for binary targets, MAE for numeric ones; plus each column's share of raw squared-distance variance. Exit codes: 0 measured, 2 usage error or not enough numeric rows.

#Additional Resources