Transforms and Outliers

#Variance-stabilising transforms

Rule of thumb: if the domain relationship is multiplicative ("10% more spend") use log; if you just need to tame a tail for a linear model, Yeo-Johnson or quantile.

#Detecting outliers before choosing a scaler

Distinguish errors (fix or drop), sentinels (convert to missing) and genuine extremes (keep, then choose a robust scaler or transform).

#Winsorising (clipping)

lo, hi = X_train["amount"].quantile([0.01, 0.99])
X_train["amount"] = X_train["amount"].clip(lo, hi)
X_test["amount"] = X_test["amount"].clip(lo, hi)        # same bounds, learned on train

Implement as a transformer so bounds are learned per fold:

from sklearn.base import BaseEstimator, TransformerMixin
import numpy as np

class QuantileClipper(BaseEstimator, TransformerMixin):
    def __init__(self, lower=0.01, upper=0.99):
        self.lower, self.upper = lower, upper

    def fit(self, X, y=None):
        X = np.asarray(X, dtype=float)
        self.lo_ = np.nanquantile(X, self.lower, axis=0)
        self.hi_ = np.nanquantile(X, self.upper, axis=0)
        return self

    def transform(self, X):
        return np.clip(np.asarray(X, dtype=float), self.lo_, self.hi_)

#Binning

KBinsDiscretizer(n_bins=10, strategy="quantile", encode="onehot") turns a skewed column into robust indicators. Useful for linear scorecards (credit) and when the effect is step-like; costs information.

#Target transforms for regression

from sklearn.compose import TransformedTargetRegressor
from sklearn.linear_model import Ridge
import numpy as np

model = TransformedTargetRegressor(regressor=make_pipeline(prep, Ridge()), func=np.log1p, inverse_func=np.expm1)

• Predictions are back-transformed automatically.
• Retransformation bias: exp(E[log y]) <= E[y]. For totals multiply predictions by the smearing factor mean(exp(residuals_on_log_scale)) (Duan 1983), or fit a Gamma/Tweedie GLM/GBM on the raw target instead.
• Metrics must be computed on the original scale for business reporting.

#Choosing between options empirically

Compare two or three transforms on the same CV folds with the final model; pick the simplest that is not worse. For tree models do not bother: they are invariant to monotone transforms of features (but not of the target).