Metrics for Imbalanced Problems

#Why accuracy fails

At 2% prevalence "always negative" is 98% accurate and catches nothing. Any metric dominated by true negatives hides the minority class.

#ROC vs PR

A model can have ROC-AUC 0.95 and PR-AUC 0.2 at 0.5% prevalence: thousands of false positives per true one. Report both, decide on PR-AUC or the operating-point metric.

#Operating-point metrics

import numpy as np
def at_k(y, s, k):
    idx = np.argsort(-s)[:k]
    tp = y[idx].sum()
    return {"precision@k": tp / k, "recall@k": tp / y.sum(), "lift@k": (tp / k) / y.mean()}

Expected cost at a threshold: (FP * C_fp + FN * C_fn) / n.

#Few positives: confidence intervals

With 34 positives the standard error of recall is ~sqrt(r(1−r)/34): at r = 0.7 that is ±0.08 (one SE). Always attach intervals:

def bootstrap(y, s, metric, n=2000, seed=0):
    rng = np.random.default_rng(seed); out = []
    pos, neg = np.where(y == 1)[0], np.where(y == 0)[0]
    for _ in range(n):
        idx = np.concatenate([rng.choice(pos, len(pos)), rng.choice(neg, len(neg))])   # stratified resample
        out.append(metric(y[idx], s[idx]))
    return np.percentile(out, [2.5, 97.5])

Stratified bootstrap keeps the positive count fixed so intervals are not distorted by resamples with too few positives.

#CV with few positives

• StratifiedKFold so each fold has its share of positives.
• RepeatedStratifiedKFold(n_splits=5, n_repeats=5) to average out fold noise.
• Pool out-of-fold predictions (cross_val_predict) and compute PR-AUC once on all of them, in addition to per-fold means: per-fold PR-AUC with 7 positives is extremely noisy.

#Multiclass with rare classes

• Per-class precision/recall/F1 (classification_report).
• Macro averages so rare classes count equally; weighted averages for frequency-weighted views.
• One-vs-rest PR-AUC per class.
• Confusion matrix normalised by true class (normalize="true").

#Summary metrics that respect both classes