Reporting Baselines

#Table template

Always include: the split and its dates, the test size and positive count, the primary metric first, the CI.

#Estimating noise

#Bootstrap CI on a fixed holdout

import numpy as np
from sklearn.metrics import roc_auc_score

def bootstrap_ci(y, p, metric=roc_auc_score, n=2000, seed=0):
    rng = np.random.default_rng(seed)
    y, p = np.asarray(y), np.asarray(p)
    stats = []
    for _ in range(n):
        idx = rng.integers(0, len(y), len(y))
        if y[idx].min() == y[idx].max():
            continue
        stats.append(metric(y[idx], p[idx]))
    return np.percentile(stats, [2.5, 97.5])

#Paired difference vs the baseline

The CI of the difference is what matters; resample the same indices for both models:

def paired_diff_ci(y, p_model, p_base, metric=roc_auc_score, n=2000, seed=0):
    rng = np.random.default_rng(seed)
    y, a, b = map(np.asarray, (y, p_model, p_base))
    d = []
    for _ in range(n):
        idx = rng.integers(0, len(y), len(y))
        if y[idx].min() == y[idx].max():
            continue
        d.append(metric(y[idx], a[idx]) - metric(y[idx], b[idx]))
    return np.mean(d), np.percentile(d, [2.5, 97.5])

"Beats the baseline" means the lower bound of this interval is above 0 (and above the minimum useful lift).

#Repeated CV for small data

RepeatedStratifiedKFold(n_splits=5, n_repeats=10): report mean and standard deviation across the 50 folds, and compare models on the same folds (ml-model-comparison-reporter has the corrected t-test).

#How to phrase the result

Good:

On the April-June 2024 holdout (100 customers, 17 churners) logistic regression reaches PR-AUC 0.56 (95% CI 0.41-0.70) vs 0.38 for the current rule; the paired difference is +0.18 (CI +0.05 to +0.31).

Bad:

The model achieves 85% accuracy.

#Reading mistakes