Reading Learning Curves

#The decomposition

For a fixed model and data distribution, as the training set size n grows:

• the training score falls (it is harder to fit more rows), approaching the best achievable score for this model class from above;
• the validation score rises towards the same limit from below;
• the gap (train - validation) is the variance part of the error; the distance from the common limit to the perfect score is bias plus irreducible noise.

So the curve answers three questions:

#Canonical shapes

HIGH VARIANCE, DATA-LIMITED        HIGH VARIANCE, FLAT              HIGH BIAS
score                               score                             score
1.0 |T T T T T                      1.0 |T T T T T T                   |
    |          T  T                     |                              |T T T
    |              v  v                 |    v v v v v v               |      T T T T
    |         v  v                      |  v                           |  v v v v v v  (both low, met)
    |   v  v                            |v                             |v
    +---------------- n (log)           +---------------- n            +---------------- n

#Metric conventions

• scikit-learn scorers are higher is better; losses appear as neg_* (e.g. neg_root_mean_squared_error around -35 000). diagnose_curve.py uses that convention by default; use --lower-better for curves stored as positive losses.
• The default gap threshold is 0.02 x max(1, |score|): two points of AUC or accuracy, 2% of an RMSE. Set --gap-threshold from what matters for the product (e.g. 0.01 AUC for a mature fraud model).
• For accuracy on imbalanced data the curve can look converged at the majority-class rate: use AUC, average precision or log loss.

#Common misreadings

#Learning curves for special data

• Time series: TimeSeriesSplit with shuffle=False: the smallest size is the earliest window, so the curve also contains time drift. A falling validation curve then can mean the older rows are less relevant; test a recency-weighted or a rolling-window model.
• Grouped data (customers, patients): use the number of groups, not rows, as the size you plan to collect.
• Imbalanced targets: what helps is more positives; report the number of positives at each size.