Ranking, Recommendation, Survival, Uplift and Anomaly Metrics

#Ranking

#DCG and NDCG@k

DCG@k  = Σ_{i=1..k} (2^{rel_i} − 1) / log2(i + 1)      # exponential gain (common in LTR)
IDCG@k = DCG@k of the ideal ordering
NDCG@k = DCG@k / IDCG@k                                  # per query, then averaged over queries

Queries with no relevant items have IDCG = 0: decide explicitly whether they count as 0, 1 or are excluded (LightGBM's lambdarank counts them as 1 by default). Linear gain rel_i instead of 2^{rel}-1 is the other convention: state which one you use.

#MRR

Mean of 1 / rank of the first relevant item. Use when one right answer matters (navigational search, QA).

#MAP@k

Mean over queries of average precision within the top k. Binary relevance.

#Recall@k / hit rate@k

Share of relevant items (or of queries with at least one relevant item) appearing in the top k.

import numpy as np
from sklearn.metrics import ndcg_score
ndcg_score([true_relevance_per_query], [scores_per_query], k=10)   # one query per row; pads must be equal length

#Recommendation

Offline gains in recall@k often do not transfer online; treat CTR/conversion/retention from an A/B test as the real success metric.

#Survival

#Concordance index

Share of comparable pairs (the earlier time is an observed event) where the model ranks risk correctly.

• Harrell's C: biased under heavy censoring.
• Uno's C: inverse-probability-of-censoring weighted; preferred when censoring is heavy or depends on time.

from lifelines.utils import concordance_index
concordance_index(durations, -risk_scores, events)       # higher risk -> shorter time, so negate
from sksurv.metrics import concordance_index_ipcw         # Uno's C

#Time-dependent AUC and integrated Brier score

• cumulative_dynamic_auc (scikit-survival) at the decision horizon (e.g. 90 days).
• Integrated Brier score over [t_min, t_max]: calibration + discrimination of the survival curve.

#Uplift

#Uplift curve and Qini

Sort by predicted uplift; at each fraction φ of the population:

Qini(φ) = Y_T(φ) − Y_C(φ) · N_T(φ) / N_C(φ)          # incremental positives among the top φ

• Qini coefficient: area between the Qini curve and the random-targeting line, normalised by the area for the ideal model.
• AUUC: area under the uplift curve (conversion-rate difference version).
• Uplift@K: (conversion_T − conversion_C) among the top K.

Valid only on randomised data (RCT or proper randomisation). See uplift-model-builder for code.

#Anomaly detection

#Clustering (diagnostics, not success)

Success for clustering is that segments lead to different actions and remain stable over time.