Diego Garcia

Back to briefs

> var / tempo

Log-Loss Is Not Enough: How Identical Scores Hide an 11-Point ROI Gap

Seven XGBoost training runs on the same La Liga dataset. Five of them land within 0.0025 log-loss of each other, and their away-betting returns are up to 11 percentage points apart. At that spread, log-loss cannot tell you which of them to bet with.

Jun 202612 min read

Cross-entropy loss is the standard evaluation metric for probabilistic classifiers. It penalises confident wrong predictions more than uncertain ones, which makes it a reasonable proxy for calibration quality. In most classification settings, two models within 0.002 log-loss of each other are, for practical purposes, equivalent.

Football betting is an exception. A calibrated distribution only matters here insofar as it finds bets where the model's probability differs from the bookmaker's implied probability by enough to justify a stake. That needs something log-loss does not directly measure: directional edge on specific outcome classes. A model can be well calibrated overall and still be systematically wrong about away wins, and a model with mediocre log-loss can still price away wins better than the market does.

The question is whether XGBoost's hyperparameter search has several local optima that reach similar log-loss through different internal representations, and whether those representations produce different betting value. Across seven walk-forward experiments on KairoX Tempo's La Liga model, they do, by enough to change which model you would deploy.

The Multimodality Problem

Optuna's TPE sampler searches the hyperparameter space efficiently, which is a different thing from finding a single global optimum. On tabular football data, XGBoost's loss surface has at least four distinct basins, each producing a different kind of model with a similar CV log-loss.

The seven La Liga training runs (EXP-002 through EXP-007 plus the production v1.0.0 model) fall into six regimes:

Regimelrn_estdepthLL meanAway ROI
Slow-Deep (EXP-002)0.04448450.9753+5.41%
Fast-Shallow (EXP-003)0.03468350.9764+15.58%
Slow-Shallow-Reg (EXP-004)0.01667430.9770+13.33%
Medium-Slow (EXP-005)0.02056260.9761+16.69%
Medium-Fast (EXP-006)0.08360370.9771+10.42%
Medium (v1.0.0)0.04858180.9746+7.48%
Slow-Deep-ELO (EXP-007)0.01475970.9831+20.38%

Seven training runs, six hyperparameter regimes. Log-loss spread across the top five: 0.0025. Away ROI spread: 11.28pp.

The top five experiments cluster within 0.0025 log-loss of each other. Their away-betting ROI spans from +5.41% to +16.69%. The best log-loss model (v1.0.0, 0.9746) ranks fifth out of seven on Away ROI (+7.48%). The worst overall calibration model (EXP-007, ECE=0.0337) produces the highest Away ROI (+20.38%).

Regime map: hyperparameter space colored by Away ROI

Marker size scales with 1/log-loss; color scales with Away ROI, red (low) to green (high).

-200-10001002003004005006007008009000.010.020.050.10Learning rate (log scale)n_estimatorsEXP-002EXP-003EXP-004EXP-005EXP-006EXP-007v1.0.0
+5.4% (EXP-002)+20.4% (EXP-007)
Figure 1: Each regime occupies a distinct region of the hyperparameter space. Color encodes Away ROI. The best log-loss model (v1.0.0, top-right) is not the best Away ROI model.

The Anti-Calibration Finding

The per-class ECE breakdown is where calibration and returns part ways. EXP-006 is the best-calibrated of the seven runs, with an overall ECE of 0.0193, and its error is even across classes: 0.019 on home wins, 0.019 on draws, 0.020 on away wins. By any calibration metric it is the model you would pick, yet it has the third-lowest Away ROI of the seven.

RunECE overallECE (Away)Away ROI
EXP-006 (Medium-Fast)0.01930.0200+10.42%
EXP-002 (Slow-Deep)0.02510.0168+5.41%
EXP-003 (Fast-Shallow)0.02510.0133+15.58%
EXP-005 (Medium-Slow)0.02920.0256+16.69%
EXP-007 (Slow-Deep-ELO)0.03370.0377+20.38%

Per-class ECE and Away ROI. EXP-003 has the best Away-class calibration (0.0133) and reasonable ROI. EXP-007 has the worst overall calibration and the best returns.

EXP-003 sits in between: it has the best Away-class ECE of all seven runs (0.0133), and it also has positive Away ROI in all three validation seasons (+21.8%, +16.1%, +10.5%). Good away-class calibration does correlate with away-class betting edge here, but the overall ECE ranking tells you very little about which model to deploy.

The Murphy decomposition helps explain this. Brier score decomposes into three terms: reliability (how close predicted probabilities are to observed frequencies), resolution (how much the predictions vary across matches), and uncertainty (the irreducible difficulty of the data). EXP-007 has the worst reliability score (0.001600) and the highest Away ROI. v1.0.0 and EXP-002 have the highest resolution scores (most discriminative predictions overall) but rank among the lowest Away ROI models. High resolution in aggregate does not mean high discrimination on the specific subset of matches where market mispricing exists.

Best calibration and best returns are different regimes

Per-class ECE (left) and the Away-class Murphy decomposition (right), all seven regimes.

Per-class ECE by regime

HDAEXP-002EXP-003EXP-004EXP-005EXP-006EXP-007v1.0.00.0320.0270.0170.0320.0300.0130.0340.0280.0230.0330.0290.0260.0190.0190.0200.0490.0150.0380.0380.0340.017
0.0130.049

Murphy decomposition, Away class

-0.0050.0000.0050.0100.0150.0200.0250.0300.00000.00040.00080.00120.0016Reliability (lower = better)Resolution (higher = better)EXP-002 +5.4%EXP-003 +15.6%EXP-004 +13.3%EXP-005 +16.7%EXP-006 +10.4%EXP-007 +20.4%v1.0.0 +7.5%
Regime
Figure 2: Left: ECE heatmap by regime and class. Right: Murphy decomposition scatter for the Away class. EXP-007 (worst reliability, bottom-right) delivers the best Away ROI. EXP-006 (best overall calibration) sits at mid-range returns.

What the SHAP Profiles Reveal

The clearest evidence comes from the SHAP importances of EXP-002 and EXP-005. The two models differ by 0.0008 log-loss, the smallest gap in the set, yet their feature weights differ by more than 50% on eight of the top twelve features.

FeatureEXP-002 SHAPEXP-005 SHAPRelative diff
away_team_away_win_rate_roll_100.01830.003779.9%
xg_op_delta_roll_30.00450.001565.3%
passes_acc_delta_roll_100.06100.023860.9%
xg_delta_roll_30.00610.002559.5%
matchday0.01600.007255.4%
points_gap0.10460.050252.0%

EXP-002 vs EXP-005: log-loss difference of 0.0008, SHAP divergence up to 79.9%. These are not the same model with different seeds.

EXP-002 concentrates weight on points_gap (0.1046) and passes_acc_delta_roll_10 (0.0610), so it is mostly modelling the gap in league standing and possession. EXP-005 spreads importance more evenly across process metrics, with roughly equal weight on xG, momentum and positional features.

That profile suggests why EXP-002 earns less on away bets. A model that over-weights points gap is encoding a strong home-advantage prior: when the home team is well ahead in the standings, it pushes probability toward the home win. Away wins in La Liga disproportionately happen when the quality gap is smaller or reversed, and EXP-002 is less sensitive to the away team's own momentum features (away_team_away_win_rate_roll_10 SHAP drops 79.9% vs EXP-005). In effect it learns that the table decides matches, which is right on average and misprices the away wins where the edges are.

SHAP feature importance by regime

Mean absolute SHAP, top 12 features. An empty cell marks a feature outside that regime's top 12, not zero importance.

EXP-002EXP-003EXP-004EXP-005EXP-006EXP-007v1.0.0xg_op_delta_roll_10points_gappasses_acc_delta_roll_10ppg_gapaway_team_away_xg_op_roll_5sib_delta_roll_10home_team_home_xg_op_roll_5clearances_delta_roll_10gd_gapxg_delta_roll_10home_ppgh2h_draws_last_50.0760.0730.0660.0670.0570.0930.0800.1050.0430.0380.0500.0360.0970.0610.0260.0240.0240.0310.0690.0530.0580.0730.0620.0710.0460.0350.0310.0230.0370.0320.0450.0310.0310.0270.0290.0300.0240.0330.0340.0270.0250.0200.0270.0250.0330.0280.0260.0210.0220.0200.0130.0300.0300.0380.0370.0290.0130.0080.0160.0180.0180.0220.0150.0340.0240.0220.0260.0170.0160.0110.0130.0170.0140.018
0.0080.105
Figure 3: Mean absolute SHAP by feature and regime. The two columns that stand out are EXP-002 (heavy concentration on points_gap and passes_acc) and EXP-007 (redistributed toward xg and elo signals).

Fold Stability and What It Costs

EXP-004 (Slow-Shallow-Regularized) is the most stable model in the set by log-loss variance across folds (σ=0.0009, six times tighter than EXP-007's σ=0.0065). Its Away ROI across the three validation seasons is +17.4%, +19.4%, +5.9%: two strong seasons and a weaker third, but always positive.

EXP-003 (Fast-Shallow) is also positive in all three seasons, but falls each year: +21.8%, +16.1%, +10.5%. The decline could be real decay (La Liga outcomes becoming more predictable for home wins in recent seasons as the gap between top teams and the rest widens) or simply fold-level variance. More seasons would be needed to distinguish the two.

EXP-006, the best-calibrated model, produces the most stable Away ROI range: +8.6% to +14.5%. If the deployment criterion is minimising the worst-case season rather than maximising expected returns, EXP-006 is a reasonable choice despite its lower peak.

The feature-set confound

EXP-002 (V1 features) and EXP-005 (V4 features) use different feature sets. Part of the ROI divergence between them may come from feature content rather than hyperparameters alone. An ablation using the same feature set with both hyperparameter configurations would isolate the pure hyperparameter effect. That experiment has not been run. The SHAP divergence makes it unlikely that features alone explain the full gap, but it cannot rule that out.

What This Changes About Model Selection

The standard workflow for XGBoost model selection is: run hyperparameter search, pick the configuration with the best CV log-loss, deploy. These runs suggest that workflow is incomplete when the model is used for betting.

Betting returns come only from bets placed, and bets are placed only on the matches where the model's probability exceeds the market's implied probability by a meaningful margin. Log-loss weights the other 85% of matches heavily, and the model's accuracy on those has no effect on returns.

A more complete selection criterion for this use case would include:

  • CV log-loss as the first filter: any model that doesn't clearly beat the uniform baseline is dropped.
  • Per-class ECE as a secondary filter, specifically on the Away class, which is the primary source of market mispricing in La Liga.
  • OOF ROI on Away bets at the target confidence threshold, computed across all three validation seasons, with stability (σ across folds) weighted alongside mean return.
  • SHAP profile review: does the model give meaningful weight to per-team momentum features, or has it collapsed onto standing-based proxies?

None of these checks is new. What these runs show is that on a single dataset, with near-identical log-loss, the gap between the best and worst regime is large enough to triple the away return, from +5.41% to +16.69%, which makes the choice of regime a deployment decision in its own right.

Open question: home ROI across regimes

Most regimes show negative or near-zero Home ROI consistently. That points to a calibration bias toward home wins shared across hyperparameter configurations, which makes it more likely a feature problem than a tuning one. What in the features or data drives the home-win overpricing is still unknown.