Model diagnostics

Overview

modelblueprint provides four diagnostic charts for evaluating model performance. Each works on a plain data frame or dispatches automatically from a modelblueprint object.

Function	Purpose
`gain()`	Rank-ordering ability — Gini coefficient
`pred_vs_obs()`	Calibration — are predicted rates accurate?
`residuals_grouped()`	Systematic bias across the prediction range
`sami()`	Where do two models disagree?

Gains chart (`gain`)

The gains chart measures how well a model rank-orders risk. Predictions are sorted from highest to lowest, and the chart shows what proportion of total observed losses is captured as we move through the portfolio.

A perfect model captures all losses in the top-risk segment. A random model follows the diagonal. The Gini coefficient summarises rank-ordering ability as a single number between -1 and 1 — higher is better.

# Gains chart — Gini shown in legend. Called without set =, one chart per
# available set (train/test/holdout) is returned as a named list.
gain(mb, set = "train")

When to use it: The gains chart is the primary performance metric in insurance pricing. A Gini above 0.3 is generally considered useful for pricing; above 0.5 is strong.

Predicted vs observed (`pred_vs_obs`)

The calibration chart bins predictions and plots the average predicted rate against the average observed rate per bin. A well-calibrated model has points close to the diagonal — predicted and observed are equal on average.

# Calibration chart
pred_vs_obs(mb, set = "train")

# More bins for finer resolution
pred_vs_obs(mb, bins = 20L)

# Equal-range bins (useful when predictions are skewed)
pred_vs_obs(mb, type_agg = "equal_range")

The chart has three elements:

Yellow bars — exposure per bin (right axis)
Blue dots — observed rate per bin
Dashed black line — predicted rate per bin

When the dots track the dashed line closely, the model is well-calibrated. Systematic divergence at the tails indicates the model underestimates or overestimates extreme risks.

When to use it: Use this after fitting to check that predicted rates match observed rates across the full prediction range. Also useful for comparing a recalibrated vs uncalibrated model.

Grouped residuals (`residuals_grouped`)

The residuals chart bins predictions by exposure and plots grouped residuals — the difference between observed and predicted — with a loess trend line and 95% confidence interval.

A well-specified model has residuals scattered randomly around zero. A systematic pattern indicates model misspecification.

residuals_grouped(mb, set = "train")

# Pearson residuals — scaled by sqrt(pred), useful for count/rate models
residuals_grouped(mb, residual_type = "pearson")

# Control grouping granularity via exposure per bin
residuals_grouped(mb, exposure_per_bin = 500)

The exposure_per_bin argument controls how many data points go into each group — smaller values give more groups and a noisier but more detailed view.

When to use it: Use this to diagnose where a model systematically over- or under-predicts. An upward trend at the right tail means the model underestimates high-risk predictions; a downward trend means it overestimates.

SAMI double lift chart (`sami`)

The SAMI chart compares two competing models by binning the ratio of one model’s predictions to another. For each bin of the ratio, it shows the observed mean alongside both model predictions.

Where the ratio is far from 1, the models disagree substantially. The observed line reveals which model is closer to reality in that region.

# Compare two frequency blueprints
mb1 <- mb_glm_poisson_freq()
mb2 <- mb_lm_regression()   # different model on different data — illustrative

sami(list(mb1, mb2), bins = 10L)

On a plain data frame with predictions already attached:

train         <- mb@train
train$pred1   <- predict(mb, train)
train$pred2   <- predict(mb, train) * runif(nrow(train), 0.8, 1.2)  # perturbed

sami(train, obs = "claim_freq", pred = c("pred1", "pred2"),
     exposure = "exposure", bins = 10)

With recalib = TRUE, both predictions are scaled to match the observed mean before computing ratios — this isolates shape differences from overall level differences.

sami(list(mb1, mb2), bins = 10L, recalib = TRUE)

When to use it: SAMI is most useful when deciding between two candidate models. It shows not just which model performs better overall, but where one model is better and by how much. The regions where the observed line tracks one model more closely than the other drive the decision.

Running everything at once (`model_validation`)

Rather than calling each diagnostic individually, model_validation() runs the full suite for every available set (train/test/holdout) and saves the results as structured HTML files — one file per plot type per set — inside a directory named after @model_display_name:

model_validation(mb, filepath = "~/model_reviews")

~/model_reviews/
  glm_poisson_freq/
    glm_poisson_freq.tar.gz          # the serialised blueprint (savemb)
    validation/              # gain + calibration + residuals per set
    oneway/                  # one-way and stability plots per set
    pdp/                     # partial dependence plots per set
    shap/                    # SHAP importance per set

Use plots to run a subset (e.g. plots = c("validation", "oneway")) and sets to restrict the splits. During development, selfcontained = FALSE saves much faster: the HTML files in each subdirectory then share a single lib/ dependency folder instead of embedding everything in every file.

Choosing the right diagnostic

Question	Diagnostic	Key metric
How well does the model rank-order risk?	gain()	Gini coefficient
Are predicted rates accurate?	pred_vs_obs()	Predicted vs observed alignment
Does the model have systematic bias?	residuals_grouped()	Residual pattern around zero
Which of two models is better, and where?	sami()	Ratio of predictions vs observed

Overview

Gains chart (gain)

Predicted vs observed (pred_vs_obs)

Grouped residuals (residuals_grouped)