Skip to contents

Generate and save model validation plots

Source: R/model_validation.R
model_validation.Rd

Runs a configurable set of diagnostic and feature-analysis plots for each requested dataset split and writes the results to structured HTML files inside a directory named after @model_display_name.

Usage

model_validation(
  mb,
  sets = c("train", "test", "holdout"),
  plots = c("validation", "oneway", "pdp", "stability", "shap"),
  validation_bins = 10L,
  one_way_bins = 10L,
  pdp_bins = 10L,
  split = NA_character_,
  filepath = getwd(),
  selfcontained = TRUE,
  seed = 1L
)

Arguments

mb

A modelblueprint object.

sets

[character] Dataset splits to process. Default c("train", "test", "holdout"). NULL splits are silently skipped.

plots

[character] Plot types to produce. Any combination of "validation", "oneway", "pdp", "stability", "shap".

validation_bins

[integer(1)] Bins for the pred-vs-obs calibration chart. Default 10L.

one_way_bins

[integer(1)] Bins for one-way charts. Default 10L.

pdp_bins

[integer(1)] Bins for PDP charts. Default 10L.

split

[character(1)] Column name to segment one-way plots by. NA (default) produces unsplit charts.

filepath

[character(1)] Parent directory for all output. Defaults to getwd(). A subdirectory named @model_display_name is created inside.

selfcontained

[logical(1)] Passed to save_plots(). TRUE (default) embeds all dependencies into each HTML file. Set to FALSE for faster saves during development.

seed

[integer(1)] Seed for the random 50/50 split used by the "stability" plots, applied via withr::with_seed() so the result is reproducible without disturbing the global RNG stream. Default 1L.

Value

The root output directory path, invisibly.

Details

Output layout

<filepath>/
  <model_display_name>/
    <name>.tar.gz
    validation/
      <name>_<set>_validation_plots.html
    oneway/
      <name>_<set>_oneway_plots.html
      <name>_<set>_stability_plots.html   # if "stability" requested
    pdp/
      <name>_<set>_pdp_plots.html
    shap/
      <name>_<set>_shap_plots.html

With selfcontained = FALSE, the HTML files in each subdirectory share a single lib/ dependency folder instead of one folder per file.

Validation plots include a gain chart, predicted-vs-observed calibration chart, and grouped residuals — one HTML file per split.

One-way plots cover every feature in @x_original_inputs. If split is supplied the column is passed to one_way(); otherwise charts are unsplit.

Stability plots are one-way charts split by a random 50/50 variable ("A" / "B"). If the two lines overlap closely the patterns are stable and not driven by random noise.

PDP plots cover every feature in @x_original_inputs.

SHAP plots use shap() with type = "importance". The modelblueprint must have @x_original_inputs set and the model must support kernel SHAP.

The modelblueprint itself is serialised via savemb() into the root output directory.

See also

gain(), pred_vs_obs(), residuals_grouped(), one_way(), pdp(), shap(), save_plots(), savemb()