# Matbench Regret (Offline)

This document defines the regret metrics we use for Matbench-style evaluations and how to
reproduce results offline with CSVs. The approach is conservative: it reports uncertainty,
uses fixed seeds, and avoids comparative claims by default.

Terminology
- Objective (`y_true`): numeric property to maximize or minimize (declared via `--mode`).
- Score: ranking function for selection. By default, we use SRMF proxies: `kappa − λ·leak`.
- Regret@k: Difference between oracle top‑k utility and model-selected top‑k utility, with
  normalized variant dividing by oracle utility.
- AURC: Area under the normalized regret curve over k (lower is better).

Offline tools
- Calibration: choose λ to minimize validation AURC, report held‑out test AURC with CIs.
  - `tools/calibrate_matbench_srmf.py`
- Evaluation: compute Regret@k and AURC using the chosen λ (or a provided score column).
  - `tools/eval_matbench_regret.py`

CSV schema
- Required features for SRMF: `band_gap, density, nsites, formation_energy_per_atom`.
- Required objective: e.g., `y_true` (set via `--objective-col`).
- Optional: `material_id` and `formula_pretty` for reporting.

Reproducible run
- Calibrate:
  - `python tools/calibrate_matbench_srmf.py --path data.csv --objective-col y_true --mode max --topk-grid 1,5,10 --lambda-grid 0.0,0.5,1.0 --bootstrap 1000 --seed 0 --out-json reports/matbench_calibration.json --scores-out reports/matbench_scores_test.csv`
- Evaluate with tuned λ:
  - `python tools/eval_matbench_regret.py --path data.csv --objective-col y_true --mode max --use-srmf --lambda-weight $(jq -r .best_lambda reports/matbench_calibration.json) --topk-grid 1,5,10 --out-csv reports/matbench_regret.csv --out-json reports/matbench_regret.json --bootstrap 1000 --seed 0`

Claims and limitations
- SRMF mapping is a proxy for manifold geometry and stability signals; it is not a surrogate for
  ab initio calculations. We report uncertainty, avoid comparative claims by default, and keep
  live integrations gated.



## Attestation and Groups
- Attestation: tools/generate_matbench_attestation.py
- Per-group regret: tools/eval_matbench_regret.py --group-col group --out-group-csv reports/matbench_regret_groups.csv



## Baselines and Layers
- Baseline CV regret CLI: tools/eval_baseline_regret.py
- Emergence exploration: tools/explore_matbench_layers.py
- Example: use quantile layers on band_gap and report AURC per-layer.