Peer Review Package

This document is a reviewer-oriented guide to reproduce and scrutinize Compitum’s results. It defines metrics, baselines, data, and statistical procedures, and provides one-shot commands to regenerate the exact artifacts used in our tables and report.

Reviewer Quickstart

  • Verify environment and tests

    • Python 3.11+ in a fresh venv

    • pip install -e .[dev]

    • make check (ruff + mypy + bandit + pytest)

  • Regenerate end-to-end artifacts (Windows, one-shot)

    • make peer-review (invokes scripts\run_peer_review.bat)

  • Inspect primary outputs

    • reports/fixed_wtp_summary.md (table with CIs for WTP = 0.1, 1.0)

    • reports/report_release.html (frontier plots, baselines, ablations)

    • reports/mutation_summary.json (Cosmic Ray summary; score and outcomes)

Data Availability and Artifacts

  • Inputs: RouterBench task inputs are locally cached and follow their upstream licenses. We do not redistribute proprietary datasets in this repository or packages.

  • Outputs: Scripts write local JSON/CSV/HTML and a SHA-256 manifest under reports/ and docs/.

  • Network: The peer-review pipeline operates offline. No judge-based model calls are used.

  • Optional: LLM briefing pack

    • Snapshot for context: https://compitum.space/docs/repo_snapshot.jsonl

    • Instruction: Use JSONL lines with type, path, content; cite path:line.

Claims and Contributions (Auditable)

  • Low regret at fixed budget: At given willingness-to-pay (WTP) budgets, Compitum achieves lower mean regret versus strong baselines, while respecting hard constraints.

  • Deterministic and reproducible: 100% line+branch test coverage; Hypothesis derandomized; fixed seeds for synthetic demo fits; scripts produce immutable, checksummed artifacts.

  • Mechanistic routing certificate: Each decision emits a structured certificate with utility components, constraint status and approximate local shadow prices, boundary diagnostics, and drift monitors.

  • Stable online updates: A Lyapunov-inspired trust-region controller caps effective step sizes using EMA and integral signals; we do not claim a formal Lyapunov proof in this release.

Near-Frontier Behavior (Interpretation)

  • When Compitum is close to the cost-quality frontier, the average gap to frontier is small and the at-frontier percentage is high, even if envelope wins are rare.

  • This indicates Compitum trades cost and performance comparably to the strongest baseline set at the given WTP (lambda), while respecting hard constraints and offering mechanistic diagnostics.

  • We therefore report both per-baseline win rates at fixed WTP and the frontier gap summary to capture near-frontier behavior without over-claiming envelope dominance.

Methodological Limitation: Flat Benchmarks vs. Curved-Manifold Analysis

Compitum’s core mechanism (SymbolicManifoldMetric, the SPD/Riemannian distance structure, the Lyapunov-based trust-region controller) is designed to exploit curved structure that emerges from sequential, evolving decisions – drift over time, evidence accumulating across related queries, a metric that adapts as it observes more of the space. Neither RouterBench nor MATBENCH is that kind of data: both are static, pre-baked, tabular benchmark repositories, evaluated as one-shot lookups rather than a live, evolving sequence. You cannot retroactively impose curved-manifold structure on a benchmark that was collected and organized flat. This is not a defect discovered in this repository’s implementation – it is a scope mismatch between what these two benchmarks measure and what the geometric mechanism is actually built to exploit, and it is part of why the project’s earlier push toward room-temperature-superconductivity materials search was paused: the same flat-repository limitation applies there too.

Concretely, on RouterBench: win rate is 0.0% at both audited WTP slices (reports/fixed_wtp_summary.md) – reported honestly, not hidden, and read below as evidence about benchmark fit, not routing correctness (constraint compliance and the mechanistic certificate remain intact regardless).

Concretely, on MATBENCH’s own demo-scale calibration (reports/matbench_calibration.json, a small n=200 synthetic split – not a full materials-science benchmark run, see the section below): the calibration step’s own lambda grid search selected best_lambda = 0.0, meaning it correctly detected that any SRMF shadow-price weighting made results worse on this split (validation AURC rises from 0.062 at lambda=0 to 0.93+ at lambda=0.5 or 1.0), and a plain ridge/lgbm baseline achieves essentially zero regret (reports/matbench_baseline_regret.json, AURC=0.0) versus SRMF’s own calibrated result (reports/matbench_regret.json, AURC=0.048). That the calibration honestly chose to disable itself rather than force a worse lambda is the pipeline working correctly – it is evidence the mechanism for detecting “does the geometric structure help here” is sound, even though the answer on this flat data is “not much.” Read both of these results as calibration and constraint-compliance checks, not as proof of the geometric routing advantage – that advantage needs a genuinely sequential or live-feedback setting to show up, which neither of these benchmarks provides by construction.

ASCII Notation (Quick Reference)

  • performance in [0, 1]; total_cost >= 0; lambda >= 0

  • U = performance - lambda * total_cost

  • regret = U_best_baseline - U_compitum

  • WTP slices: lambda in {0.1, 1.0}; sensitivity grid {1e-4, 1e-3, 1e-2, 0.1, 1.0, 10.0}

Constraints Summary

  • Default constraints live at configs/constraints_us_default.yaml and are applied as linear inequalities A x <= b over process variables (e.g., usage, availability, policy limits).

  • The routing certificate reports feasible and per-constraint approximate local shadow prices (finite-difference viability diagnostics) so you can see which limits are active and how they shape utility.

  • We report a constraint compliance rate (target ~100%) alongside results.

Fair Evaluation Notes

  • Panel definition: see Panel Summary for tasks, models, WTP slices, and eval unit counts.

  • Seeds and determinism: seeds are fixed across baselines and Compitum; scripts and demo fits accept --seed.

  • Cost accounting: identical token/cost accounting across baselines and Compitum for a given WTP (lambda).

  • Best baseline: computed per evaluation unit at fixed WTP; comparisons are paired.

  • Configs: evaluation YAMLs live under data/rb_clean/ (e.g., evaluate_routers.yaml, evaluate_routers_multitask.yaml).

Evaluation Protocol (Preregistered)

  • Utility and regret

    • Let performance be task score in [0,1]; cost is token or dollar cost; WTP lambda >= 0.

    • Utility per evaluation unit: U = performance - lambda * total_cost.

    • Regret per unit: r = U_best_baseline - U_compitum (lower is better).

  • Primary budgets (WTP)

    • Fixed slices: lambda in {0.1, 1.0} for headline table; sensitivity grid {1e-4, 1e-3, 1e-2, 0.1, 1.0, 10.0} for frontier plots.

  • Primary metrics (per task and panel averages)

    • Mean regret and P95 regret (tail behavior)

    • Win rate: fraction of evaluation units where U_compitum >= U_best_baseline

    • Avg cost delta on wins: E[cost_compitum - cost_best_baseline | win]

    • Constraint compliance rate: fraction feasible (should be 100% by design)

  • Statistical procedure

    • Evaluation unit: a single (prompt, task) item under a given lambda.

    • Nonparametric bootstrap (1,000 resamples) over evaluation units; report 95% percentile CIs.

    • Paired significance: all deltas computed per unit before aggregation.

    • Seeds fixed; any randomized baselines evaluated with the same seeds/grid.

Acceptance criteria (example)

The primary, load-bearing claim is mean regret, not win rate – see Near-Frontier Behavior above and the Methodological Limitation note below for why envelope wins are expected to be rare on flat, one-shot benchmark data. The win-rate bullet below is illustrative methodology (what a reviewer could also check), not a pass/fail bar this release is claimed to clear – the current run’s win rate is 0.0% at both audited WTP slices (see Results (Fill-In Table)), which is consistent with, not a violation of, the mean-regret claim above.

  • Panel average mean regret lower than best baseline at lambda in {0.1, 1.0} (the actual claim)

  • Constraint compliance rate >= 99.9% (the actual claim)

  • Win rate > 50% with non-overlapping 95% CI vs. parity for at least one lambda slice (illustrative secondary methodology only; not expected to hold on flat benchmark data by this document’s own Near-Frontier framing)

Datasets, Tasks, and Baselines

  • Tasks (bounded panel)

    • Grade-school math, HellaSWAG, MBPP, selected MMLU subtopics. The exact panel is configured and cached; see scripts for the manifest.

    • Licensing: RouterBench inputs follow their upstream licenses; we do not redistribute proprietary content.

  • Baselines

    • KNN/MLP/cascade gates and common RouterBench routers (budget-aware and naive). “Best baseline” is defined per evaluation unit at fixed lambda.

    • Fairness: equal prompt sets, identical lambda, identical token accountings; report any hyperparameter deviations.

MATBENCH: Materials Property Regret

A second, parallel evaluation surface alongside RouterBench – featured here, not a replacement for it. See the Methodological Limitation note above before reading these numbers as more than a calibration/mechanism check: this is a flat, tabular benchmark repository, the same limiting case as RouterBench, evaluated for the same honest reasons (real, reproducible, checked-in evidence) rather than as proof of a geometric routing advantage.

  • Pipeline: tools/calibrate_matbench_srmf.py (honest train/val/test split, held-out test scores) -> tools/eval_matbench_regret.py (regret/AURC on the held-out split) and tools/eval_baseline_regret.py (ridge/lgbm k-fold out-of-fold baseline, same regret metric, --mode max|min for the task’s optimization direction). See Matbench (Offline Adapter), Matbench Regret (Offline), and How We Evaluate Matbench (Offline) for the full mechanics.

  • Current committed evidence (reports/matbench_calibration.json, matbench_regret.json, matbench_baseline_regret.json) is from data/matbench_demo.csv, a small synthetic demo split (train=120, val=40, test=40) used to validate the pipeline’s own mechanics – not a full Materials Project evaluation. A real end-to-end run (--from-mp, with presets for kagome/ nickelate/FeSe chemistries) requires a Materials Project API key, which is not currently available in this environment; that run remains outstanding rather than simulated.

  • Reproduce the demo-scale pipeline end to end: make matbench-pipeline (chains matbench-demo -> matbench-calibrate -> matbench-eval -> matbench-attest; see Makefile).

  • Artifacts and provenance: tools/generate_matbench_attestation.py produces a hash-manifested attestation (reports/matbench_attestation.json) of the calibration + regret run, following the same compitum.release-attestation/v2-style schema as the rest of this document’s evidence chain.

One-Shot Reproduction

We provide Windows-first commands (cross-platform alternatives below). All scripts write JSON/HTML/CSV artifacts and a manifest with checksums and provenance.

  1. Quality gates (lint, type, tests)

scripts\run_quality.bat

Artifacts: reports\quality_*.json (mypy/ruff/pytest logs).

Optional: RouterBench Tests (separate venv)

RouterBench integration tests run in a separate virtual environment to avoid import conflicts. They are optional and excluded from the default PyTest profile.

Windows:

CALL .\.venv-routerbench\Scripts\activate.bat
set PYTHONPATH=%CD%;%CD%\src
python -m pytest -q -m routerbench

Linux/macOS:

source .venv-routerbench/bin/activate
export PYTHONPATH="$PWD:$PWD/src"
python -m pytest -q -m routerbench

Notes:

  • Run without coverage enforcement for this subset (coverage fail-under=100 can trip when only routerbench tests run). If you do append coverage, pass --cov=compitum --cov-branch --cov-append and ensure fail-under is overridden.

  • The peer-review scripts already handle RouterBench evaluations separately using an isolated environment and PYTHONPATH.

  1. Evaluate baselines and Compitum on bounded panel

scripts\validate_compitum.bat

Artifacts: reports\routerbench_report.md, per-task CSVs under data\rb_clean\eval_results\....

  1. Fixed-WTP summary with 95% CIs (peer-review table)

set PYTHONPATH=%CD%;%CD%\src
.\.venv-routerbench\Scripts\python tools\analysis\fixed_wtp_ci.py ^
  --input data\rb_clean\eval_results\<latest-compitum-csv>.csv ^
  --wtps 0.1 1.0 --bootstrap 1000 ^
  --out-json reports\fixed_wtp_summary.json --out-md reports\fixed_wtp_summary.md

Artifacts: machine-readable JSON and a human-readable Markdown table.

  1. Consolidated HTML report (frontier plots, tables)

set PYTHONPATH=%CD%;%CD%\src
.\.venv-routerbench\Scripts\python tools\ci_orchestrator.py --all ^
  --config data\rb_clean\evaluate_routers.yaml --max-evals 0 ^
  --wtp-list "0.0001,0.001,0.01,0.1,1.0,10.0" --report-out reports\report_release.html

Artifacts: reports\report_release.html

Cross-platform shortcuts

make peer-review   # calls scripts\run_peer_review.bat

POSIX equivalents (community-maintained)

# Create and activate venv
python3 -m venv .venv
. .venv/bin/activate

# Install
pip install -e .[dev]

# Quality gates
ruff check .
mypy src/compitum
bandit -q -r src/compitum -x src/routerbench
pytest -q

# Fixed-WTP summary (adjust latest compitum CSV path as needed)
PYTHONPATH="$PWD:$PWD/src" python tools/analysis/fixed_wtp_ci.py \
  --input data/rb_clean/eval_results/<latest-compitum-csv>.csv \
  --wtps 0.1 1.0 --bootstrap 1000 \
  --out-json reports/fixed_wtp_summary.json --out-md reports/fixed_wtp_summary.md

# Consolidated HTML report
PYTHONPATH="$PWD:$PWD/src" python tools/ci_orchestrator.py --all \
  --config data/rb_clean/evaluate_routers.yaml --max-evals 0 \
  --wtp-list "0.0001,0.001,0.01,0.1,1.0,10.0" --report-out reports/report_release.html

# Generate docs tables and build
python tools/generate_eval_tables.py
python -m sphinx -b html docs docs/_build/html

Environment, Seeds, and Determinism

  • Python >= 3.11; create an isolated venv; install with pip install -e .[dev].

  • Determinism: Hypothesis tests set derandomize=true; demo fits accept --seed; orchestration scripts fix seeds.

  • Provenance: a manifest records git SHA, tool versions, WTP grid, and SHA-256 of generated artifacts.

Engineering note: We follow the same Control‑of‑Error ethos in development. Invariants + Hypothesis, mutation testing, strict docs gates, and audience‑specific analyses (CEI, reliability, decision/control KPIs) provide immediate, judge‑free feedback for the codebase. During release preparation we briefly edited and promptly reverted one source file; all results and reports here were generated from the current frozen release-candidate commit with fixed seeds and a recorded environment.

Mutation Testing (mutmut + Cosmic Ray)

Mutation testing runs two complementary tools rather than a single aggregate score: mutmut (the primary per-file certification tool, sharded across a matrix of 20 src/compitum/ modules in .github/workflows/mutation.yml) and Cosmic Ray (a secondary cross-check with narrower, flat-module scope). A single “mutation_score: 1.0000”-style number is not, by itself, a certification – see MUTATION_HARDENING_STATUS.md at the repo root for the authoritative, per-file accounting: every survivor is classified as a genuine defect (fixed), a proven equivalent mutant, or an explicitly accepted defensive/unreachable case with its exact source location and reasoning recorded, rather than collapsed into a bare percentage.

Current status: every release-critical module in the 20-file matrix is fully classified. Exactly 2 survivors remain, both explicitly accepted (not gaps): constraints.py ID 62 and metric.py ID 125, each a documented defensive/pragma-exempted case. A fresh CI-run re-verification of the full matrix against this exact commit has not yet completed as of this writing (the mutation workflows were recently repaired – a missing Cosmic Ray baseline step, a silent-pass bug in gate-checking, and shard-identity/nested-path bugs were all found and fixed this pass).

  • Per-file detail, survivor dispositions, exact commits: MUTATION_HARDENING_STATUS.md

  • Aggregate summary (JSON, when a sweep has run): reports/mutation_summary.json

  • Cosmic Ray raw dump (NDJSON, when a sweep has run): reports/cr_report.json

Cross-platform notes

  • Linux/macOS: replace Windows virtualenv paths with your venv’s python; keep PYTHONPATH to include src/.

  • Some RouterBench integrations are heavy; our scripts use a bounded panel and cached inputs to keep runtime manageable.

Results (Fill-In Table)

Populated from a real, fresh RouterBench evaluation run against this exact 0.2.0 candidate (not stale data – reproduces the previously-committed panel-average numbers almost exactly). Per-task rows are single point estimates (this repo’s shipped bootstrap tooling resamples over tasks, i.e. eval_name values, as the evaluation unit – see tools/analysis/fixed_wtp_ci.py – so a single-task row has nothing further to resample; only the Panel Avg row has a genuine 95% bootstrap CI, carried over unchanged from reports/fixed_wtp_summary.md). “lambda Utility” is not a metric computed by any script in this repo – left as N/A rather than fabricated; if this column is meant to mean something specific, it should be defined or removed in a future pass. “Avg Cost delta (wins)” is undefined everywhere here since win rate is 0.0% (no wins to compute a cost delta over).

WTP = 0.1

Task

Best Baseline Utility

Compitum Utility

lambda Utility

Mean Regret

P95 Regret

Win Rate

Avg Cost delta (wins)

grade-school math

-3.838395

0.388201

N/A

4.226597

4.226597

0.0%

N/A

hellaSWAG

-5.931090

0.663423

N/A

6.594513

6.594513

0.0%

N/A

MBPP

0.392420

0.846178

N/A

0.453759

0.453759

0.0%

N/A

Panel Avg

0.249926

0.850629

N/A

0.601950 [0.456828, 0.801042]

1.011092

0.0% [0.0%, 0.0%]

N/A

Panel Avg’s Mean Regret carries its real 95% bootstrap CI (1000 resamples over the 86-task panel); per-task rows above it are single-point values, not independently bootstrapped.

WTP = 1.0

Task

Best Baseline Utility

Compitum Utility

lambda Utility

Mean Regret

P95 Regret

Win Rate

Avg Cost delta (wins)

grade-school math

-44.347208

-0.281866

N/A

44.065342

44.065342

0.0%

N/A

hellaSWAG

-62.135830

-1.015686

N/A

61.120145

61.120145

0.0%

N/A

MBPP

-1.850978

0.663190

N/A

2.514167

2.514167

0.0%

N/A

Panel Avg

-2.100669

0.731502

N/A

2.842856 [1.396108, 4.863837]

5.591175

0.0% [0.0%, 0.0%]

N/A

The regret gap is concentrated in grade-school math and hellaSWAG specifically (Compitum’s utility is strongly negative on both, at both WTP slices) – MBPP is comparatively competitive (mean regret 0.45 at WTP=0.1, vs. 4.2-6.6 for the other two tasks). This is a real, current characteristic of this evaluation run, not a stale-data artifact – see the Methodological Limitation note above for the scope reason this benchmark isn’t expected to showcase Compitum’s geometric mechanism.

Results (Panel Averages - Auto-Included)

The table below is auto-generated from the latest reports/fixed_wtp_summary.md after running the peer-review script. If empty, run make peer-review to refresh.

Fixed-WTP Analysis (95% CI)

WTP

Mean Regret

Win Rate

Avg Cost Delta (wins)

0.10

0.601950 [0.456828, 0.801042]

0.0% [0.0%, 0.0%]

N/A

1.00

2.842856 [1.396108, 4.863837]

0.0% [0.0%, 0.0%]

N/A

Routing Certificate: Anatomy and Example

Each routing decision emits a certificate used for mechanistic auditing and ablations:

{
  "model": "auto",
  "utility": -0.617274,
  "utility_components": {"quality": 0.598477, "latency": -0.841229, "cost": -2.290030, "distance": -2.299428, "evidence": 0.0, "uncertainty": 0.117909},
  "constraints": {"status": "optimal", "violations": [], "feasible": true, "shadow_prices": {"lambda_0": 0.0, "lambda_1": 0.0, "lambda_2": 0.0, "lambda_3": 0.0}},
  "boundary": {"winner": "auto", "runner_up": "fast", "utility_gap": 0.020410, "entropy": 1.098564, "uncertainty": 0.117909, "is_boundary": false},
  "drift": {"trust_radius": 1.088503, "drift_ema": 0.229943, "drift_integral": 2.184457, "lyapunov_function": 4.771852}
}

See Certificate Schema for the full field reference – shadow_prices is a dict keyed by constraint row (lambda_0, lambda_1, …), not a list, and the top-level JSON keys are boundary/drift (not the dataclass’s internal boundary_analysis/drift_status attribute names).

Reproduce locally:

compitum route --prompt "Sketch a proof for AM-GM inequality." --trace

Interpretation guide

  • utility_components: additive terms before constraints; sign matches contribution to U.

  • constraints: feasible and Lagrange multipliers (shadow prices) for active limits.

  • boundary_analysis: gap (utility gap to runner-up), entropy (softmax uncertainty), sigma (spread of model scores).

  • drift_status: trust-region monitors (instantaneous, EMA, and integral).

Reviewer Checklist

  • [ ] Exact Python version, dependency pins, and seeds are documented.

  • [ ] Scripts produce the same CSV/JSON/HTML artifacts with matching SHA-256.

  • [ ] Primary results table contains per-task values and panel averages with 95% CIs.

  • [ ] Baselines share prompts, budgets (lambda), and accounting; “best baseline” is computed per evaluation unit.

  • [ ] Constraint compliance reported and >= 99.9%.

  • [ ] Failure modes and ablations are presented (boundary ambiguity, constraint removal, metric rank/lambda sweeps).

Ablations and Frontiers (Suggested)

  • Remove constraints + measure violations and utility changes.

  • Remove boundary diagnostics + measure missed deferrals/alerts under ambiguity.

  • Metric rank/lambda sweeps + stability vs. fit capacity.

  • Cost-quality frontier: utility vs. cost curves across the lambda grid; report area-under-frontier.

Known Limitations and Threats to Validity

  • Bounded panels may miss OOD behavior; include targeted shifts if feasible.

  • Performance/cost proxies depend on upstream pricing and tokenization; results may vary under different cost models.

  • Learned baselines can be sensitive to hyperparameters; we report configs and search ranges where applicable.

Failure Modes and Guardrails

  • Ambiguous boundary (high entropy, low gap): operator deferral recommended.

  • Tight constraints with high lambda: can reduce apparent performance; report compliance rate and active constraints.

  • Distribution shift: drift monitors trigger smaller trust radius; document policy for escalation.

Limitations and Threats to Validity

  • Panel boundedness may hide failure modes; include OOD tasks as stress tests.

  • Baseline tuning: ensure equal budgets and fair hyperparameters; document any deviations.

  • Cost model simplifications (token price proxies) affect lambda sweeps; report assumptions.

Ethical and Responsible Use

  • Constraint design should reflect compliance and safety requirements; routes that violate constraints are rejected by construction.

  • No redistribution of proprietary datasets; scripts fetch or expect locally-available licensed inputs.

How to Cite

If this work informs your research, please cite the repository and version number (see compitum --version).

For convenience, a condensed results table is also maintained in Results Summary.

Per-Baseline Win Rate

Per-Baseline Win Rate (Standalone)

No comparable per-eval rows found.

Frontier Gap

Frontier Gap (Standalone)

WTP

Avg Gap to Frontier [95% CI]

At Frontier

N

0.10

0.139328 [0.078752, 0.230927]

50.0%

172

1.00

0.000000 [0.000000, 0.000000]

100.0%

86

Decision Rule and Control of Error

  • Decision rule: among feasible models (capabilities + AxB ≤ b), select the model with maximum utility U (src/compitum/router.py:80). Coherence contributes as a bounded prior term to U (src/compitum/energy.py:33; src/compitum/coherence.py:41).

  • Detect: boundary ambiguity via gap/entropy/uncertainty (src/compitum/boundary.py:19); feasibility and approximate shadow prices for auditing (src/compitum/constraints.py:36); uncertainty from component quantiles and distance variance (src/compitum/energy.py:33).

  • Correct: Lyapunov-inspired trust-region control for stable metric updates (src/compitum/control.py:15; src/compitum/metric.py:106).

  • Certify: routing certificate includes utility components, feasibility, boundary diagnostics, and drift/trust radius (src/compitum/router.py:25).

Mathematical Guarantees and Approximations

  • Guarantees by construction

    • Positive definiteness: metric M = L L^T + δI; defensive Cholesky (src/compitum/metric.py:23, src/compitum/metric.py:39).

    • Feasibility-first selection: constraints enforced before argmax utility (src/compitum/constraints.py:36).

    • Bounded updates: trust-region controller caps effective step sizes (src/compitum/control.py:15).

  • Documented heuristics

    • Shadow prices: approximate local diagnostics via finite-difference viability; report-only, not used for selection (src/compitum/constraints.py:36).

    • Coherence: KDE log-density prior in whitened space; influence bounded by clipping and small β_s (src/compitum/coherence.py:41).

    • Batch updates: per-sample adaptation in batch_route is order-dependent; acceptable throughput trade-off (src/compitum/router.py:147).

Sensitivity and Robustness

  • β_s (coherence weight): conclusions robust to modest sweeps; clipping bounds the influence.

  • Metric rank: modest changes preserve headline results; rank controls bias–variance in metric learning.

  • Update stride: affects throughput and adaptation cadence, not decision rule correctness.

Calibration and Statistical Notes

  • Bootstrap CIs: nonparametric bootstrap (B = 1000) over evaluation units with pairing preserved; report 95% percentile intervals.

  • Calibration diagnostics: reliability curves (uncertainty buckets vs. |regret|), Spearman ρ(uncertainty, |regret|) (also in CEI). ECE‑style summaries can be added but are secondary for non‑probabilistic scales.

  • KDE prior: Scott’s bandwidth in whitened features; clipping bounds influence; β_s sweeps reported for robustness.

  • See Statistical Methodology for methodology details aimed at stat.ML reviewers.

  • See Learning Perspective (cs.LG) for a cs.LG‑oriented framing (problem setup, decision rule, learning components, evaluation).

Reproducibility Snapshot

Save environment with artifacts to aid replication:

python -m pip freeze > reports\env.txt
python - <<PY
import sys, platform
print(sys.version)
print(platform.platform())
PY

Include env.txt and system info with your report files.

Reviewer FAQ

  • Are shadow prices true dual variables? No. They are approximate local signals derived from finite-difference feasibility checks. They are reported for auditing and do not influence selection.

  • What stabilizes updates without a Lyapunov proof? A Lyapunov-inspired trust-region controller caps effective step sizes using EMA and integral signals.

  • Can the KDE prior dominate decisions? No. Its influence is bounded by clipping and a small β_s; conclusions are robust to modest β_s sweeps.

  • Is batch metric update unbiased? The per-sample update in batch routing is order-dependent; it is a throughput trade-off that does not affect the feasibility-first decision rule.

Data Lock (Optional, Security-Oriented)

To satisfy security-minded reviewers, you may lock and verify input data/configs by hashing them before evaluation and verifying after:

python tools\security\data_lock.py --write reports\data_manifest.json --paths data\rb_clean configs --exts .csv .json .yaml .yml
:: After evaluation
python tools\security\data_lock.py --verify reports\data_manifest.json --paths data\rb_clean configs --exts .csv .json .yaml .yml

We recommend sharing reports/data_manifest.json alongside evaluation outputs.

Control-of-Error Index (CEI)

We summarize instantaneous, judge‑free feedback quality with a CEI composed of:

  • Deferral quality (boundary vs. high‑regret): AP/AUROC

  • Calibration (uncertainty vs. |regret|): Spearman ρ

  • Stability (trust‑region shrink vs. future regret decrease): Spearman ρ

  • Compliance: feasible rate

Helper script:

python tools\analysis\cei_report.py ^
  --input data\rb_clean\eval_results\<latest-compitum-csv>.csv ^
  --out-json reports\cei_report.json ^
  --out-md reports\cei_report.md

Include the CEI report alongside fixed‑WTP summaries.

cs.LG Fixed‑λ Summary (Paired Bootstrap)

Helper script for regret/win/compliance with 95% CIs per λ slice:

python tools\analysis\lg_summary.py ^
  --input data\rb_clean\eval_results\<latest-compitum-csv>.csv ^
  --lambdas 0.1 1.0 ^
  --bootstrap 1000 ^
  --out-json reports\lg_summary.json ^
  --out-md reports\lg_summary.md

Attach reports/lg_summary.md to summarize per‑slice results for cs.LG reviewers.

cs.CL Summary (Per-Task and Routing Mix)

Helper script for per-task win/boundary rates and routing distribution:

python tools\analysis\cl_summary.py ^
  --input data\rb_clean\eval_results\<latest-compitum-csv>.csv ^
  --out-json reports\cl_summary.json ^
  --out-md reports\cl_summary.md

Attach reports/cl_summary.md to summarize per-task behavior and selection distribution for cs.CL reviewers.

Trust From Regret

  • We frame trust as calibrated expectation of low future regret under bounded updates and instantaneous feedback.

  • Report together:

    • Fixed‑WTP regret/win‑rate with paired bootstrap CIs.

    • CEI components: deferral quality (AP/AUROC), calibration (Spearman ρ; reliability curve), stability (ρ(shrink, future improvement)), compliance (~100%).

    • Control KPIs: trust‑radius shrink/expand/steady counts; r statistics; shrink→improve correlation.

  • Helper scripts: see Control of Error, Statistical Methodology, and Control Perspective.