Compitum Test & Benchmark Report

Overview

This report summarizes Compitum's test and benchmark results alongside common baselines.

• Performance: average task accuracy (higher is better).
• Total cost: average token-compute cost in USD (lower is better).
• Utility: performance - WTP * cost, where WTP (willingness to pay) scales the cost penalty.
• Mean regret: average gap to the best baseline utility at a selected WTP (lower is better).
• Win rate: fraction of evaluations where Compitum's utility >= best baseline utility at the selected WTP.

Charts: bars include Compitum (blue/green) and baselines (gray). The scatter shows average cost vs performance for all models.

Topline Takeaways

• Win rate (share of evaluations where Compitum's utility >= best baseline): 2.3%.
• Mean regret (utility gap to best baseline at selected/best WTP): 0.201801. (on this evaluation suite). Lower is better.
• Avg cost delta on wins (Compitum - best baseline cost on wins): 0.00 USD (parity on wins).
• Average performance: Compitum 0.5111 vs gpt-4-1106-preview 0.7091.
• Average cost: Compitum 2.611772 vs claude-instant-v1 0.257096 USD.

Unit Tests

(skipped)

Exit code: N/A

Step Status

Step	Return Code	Timed Out	Duration (s)	Timeout Cap (s)
Unit Tests	N/A
RouterBench	N/A
Compitum	0	NO	591.044	900

Full machine-readable log stored alongside this report as JSON.

RouterBench Artifacts

Compitum Artifacts

• data\rb_clean\eval_results\eval_results-eval-all-07-21-21-31-val_split.csv

Numerical Summary

Model	Avg Performance	Avg Total Cost
compitum	0.5111	2.611772
claude-instant-v1	0.3923	0.257096
claude-v1	0.4612	2.491849
claude-v2	0.5111	2.611772
gpt-3.5-turbo-1106	0.5842	0.300727
gpt-4-1106-preview	0.7091	3.371482

Regret & Wins (WTP-selected)

Mean Regret	P95 Regret	Win Rate	Avg Cost Delta on Wins
0.201801	0.486364	2.3%	0.000000

Regret computed at best WTP=0.001

Glossary

• Performance: Average task accuracy across the evaluation set (higher is better).
• Total cost: Average token and compute cost in USD (lower is better).
• Willingness to Pay (WTP): How much performance is worth relative to cost; utility = performance - WTP * cost.
• Utility: Single-number trade-off of performance and cost at a chosen WTP.
• Mean regret: Average utility gap to the best baseline at the chosen WTP (lower is better).
• Win rate: Share of evaluations where Compitum's utility >= best baseline utility.