Compitum RouterBench Evaluation Summary

This report compares Compitum against baseline routers on a bounded evaluation set. Higher oracle_match indicates lower regret relative to the oracle assignment.

Metrics

• compitum

  • accuracy_mean: 0.5022

  • cost_mean: 1.0159

• WizardLM/WizardLM-13B-V1.2

  • accuracy_mean: 0.4423

  • cost_mean: 0.0902

• claude-instant-v1

  • accuracy_mean: 0.3923

  • cost_mean: 0.2571

• claude-v1

  • accuracy_mean: 0.4612

  • cost_mean: 2.4918

• claude-v2

  • accuracy_mean: 0.5111

  • cost_mean: 2.6118

• gpt-3.5-turbo-1106

  • accuracy_mean: 0.5842

  • cost_mean: 0.3007

• gpt-4-1106-preview

  • accuracy_mean: 0.7091

  • cost_mean: 3.3715

• meta/code-llama-instruct-34b-chat

  • accuracy_mean: 0.4275

  • cost_mean: 0.2264

• meta/llama-2-70b-chat

  • accuracy_mean: 0.4853

  • cost_mean: 0.2640

• mistralai/mistral-7b-chat

  • accuracy_mean: 0.4401

  • cost_mean: 0.0589

• mistralai/mixtral-8x7b-chat

  • accuracy_mean: 0.5768

  • cost_mean: 0.1757

• oracle

  • accuracy_mean: 0.8704

  • cost_mean: 0.2064

• zero-one-ai/Yi-34B-Chat

  • accuracy_mean: 0.5883

  • cost_mean: 0.2368

Where Compitum Wins

• Cost mean vs WizardLM/WizardLM-13B-V1.2: +0.9257

• Accuracy mean vs WizardLM/WizardLM-13B-V1.2: +0.0599

• Cost mean vs claude-instant-v1: +0.7588

• Accuracy mean vs claude-instant-v1: +0.1100

• Cost mean vs claude-v1: -1.4759

• Accuracy mean vs claude-v1: +0.0410

• Cost mean vs claude-v2: -1.5958

• Accuracy mean vs claude-v2: -0.0089

• Cost mean vs gpt-3.5-turbo-1106: +0.7152

• Accuracy mean vs gpt-3.5-turbo-1106: -0.0820

• Cost mean vs gpt-4-1106-preview: -2.3555

• Accuracy mean vs gpt-4-1106-preview: -0.2069

• Cost mean vs meta/code-llama-instruct-34b-chat: +0.7895

• Accuracy mean vs meta/code-llama-instruct-34b-chat: +0.0747

• Cost mean vs meta/llama-2-70b-chat: +0.7519

• Accuracy mean vs meta/llama-2-70b-chat: +0.0169

• Cost mean vs mistralai/mistral-7b-chat: +0.9570

• Accuracy mean vs mistralai/mistral-7b-chat: +0.0622

• Cost mean vs mistralai/mixtral-8x7b-chat: +0.8402

• Accuracy mean vs mistralai/mixtral-8x7b-chat: -0.0745

• Cost mean vs oracle: +0.8095

• Accuracy mean vs oracle: -0.3682

• Cost mean vs zero-one-ai/Yi-34B-Chat: +0.7791

• Accuracy mean vs zero-one-ai/Yi-34B-Chat: -0.0860

Regret (accuracy gap to oracle)

• Compitum: +0.3682

• WizardLM/WizardLM-13B-V1.2: +0.4281

• claude-instant-v1: +0.4782

• claude-v1: +0.4092

• claude-v2: +0.3593

• gpt-3.5-turbo-1106: +0.2863

• gpt-4-1106-preview: +0.1613

• meta/code-llama-instruct-34b-chat: +0.4429

• meta/llama-2-70b-chat: +0.3851

• mistralai/mistral-7b-chat: +0.4304

• mistralai/mixtral-8x7b-chat: +0.2937

• zero-one-ai/Yi-34B-Chat: +0.2822

Determinism

Compitum routing is deterministic given fixed models and parameters, reducing variance and improving reproducibility compared to routers relying on stochastic LLM calls for decisions.