Local vs Frontier Model Benchmark

What it is

A pre-registered benchmark of open-weight models running on a single 12 GB consumer GPU against Claude Opus 5 on investment-adviser document tasks: fund-term extraction, change detection, grounded Q&A, filing briefs, meeting notes to CRM, client drafting and marketing review. Report No. 01 found Qwen2.5-14B extracting 20 fund terms with citations at 98.3% field accuracy against Claude's 100%, with 0% hallucination on both. Corpus, answer keys, frozen prompts, raw outputs and graders ship in an anonymized public data release so anyone can rebuild the numbers without a GPU or API key.

The problem

Firms considering private on-premise AI have no honest, reproducible numbers on what a local model gives up compared with a frontier model.

How it works

1. A frozen, hash-pinned corpus, answer keys and prompts, with a registration file pinning read-only inputs; local models run through Ollama (run_local.py).
2. grade.py and report.py score field accuracy, unsupported content, trap handling and supersession handling against tiers (parity, usable with review, gap) fixed before any run.
3. Ablations show a 7B model cost 3.3 points and broke supersession handling, and Ollama's default 4K context collapsed accuracy to 0.8% versus 16K.
4. build_public.py generates the public release and scan_public.py audits it; every published number is a field in a results.json cited by JSON path.

Stack

Disciplines

For your business

A reproducible benchmark that shows a client exactly which of their document tasks a private local model can handle and which still need a frontier model.