The most expensive models are not the best at most real-world agent tasks

A single premium model simplifies procurement and mental load. It also multiplies cost on paths where quality has already plateaued — routing, daily briefs, email triage — while hiding regressions on tasks where tool selection matters more than prose polish.

Agentic systems are not one chat completion. They are a portfolio of tasks with different failure modes:

• Routing — pick the right delegate from a catalogue (accuracy is binary; latency matters on every message).
• Tool agents — choose and execute MCP tools under persona constraints (wrong tool is worse than mediocre prose).
• Prose generators — markdown briefs with strict contracts (links, headings, no hallucinated ticket counts).
• Email flows — skills, ticket creation, and refusal to over-react on digests and newsletters.

If you evaluate models on a generic benchmark or a single chat prompt, you will over-pay for the easy paths and under-diagnose the hard ones.

Our shootout harness (internal ticket AD-084) runs against the live llm-service code paths — not a simplified API playground.

• Real scenarios — golden prompts tagged per subagent, plus production trace replays (actual inbound emails, actual council queries).
• LLM-as-judge — Haiku scores prose on clarity, structure, grounding, and contract compliance; agent runs score tool selection, execution, and outcomes.
• Multi-faceted composite — quality, latency (ms), and cost ($/call) recorded for every run. No single number pretends to capture trade-offs.
• Canonical seven-model lineup — Haiku 4.5, Sonnet 4.6, Groq GPT-OSS 120B, Gemma 4 26B, Qwen 3.6 Plus, Minimax M2.7, Mistral Small 2603 — synced from config/eval/shootout_models.json.
• Repeatable scripts — make model-shootout SUBAGENT=<slug>, make email-shootout, make focus-brief-shootout from the monorepo Makefile.

We cull models that fail smoke runs, fix eval-path bugs before blaming the model (more on Gemma below), and tune prompts before upsizing — Mistral on focus brief needed explicit link-format rules, not a bigger model.

Illustrative daily load for an active solo operator: 100 chairman routes, 24 focus briefs, 40 PA emails, 12 growth-partner runs, 8 UX researcher tasks. Numbers from June 2026 shootout averages.

Opus would widen the gap further; we did not include it in the lineup because the lesson already lands without it: uniform premium models multiply invoice on high-frequency paths where quality plateaus.

Job: read the user message, pick the first delegate from the subagent catalogue (UX researcher, growth partner, project manager, tech lead, support assistant, miniapp coder).

Scenarios (6): UX continuation, LinkedIn/BD priorities, ticket creation, tech golden-path setup, product how-to, interactive miniapp build.

Result: all seven models scored 100% routing accuracy on every scenario. Differentiation is purely cost and latency.

Recommendation: Mistral Small or Groq GPT-OSS for chairman routing — sub-second to ~1s, ~$0.0009 per route. Sonnet adds no accuracy on this golden set.

Job: triage inbound mail — Sentry alerts, GitHub CI, newsletters, Search Console, meeting invites, Fireflies recaps — with skills, ticket creation rules, and LLM judge on outcomes.

Harness: 16 scenarios (golden + production trace replays), full agent mode with tools.

Haiku and Minimax tie on quality; Sonnet is mid-pack at six times the cost of Mistral. Groq scored lower on quality but is a viable fast/cheap tier — early 429 errors in batch shootouts were TPM saturation, not a production architecture flaw; we added retry with rate-limit header parsing.

Job: LinkedIn post drafts, daily BD briefs, ICP refinement — scenarios that expect get_document, linkedin_list_posts, list_tickets, and related tools.

This is where the “most expensive = best” story breaks hardest.

Job: generate the planner morning/midday/shutdown brief — markdown prose from calendar, tickets, git activity, and email context. No tools. Strict contract: ticket links, meeting join URLs, no total ticket counts, no emoji.

Harness: production buildFocusBriefUserPrompt(), LLM judge on clarity / structure / grounding / contract, four scenarios including a heavy-context path (large git + email sections simulating a future PA-skill merge).

On a single morning scenario after prompt tuning, Haiku hit 90% judge score; Mistral hit 82% at 1.7s and $0.002 — our recommended path for a brief that runs every time someone opens the planner.

Gemma eval bug (worth knowing if you run OpenRouter): first shootout scored Gemma at 0% — empty responses with exactly 2,048 completion tokens. Cause: eval path enabled OpenRouter reasoning: medium while capping tokens; Gemma burned the budget on hidden reasoning. Production never had this issue. Fix: dedicated focus-brief run mode with reasoning: none and 8,192 max tokens.

Prompt > model: we added LINK RULES, a reference example, and a user-prompt footer requiring [Name](url) for tickets. Mistral jumped from missing links to production-quality output without changing model tier.

Job: continue UX discovery — list projects, read documents, start skills, write IA sections.

Scenarios (4): heuristic research summary, project status read, IA phase start, heuristic audit framing.

Sonnet won the prose-only research_summary scenario (88%) — it was the only model to reach for web_search — but collapsed to 2% on ux_heuristic_audit after a 118-second tool loop. Minimax led composite quality; Mistral is the best cost/latency tradeoff for agent UX work at ~10× lower cost than Sonnet.

These are stored per subagent row in our database (provider + model_id). We re-run shootouts before every lineup change.

• The moat is not “we use Claude.” It is task-specific evaluation at production fidelity — and the discipline to run Mistral on routing when Sonnet adds nothing but invoice.
• Eval path must match production path. Reasoning flags, token caps, and provider routing that differ from prod will lie to you (Gemma's 0% focus-brief score).
• Prompt contracts beat model upsizing for structured prose — link rules and a reference example fixed Mistral faster than switching to Sonnet.
• Batch shootouts stress rate limits differently from prod. Groq 429s under back-to-back eval were not a production risk; single-email flows with tool gaps behave fine with retry.
• LLM-as-judge scales prose QA where must_contain heuristics give everyone 100%. Multi-dimensional judge scores spread models meaningfully on focus brief.

If you are picking one model for your whole agent stack, ask what each task actually needs. Routing is not email triage is not BD tool selection is not a daily markdown brief. We measured each separately — and the most expensive model was not the best at most of them.

That is the work we are building in public: not bigger models for their own sake, but specialists with the right brain for the job, chosen with evidence.