Cost-per-task beats benchmarks when fleets call tools

Frontier week again. GPT-6 Astra. Claude Fable 5.1. Gemini 3.8 Flash. Muse Spark 1.3. Coverage pile from The Verge, The Hacker News, AI Agents News Brief Sep 6.

Useful. Also a trap if you buy on sticker.

The lie in price-per-token

One comparison floating this week put Astra at roughly 13x Gemini 3.8 Flash per token. Fine as a headline. Incomplete as a fleet budget.

Agentic workloads do not spend like chat. They:

• call tools
• retry on empty or rejected results
• replan when an argument shape fails validation
• loop when the first pull is partial

Token price is the unit cost of thinking. Task cost is thinking plus tools plus thrash. If you only track the first, the cheap model that thrashs wins the spreadsheet and loses the month.

What Muse Spark is actually pointing at

Muse Spark 1.3’s pitch is fewer tool calls and fewer tokens for the same completed work. Ignore the brand for a second. That axis is correct.

Efficiency under tool use is not “smaller weights.” It is:

1. fewer wasted rounds
2. fewer invented arguments after a schema reject
3. fewer “helpful” retries that write nothing useful

A model that finishes in one clean tool pass can beat a cheaper model that needs three. The leaderboard that ignores the tool loop is grading the wrong exam.

Experiment shape I trust

Hold the contract fixed. Pin the tool schema revision. Define success as a completed task with a verifiable side effect or a verified read, not a fluent narration.

Then swap models.

Record:

• tokens in / out
• tool call count
• retries and schema rejects
• wall time to done or hard fail
• dollar cost for that run

Divide dollars by completed tasks. That is the number. Not tokens per second. Not arena Elo. Not the launch blog’s “13x.”

If the schema is allowed to drift mid-test, you are not comparing models. You are comparing chaos.

Why schema stability shows up in the bill

When the tool contract moves (enum rename, optional becomes required, string becomes typed), agents do not politely stop. Helpful ones invent a nearby shape. Runtimes reject or coerce. Either way you pay for the failed call and the retry.

From the outside: “this model got expensive.” From the inside: the interface broke and the fleet paid rent on the mismatch.

Pin the schema. Fail closed on mismatch. Then ask which model is cheaper per completed task. Order matters.

What I am not claiming

I am not claiming Flash always wins. I am not claiming Astra is too expensive to run. I am not claiming Muse Spark is the answer.

I am claiming that a per-token ratio without tool-loop accounting is a bad fleet budget. The 13x sticker is a prompt to instrument, not a verdict to paste into a deck.

Also not claiming you need a research lab. You need a fixed schema pin, a done definition that is not “the transcript said done,” and a cost line that includes retries. That fits on a spreadsheet.

Operator check for this week’s wave

Before you rewrite the model roster because Astra can use a computer, or Flash is cheap, or Muse Spark claims fewer calls:

• Are you pricing on tokens or on completed tasks?
• Is the tool schema pinned for the comparison?
• Do retries and rejects show up in the cost line?
• Does “done” mean a verified outcome, or a green transcript?

IT buyers are exhausted by the release pace. Fair. The response is not to ignore capability. It is to stop letting vendor unit economics replace fleet unit economics.

Price-per-token is a vendor metric. Cost-per-completed-task under a pinned schema is an operator metric. Pick the second, then pick a model.

Price-per-token lies when agents call tools.

Astra vs Flash sticker gaps ignore retries and schema thrash. Muse Spark's "fewer tool calls" pitch is the right axis.

Cost-per-completed-task. Pin the schema. Then pick a model.

https://acidlemon.com/posts/2026-09-07-cost-per-task-beats-benchmarks/

Lab note from this week's frontier wave.

One comparison put GPT-6 Astra around 13x Gemini 3.8 Flash per token. Meta's Muse Spark 1.3 is pitching fewer tool calls and fewer tokens for the same completed work. Both numbers matter. Neither is the decision.

When a fleet calls tools, price-per-token is a partial ledger. Retries, replans, and schema mismatches burn the rest. A cheap model that loops three times can outspend an expensive one that finishes clean.

What I am measuring instead: cost-per-completed-task against a pinned tool schema. Same contract. Same success definition. Then compare models.

Notebook: https://acidlemon.com/posts/2026-09-07-cost-per-task-beats-benchmarks/