Stop counting tokens, start counting what ships

Introduction

When developers hand an AI agent a coding task, the agent does more than answer a single prompt—it plans, writes code, runs tests, and iterates until the code passes review. That workflow can consume ten to fifty times more tokens than a simple chat, and per‑token pricing turns valuable development cycles into costly line items. The instinctive reaction is to impose caps, seat limits, or approval gates. While those controls curb spend, they also throttle the very workflows that deliver the most business value. The real question isn’t how many tokens an organization uses, but how much shipped software those tokens produce.

Measure What Ships, Not What Flows

A more meaningful KPI is cost per merged pull request (MPR): the total AI‑assisted development spend divided by the number of changes that pass review and are merged. Pair this with defect and rework metrics so that cheaper changes don’t hide quality erosion. If the MPR cost drops while AI usage rises, the team is converting consumption into productive output.

Why Token‑Based Metrics Miss the Mark

• Noise vs. value – Retries, runaway loops, and exploratory prompts inflate token counts without delivering code.
• No quality signal – Tokens say nothing about bugs, technical debt, or maintainability.
• Blind to ownership – Per‑token pricing treats every call as a variable cost, ignoring the economics of owned inference capacity.

Match the Model to the Task

Agentic coding rarely needs the most powerful model for every step. A pragmatic routing strategy can dramatically lower spend:

1. Frontier models for high‑value moments – Complex planning, novel reasoning, or policy‑driven prompts get sent to the latest, most capable APIs.
2. Open‑weight, on‑prem models for routine work – Code generation, test scaffolding, and iterative refactoring can run on self‑hosted models that cost far less per token.
3. Gateway orchestration – A lightweight router evaluates request complexity, data sensitivity, latency, and cost, directing each call to the appropriate backend.

Open‑weight models have closed the performance gap on many engineering benchmarks, but they aren’t universally equivalent. Teams should evaluate candidates against their own repositories, test suites, and acceptance criteria before delegating production work.

Ownership vs. Rental: The Economics of Repeated Use

With rented inference, every call adds to a variable‑cost bill, making heavy usage increasingly expensive. Owned capacity, by contrast, front‑loads capital costs (hardware, power, ops) and then amortizes them across calls, driving the marginal cost of each additional request down.

• Scalability – As usage grows, owned capacity becomes cheaper per request.
• Security – Running code generation on‑prem keeps proprietary code inside the enterprise, limiting exposure to external models.
• Control – A governed gateway can enforce policy, allowing only approved planning prompts to leave the corporate network.

Test, Baseline, and Iterate

A practical rollout looks like this:

1. Baseline – Record cost per active developer and cost per MPR under the existing API‑only model.
2. Pilot owned capacity – Run a full development cycle on a hybrid platform (e.g., on‑prem open‑weight models for volume, frontier APIs for edge cases).
3. Compare – If both baseline metrics improve, the hybrid approach is delivering value; if not, the organization may need to adjust routing rules or stick with the API model.

Data from early adopters shows a 30‑50 % reduction in token spend after moving routine workloads to owned capacity, while maintaining or improving delivery speed.

Build for Adoption, Not Rationing

Hybrid models suit most enterprises: low‑or‑sporadic usage stays on APIs (no idle hardware), while sustained, repeatable demand moves to owned capacity. A governed gateway provides the best of both worlds—frontier performance when needed, plus sovereignty over core code.

The strategic shift is simple: stop counting tokens and start counting shipped pull requests. When AI‑assisted development is measured by the software it actually delivers, organizations can let usage grow without fearing runaway bills. The companies that win will be those that align AI spend with tangible output, not those that choke adoption with arbitrary caps.

--- This article synthesizes concepts from the MIT Technology Review piece “Stop counting tokens, start counting what ships” and HPE’s hybrid AI platform experience. It is intended for technology leaders evaluating AI‑driven software development economics.