Companies have collectively poured $30-40 billion into AI and only 5% of that investment is generating real return. Behind these shocking figures is something deceptively small: the token, now the No. 1 thing IT and finance teams want more visibility into.
Every time an AI model reads your prompt or writes a response, it’s working in tokens: small, machine-readable chunks of text that get counted, billed, and multiplied across every employee, every tool, and every AI agent running in your organization.
Individually, tokens cost fractions of a cent. At the scale of a large enterprise – tens of thousands of employees, AI agents running multi-step workflows in the background, tools IT hasn’t approved – those fractions of a cent become a line item nobody can fully explain.
This has ballooned into a crisis: most companies can’t answer the basics about their AI spend. Who’s using it? Which tools are burning through the most tokens? Is any of it even working? In one study from McKinsey, 80% of leaders said they had no real way to answer these questions – and when they go unanswered, costs go down a black hole.
That explains why Gartner predicted in 2025 that 40% of AI agent projects would be canceled by the end of 2027. Not because AI isn’t worth using, but because they couldn’t map costs to business value.
Tokens Aren’t New. We’re Just Starting to Pay Closer Attention to Them.
Token-based billing has been the standard since OpenAI launched the GPT-3 API back in 2020. What’s changed isn’t the mechanism but the scale. The sudden realization that a line item nobody watched for years is now big enough to warrant its own governance conversation. Last year, that realization briefly curdled into “tokenmaxxing” – the idea that more AI usage automatically means more value. It doesn’t, and reports are already calling tokenmaxxing dead.
How to Get Tokens Under Control
- Visibility is your most urgent priority. Without it, you don’t know where to look: which AI models to right-size, which employees to cap on tokens, or which to raise limits for. Most companies are leaving a lot of savings on the table because they’re relying on manual guesswork instead of real, tagged data.
- Stop spending on work that doesn’t matter, and use the right metrics to track the work that does. Not every task is worth the tokens, and the metrics you track need to mature as your AI use does – from hours saved to hard bottom-line impact. Leaders are making this shift: enterprises reporting “productivity gains” as their primary AI ROI metric fell over the last year while “financial impact” nearly doubled.
- Fix your data before you scale your AI. Messy, ambiguous data makes for an inflated token bill. Every clarifying question, every needless back-and-forth…it all adds up.
- Define success before you spend. There’s a difference between a defined, measurable success metric (a real KPI) versus feelings or impressions (“people seem to like it,” “we’re using AI a lot,” “it feels like it’s helping”). A real KPI is something specific and trackable – cost per inference, time-to-resolution, adoption rate – set up before the initiative launches, so you can check whether it’s working. Less than 20% of companies track real KPIs for their AI initiatives.
- Keep measuring, always. The organizations winning with AI aren’t the ones that adopted fastest. They’re the ones still asking “is this working?” long after the initial rollout excitement wears off. Like any other IT domain, AI cost management is iterative.
Whether your business buys tokens from a provider like OpenAI or Anthropic or runs models on its own infrastructure, the challenge is the same: spend that’s hard to see, and even harder to attribute. That’s an area Tangoe is focused on bridging — building on existing IaaS cost governance to bring similar visibility and control to AI token usage over time.
AI isn’t going to slow down, and neither is the spend that comes with it. The only real question is whether your company is watching it or finding out about it after the damage is done.
Our latest guide has a full breakdown of these strategies and so much more to bring your AI token spend under control. You can check it out here, or learn more about our solutions and support here.