The Visibility Trap: Measuring AI Answers Beyond the Ranking with Stuart Bruce, Purposeful Relations
Stuart Bruce, co-CEO of Purposeful Relations, argues AI is not just a tool. It is a stakeholder describing you to everyone else...
Watch videoA 500 employee SaaS company built an agent to handle refund requests and in the demo, it looked great checking the policy, drafting a reply and even knowing when to offer a discount.
Then the CFO saw the bill. Each request was costing about £1.50 in compute to recover roughly £5 in margin. Worse, around one in ten replies misinterpreted the policy and that was enough to shut down the project.
Most firms think about agents like software subscriptions, but they behave more like digital workers. See what mid-market finance leaders are learning about true AI costs for the research behind this. Every step they take has a cost and their reasoning shows up on the bill.
Treat them like software and you won’t see the cost until it’s too late. If you treat them like headcount, you can ask a harder question. Is this role actually paying for itself?
The real aim is simpler. Build agents where machine thinking costs comfortably less than the human time spent on the same problem. That gap is where the value appears.
Treating agents as a one-off build cost is a common mistake.
Agents sit firmly in operational expenditure. Their costs rise with usage. As volume grows, spend grows with it. If the business doubles, agent costs double too, unless someone is actively paying attention. That’s how margins start to slip.
Without a clear P&L owner, costs drift. Tokens accumulate. Models get upgraded “just in case”. Features creep in without anyone checking what they do to the bill.
More dashboards don’t fix that. Ownership does. Every agent needs a named P&L owner. Not the engineer who built it. Not the data scientist tuning it. A business owner who notices margin movement and steps in early.
That’s the problem the next framework is built to address.
Before anything gets built, it forces two questions.
Every proposed agent should be placed on this matrix. The position determines whether you build, whether you require human oversight, or whether the work should be refused altogether.
Start with the human baseline.
| Human cost | Agent cost |
| (Annual salary + benefits + overhead) ÷ annual task volume Example: £45,000 ÷ 5,000 tasks = £9 per taskThen calculate the agent cost. | API calls + lookups + retries + error correction, plus a buffer A 20% buffer is a sensible minimum Example: £0.50 per taskFrom this, set a hurdle rate. |
An agent needs to be at least 10x cheaper than a human to justify full automation. That level of spread creates enough room for edge cases, growth, and mistakes.
If the agent is less than 2x cheaper, the economics are fragile. Small changes in volume or complexity will wipe out the benefit.
Next, layer in risk. Ask one blunt question: if the agent fails, what breaks?
| Low Risk of Failure | High Risk of Failure | |
| High Arbitrage (>10x savings) | Zone 1: The “No-Brainer” Build immediately. Let it run on auto-pilot. | Zone 2: The “Tethered Agent” Build, but mandate human-in-the-loop. |
| Low Arbitrage (<2x savings) | Zone 3: The “Distraction” Refuse. Maintenance outweighs value. | Zone 4: The “Money Pit” Burn the project. High risk, low margin. |
Zone 1 is where you scale. The savings are real and the downside is limited.
Zone 2 is where agents support humans, not replace them. You gain efficiency, but retain control.
Zone 3 looks tempting but rarely pays off. Ongoing maintenance consumes the margin.
Zone 4 should be rejected early. High risk and thin economics make failure likely.
Initial placement
The firm attempted to build a fully autonomous refund agent. It was treated internally as a ‘God Agent’ – checking policy, making decisions, and issuing refunds without review.
On the matrix, this sat squarely in Zone 4.
The outcome was predictable. The agent created real financial exposure, and the CFO shut the project down.
Revised placement
The fix wasn’t better prompts or a larger model. It was a change in design.
The agent was re-scoped into Zone 2.
Result
The same agent became viable once it was placed in the right zone.
The first question isn’t whether an agent can do the task. It’s whether the P&L can live with the cost of failure. If not, you shouldn’t build it. The strength of an agent programme shows up in what you refuse, not what you deploy.
The demo hides the running cost. One refund agent spent about 1.50 pounds in compute to recover 5 pounds of margin, and roughly one reply in ten misread the request. Per-task compute and the cost of errors add up quickly once volume is real.
Measure cost per completed task against the value it recovers, and include the cost of wrong answers, not just compute. A per-request figure multiplied by volume, minus the margin lost to errors, gives a truer picture than any demo.
Route only the cases that need the model, use cheaper steps for simple ones, cap retries, and fix the error rate so you are not paying to redo work. Design for the common case first, then handle exceptions deliberately.
Stuart Bruce, co-CEO of Purposeful Relations, argues AI is not just a tool. It is a stakeholder describing you to everyone else...
Watch video
Stuart Bruce is co-founder and co-CEO of Purposeful Relations, which advises communications teams on AI,...
Read more
Everyone now knows that most AI projects fail. Most of any type of project fails after all. The 80% figure from...
Read moreGet ahead with the most actionable insights, playbooks and real-world AI use cases you can adopt right now, in your inbox every week