What 22 operators say goes wrong with AI pilots

Across 22 interviews with executives since we started the aibl newsletter, one question has kept surfacing. What happens when organisations start with AI before working out the problem it is meant to solve, or the work it needs to change?

Why a pilot can look like progress

Paul O’Sullivan, CTO of Salesforce UK and Ireland, describes the first wave of generative AI adoption as wonder followed by panic. Executives saw competitors moving and felt they had to do something too.

“FOMO will get you to a pilot,” he says. Intention about what the business is trying to change is what gets you beyond it.

That pressure can produce activity without much changing underneath. Rana Gujral, CEO of Behavioral Signals, sees businesses start with the model and then go looking for a problem worthy of it. In his experience, that “almost always ends in a pilot that demos well and dies quietly”.

The AI becomes a feature grafted onto an existing process, while the decisions and incentives around it stay largely as they were.

Dr Laura Weis, Global Human AI Strategy Lead at WPP, sees the result in the extra work hidden behind apparent speed. She describes AI as an amplifier. Put it into a strong system and it can multiply value. Put it into a stretched or unclear one and it can multiply the noise.

That creates what she calls ‘shadow work’, with people filtering, checking, and correcting AI output. Often that work lands on the strongest performers. The result can be more cognitive load and burnout, even while individuals feel as though they are moving faster.

What has to change around the tool

Dropping AI into existing work only makes sense if you understand how that work operates in the first place.

Anca Pintilie of JP Morgan Chase asks: “Are they able to describe how work actually happens within their function?” That means understanding how decisions get made, which ones trigger spend, what data feeds them, and where that data comes from.

Her analogy is the TikTok bag-check. Someone empties their bag on camera and out comes whatever they have not dealt with for weeks. Buy or build AI before looking properly at how the business works, and that is what gets automated.

“The AI is almost never the thing that’s broken.”

That means codifying decisions that currently sit in people’s heads and surfacing contradictions between how different parts of the business think the work happens. “If a team member leaves tomorrow, do we know how they made decisions and where the value came from?”

When Pintilie inherited a large set of regional teams, what was supposed to be two months of discovery took five to six months and around 72 calls. Roughly 300 to 400 people turned out to be working across 13 different operating models.

Those models had drifted apart for defensible reasons, with different teams optimising for different priorities and relationships with the business.

Tim Flagg, co-founder and CEO of UKAI, makes the business need the starting point. Companies ask how to build AI for the business when they should first be asking what they are trying to solve. That usually leads quickly to the less glamorous work around fragmented data and legacy systems.

“Work out what you’re solving, get the data right, then pick the tools.”

Andy Haley, chief executive of Sullivan & Stanley, asks three questions. Is the value there? Do you know what you need to influence to capture it? Can you influence it?

“If you can’t say yes to those three things you probably shouldn’t start.”

Start small, but learn from it

All of that upfront work creates another risk: firms just wait.

Patrik Hedljung, AI Catalyst at Scania, sees organisations waiting for everything to fall into place. “They wait for the technology, they wait for data, they wait for governance, they wait for a mandate.”

“Nobody has the blueprint yet,” he says. “And certainly not for you.”

That does not mean skipping the upfront work, but it need not take six months. In Pintilie’s view, smaller teams can get people in a room, be honest about what is happening, simplify what they can see, and make decisions from there. “Sometimes it’s okay to go in stages.”

O’Sullivan narrows the first step further. Rather than spending years trying to fix all of their fragmented data before doing anything, firms can start with what the first use case needs and take it all the way to production: one workflow, one measurable outcome, and only the data required to make it work.

Because those early pilots are tightly bounded, failure does not have to be wasted effort. Sara Maldon, Head of AI Automation and Transformation at Make, found that some failed projects were more useful than the ones that shipped. They revealed where the real friction sat, from teams without a process for acting on the output to data that was not flowing where it needed to. Her team used those findings to fix the fundamentals rather than keep pushing the same solution.

Once those early gains start to arrive, Haley has a warning for the people leading the programme: don’t bank them.

Frame those gains as investment for what comes next. The time or money released by the first use cases can help make the case for the data, architecture, and governance needed to tackle something more ambitious. Bank it immediately and those foundations have to justify themselves from scratch.

“Too often that saving gets banked,” Haley says, rather than being reinvested in “the architecture foundations that will unlock the next order of business case.”