The Economist Group keeps its AI chatbot grounded with LLM evaluation at two stages. Analysts score it before every release, and every live conversation is labelled. Fionn O’Raghallaigh, Group Product Manager AI, explained the process at aiblLIVE London 2026.
The tool is Lens, a chat service for clients of The Economist Intelligence Unit. Here is how the team decides when it is good enough to launch, and how it keeps checking once it is live.
All figures as stated on stage by Fionn O’Raghallaigh.
It runs a test set of questions through Lens and has EIU analysts score the answers. They check groundedness, citation relevance, answer relevance and trustworthiness. Automated checks test tool use and refusals. Lens must hit set thresholds before its first release and before every update, and the process also won the analysts over.
Every conversation is labelled automatically for task completion, frustration and repeated questions. Those labels are joined to what users did before and after using Lens, through the product analytics tool Amplitude. O’Raghallaigh gets daily updates, replacing the manual analysis that page views alone forced on the team.
You don’t want to build something just because it’s AI, you want to be genuinely adding value.
Fionn O’Raghallaigh, Group Product Manager AI, The Economist Group
It exposed users typing one word, such as “Brazil”, who got a clarifying question and left. The team changed the system prompt so Lens returned a summary with links instead. That cohort then asked more follow-on questions and clicked through more to the underlying EIU analysis, which the business wants readers to do.
Because good answers depend on finding the right, recent content. O’Raghallaigh says weak answers often trace back to an article about six months old. An archive back to 2011 was too much search space. The team narrowed it to about five years and now searches in tiers, starting with the most recent month.
What good looks like today might look a little bit different to what it looks like in 3 months.
Fionn O’Raghallaigh, Group Product Manager AI, The Economist Group

Fionn O’Raghallaigh is Group Product Manager AI at The Economist Group and a former journalist. He leads work on customer-facing generative AI, including Lens for The Economist Intelligence Unit. He spoke on the AI Infrastructure track at aiblLIVE London on 20 October 2026.
If you need to prove an AI feature is good enough to ship, this is a process to borrow. The full PDF goes deeper than this page, in the speaker’s own words.
For product leaders, CTOs, CDOs and AI leads building customer-facing generative AI. Six pages, about an eight-minute read.