Practical AI adoption insight, every week. Join 5,700+ aiblBRIEF subscribers
aibl
  • Practical AI Insights
    • By Function
      • CEO & Board
      • AI & Transformation Leaders
      • Sales & Marketing Leaders
      • HR & People Leaders
      • Customer & CX Leaders
      • Operations & Finance Leaders
      • Technology & Data Leaders
    • By Topic
      • AI Return on Investment
      • The AI Workforce
      • AI Governance & Risk
      • AI Agents
      • Hands-On AI Leadership
      • AI-Ready Data & Knowledge
    • By Type
      • Latest articles
      • AI explained
      • AI case studies
      • AI playbooks
      • AI workshop guides
      • Boardroom findings
      • AI leader interviews
      • Newsletter archive
  • Events
    • aiblLIVE 2027
      • Overview
      • Register your interest
      • FAQs
    • Leadership Series
      • AI in Travel, 11 Feb 2027
      • About the Leadership Series
  • Previous Events
    • aiblLIVE 2026
      • Overview
      • The agenda
      • Session downloads
      • Workshops and roundtables
      • Photo Gallery
      • Who attended
      • What they said
      • Our partners
    • Leadership Series
      • workforceLIVE
      • growthLIVE
  • Research
    • State of UK AI Adoption 2026
    • What our research says
    • AI analysis
  • AI Benchmark Tool
    • Benchmark your AI ROI (Beta)
  • Find AI Enablement Partners
    • By Category
      • Search all listings
      • AI Strategy & Readiness
      • AI Implementation & Deployment
      • AI Training & Capability Building
      • AI Agents & Agentic Workflows
      • AI Governance & Risk
      • Generative AI Applications
      • Data & Analytics
      • Process & Document Automation
    • By Industry
      • Search all listings
      • Cross-Industry
      • Financial Services
      • Healthcare & Life Sciences
      • Manufacturing & Industrial
      • Media & Marketing
      • Professional Services
      • Public Sector
      • Retail & Consumer
      • Technology & SaaS
    • AI Enablement Listings
      • Register your business
  • Partner
    • Partner with us
    • Apply to list your firm free
    • Insights for vendors
  • About
    • About Us
    • What we believe
    • Contact Us
Subscribe to aiblBRIEF
  • Practical AI Insights
    • By Function
      • CEO & Board
      • AI & Transformation Leaders
      • Sales & Marketing Leaders
      • HR & People Leaders
      • Customer & CX Leaders
      • Operations & Finance Leaders
      • Technology & Data Leaders
    • By Topic
      • AI Return on Investment
      • The AI Workforce
      • AI Governance & Risk
      • AI Agents
      • Hands-On AI Leadership
      • AI-Ready Data & Knowledge
    • By Type
      • Latest articles
      • AI explained
      • AI case studies
      • AI playbooks
      • AI workshop guides
      • Boardroom findings
      • AI leader interviews
      • Newsletter archive
  • Events
    • aiblLIVE 2027
      • Overview
      • Register your interest
      • FAQs
    • Leadership Series
      • AI in Travel, 11 Feb 2027
      • About the Leadership Series
  • Previous Events
    • aiblLIVE 2026
      • Overview
      • The agenda
      • Session downloads
      • Workshops and roundtables
      • Photo Gallery
      • Who attended
      • What they said
      • Our partners
    • Leadership Series
      • workforceLIVE
      • growthLIVE
  • Research
    • State of UK AI Adoption 2026
    • What our research says
    • AI analysis
  • AI Benchmark Tool
    • Benchmark your AI ROI (Beta)
  • Find AI Enablement Partners
    • By Category
      • Search all listings
      • AI Strategy & Readiness
      • AI Implementation & Deployment
      • AI Training & Capability Building
      • AI Agents & Agentic Workflows
      • AI Governance & Risk
      • Generative AI Applications
      • Data & Analytics
      • Process & Document Automation
    • By Industry
      • Search all listings
      • Cross-Industry
      • Financial Services
      • Healthcare & Life Sciences
      • Manufacturing & Industrial
      • Media & Marketing
      • Professional Services
      • Public Sector
      • Retail & Consumer
      • Technology & SaaS
    • AI Enablement Listings
      • Register your business
  • Partner
    • Partner with us
    • Apply to list your firm free
    • Insights for vendors
  • About
    • About Us
    • What we believe
    • Contact Us
  • Subscribe to aiblBRIEF

LLM evaluation at The Economist Group: keeping its AI chatbot grounded

Downloads | AI case studies, AI governance and risk, AI-ready data and knowledge, Technology and data leaders
Share this session:

The Economist Group keeps its AI chatbot grounded with LLM evaluation at two stages. Analysts score it before every release, and every live conversation is labelled. Fionn O’Raghallaigh, Group Product Manager AI, explained the process at aiblLIVE London 2026.

The tool is Lens, a chat service for clients of The Economist Intelligence Unit. Here is how the team decides when it is good enough to launch, and how it keeps checking once it is live.

The Economist Group in numbers

  • 183 years of Economist journalism set the quality bar for any AI answer.
  • 1946 is when the Economist Intelligence Unit began, in response to reader questions.
  • About 5 years is the search window Lens now uses, cut from an archive going back to 2011.

All figures as stated on stage by Fionn O’Raghallaigh.

How does The Economist Group run LLM evaluation before launch?

It runs a test set of questions through Lens and has EIU analysts score the answers. They check groundedness, citation relevance, answer relevance and trustworthiness. Automated checks test tool use and refusals. Lens must hit set thresholds before its first release and before every update, and the process also won the analysts over.

How does it monitor AI answer quality after launch?

Every conversation is labelled automatically for task completion, frustration and repeated questions. Those labels are joined to what users did before and after using Lens, through the product analytics tool Amplitude. O’Raghallaigh gets daily updates, replacing the manual analysis that page views alone forced on the team.

You don’t want to build something just because it’s AI, you want to be genuinely adding value.

Fionn O’Raghallaigh, Group Product Manager AI, The Economist Group

What did the evaluation data change?

It exposed users typing one word, such as “Brazil”, who got a clarifying question and left. The team changed the system prompt so Lens returned a summary with links instead. That cohort then asked more follow-on questions and clicked through more to the underlying EIU analysis, which the business wants readers to do.

Why is retrieval the hardest part?

Because good answers depend on finding the right, recent content. O’Raghallaigh says weak answers often trace back to an article about six months old. An archive back to 2011 was too much search space. The team narrowed it to about five years and now searches in tiers, starting with the most recent month.

What good looks like today might look a little bit different to what it looks like in 3 months.

Fionn O’Raghallaigh, Group Product Manager AI, The Economist Group

Key takeaways

  • Set a release threshold. Score answers before launch and before every update.
  • Involve your experts. Analysts who score the tool come to trust and use it.
  • Label live conversations. Join quality labels to what users do next.
  • Refresh the test bank. Feed failures and good examples back into evaluation.

About the speaker

The Economist Group logo

Fionn O’Raghallaigh is Group Product Manager AI at The Economist Group and a former journalist. He leads work on customer-facing generative AI, including Lens for The Economist Intelligence Unit. He spoke on the AI Infrastructure track at aiblLIVE London on 20 October 2026.

What’s in the full case study

If you need to prove an AI feature is good enough to ship, this is a process to borrow. The full PDF goes deeper than this page, in the speaker’s own words.

  • The offline evaluation: the four scoring criteria and the automated checks behind every release.
  • The online evaluation: how conversation labels and behaviour data work together.
  • The one-word search fix, step by step, and what it did to engagement.
  • Five actions to take back to your own organisation.

For product leaders, CTOs, CDOs and AI leads building customer-facing generative AI. Six pages, about an eight-minute read.

Ship AI answers you can trust

LLM evaluation at The Economist Group: keeping its AI chatbot grounded
aibl
Subscribe to aiblBRIEF

Explore

  • About
  • aiblBRIEF newsletter
  • AI Enablement Directory
  • Benchmark your AI ROI
  • aibl Information Pack for AI’s
  • Contact

AI Insights

  • All articles
  • Insights by role
  • AI case studies
  • Boardroom findings
  • State of UK AI Adoption 2026

Events

  • All Events
  • aiblLIVE 2027
  • aiblLIVE FAQs
  • aiblLIVE 2026 session library
  • aiblLIVE London 2026 photo gallery
  • aiblLIVE London 2026 partners
  • UK AI events guide 2027
  • Compare AI Events

Partnership

  • Apply to list your firm free
  • Partner with us
  • Insights for vendors
  • aiblLIVE London 2026 partners
© 2026 aibl media limited (trading as aibl, AI for business leaders). Company registration number: 16634894. The organisers of aiblLIVE never make attendee data available to third parties for resale. If you receive an email offering data that claims to come from aiblLIVE, or from any other event organised by aibl media limited, please delete it, block the sender, and do not respond.
  • Privacy Policy
  • Cookie Policy
  • Terms of Use