This talk presents an automated regression testing framework designed to validate the migration of a virtual assistant from a rule-based, session-oriented conversational platform to a modern, LLM-driven orchestration platform. The framework replays multi-turn conversations against both systems in parallel and produces a side-by-side comparison of response quality, latency, and fallback behaviour, scored using two complementary text similarity metrics: Jaccard similarity for lexical overlap, and cosine similarity over sentence embeddings for semantic equivalence. A three-phase test runner coordinates live conversation capture, automated input replay, and a consolidated metrics report covering per-turn detail and unanswered-question coverage. The project was developed with the assistance of an AI-powered coding agent. The work establishes a repeatable, evidence-based process for verifying conversational parity and identifying gaps before full production cutover.
Peter Fox is a mid-level Software Client Engineer in IBM Client Engineering's Financial Services squad in London, where he has spent over five years building client-facing AI pilots using the garage methodology - working across AI agents, generative AI and LLMs, AI governance, database access management, and computer vision. Each engagement typically runs for two to three weeks with a multi-disciplinary team, tackling some of the most complex business challenges faced by enterprise clients in financial services and beyond. Before specialising in AI engineering he held roles in technical AI sales - supporting higher education and life sciences clients and contributing to a sign language recognition app built on deep learning - and in cybersecurity with a focus on SOAR platforms. A proficient Python developer and professional mentor with science teaching and outreach experience at both secondary school and university level, Peter holds a PhD from Imperial College London in theoretical physics and brings strong analytical and problem solving depth to every team he works in.
