PEC is a worldwide series of community events for engineers, researchers and builders working with prompts, context and AI interaction. Across cities around the world, we bring together people designing and operating LLM-powered systems to share real-world techniques, compare approaches and explore how prompt engineering is evolving alongside agents, models and AI-native software.
all.health, AWS, Blindsight, Caringdots, Citi, Comply2Reg, IBM, IFF Research, inDrive, LLM4S Foundation, LSE, NOFire AI, Optimizely, OTTRA / GitLab, Red Buffer, The Trade Desk, Universite Paris Dauphine - PSL, Whitney AI
Crossrail Place,
Canary Wharf,
E14 5AR, London, UK
Level -2
Tube access
Jubilee, Elizabeth and DLR lines: Canary Wharf station
Building AI agents is getting easier. Improving them systematically is still hard. In this talk, I’ll show how we can close that loop by combining automated experiments, evaluation, observability, and coding assistants. Using Pegasus, a platform I built to automate agent experiments and evals, I’ll demonstrate how OpenTelemetry traces and spans can become feedback that a coding assistant can inspect directly. The assistant can analyse how an agent behaved, identify failure modes, make changes, rerun experiments, and evaluate whether those changes actually improved the system. Attendees will learn how to build a practical self-improvement flywheel around agentic systems: observe what happened, understand why it happened, change the system, measure the result, and repeat.
Rafael Pierre is a Lead AI Engineer specialising in production AI systems, agentic applications, evaluation, and observability. He currently leads the development of Pegasus, an evaluation and observability platform for production AI agents. Over the past 7+ years, he has built and deployed AI and ML systems across organisations including Citi, Hugging Face, Databricks, ING, and ABN AMRO, with a focus on taking complex AI systems from prototype to reliable production deployments.
Embedding models for retrieval have long been treated as black boxes. That has started to change. A young research field at the intersection of mechanistic interpretability and information retrieval has begun opening retrievers up, and the findings quickly turn into capabilities: embeddings that unfold into interpretable latent terms, so you can actually read why a document matched; sparse features usable as an indexing vocabulary in their own right; and relevance signals you can locate and steer, changing what a retriever favours without retraining it. This talk is a tour of these findings and what they make possible in practice, including what the same lens reveals about classic failure modes such as hub documents and collapsing multi-intent queries.
Alper Nebi Kanlı is a Staff Software Engineer (AI Search) at Optimizely in London, where he builds search and retrieval systems. He has spent a decade working on search, AI engineering, and machine learning in production. He writes about language models at alpernebikanli.com, time to time speaks at London AI meetups including PyData London, and co-organizes the AI Signals meetup.
Most LLM-in-healthcare talks come from one side or the other: either the clinical side, which tends to stay conceptual, or the engineering side, which tends to treat "clinical context" as a footnote rather than a design constraint. I sit across both — as a registered nurse and clinical advisor with extensive hands-on experience, and as a product developer who now builds and architects these systems directly. That combination is rare, and as a woman working in a space that's still heavily engineering-dominated, it's a perspective I don't think gets enough airtime either. It's exactly what lets a talk like this go deep on the technical side without losing what actually makes these systems safe or dangerous in a real ward.
Martina Wyss is an LLM and Healthcare AI Specialist working at the intersection of large language models, medicine, and digital health. With hands-on expertise in training, evaluating, and fine-tuning LLMs—specializing in prompt engineering and Reinforcement Learning from Human Feedback (RLHF)—she focuses on building safe, clinical-grade AI applications. Combining her technical LLM focus with a background as a registered nurse (Dipl. Pflegefachfrau HF) and a degree in human-centred design from the University of the Arts London, Martina bridges the gap between AI engineers, clinical workflows, and patient needs to ensure healthcare LLMs are accurate, reliable, and practical for real-world deployment.
If we analyze recent publications (e.g., Karen Hao's book) showing how Large Language Models (LLMs) were trained (AI Agents use these models), we may have doubts when it comes to their quality and real possibilities. In this case, I am thinking about the data - its organization and the value it holds. On the other hand, in the paper published in Nature (2026) the Authors showed how easily LLMs can be manipulated. Even these two situations can raise an alarm in our heads. The main question is - can we trust LLMs and AI Agents? In this presentation, I will not only give an answer to the previously asked question but also show how can we improve (adjust) LLMs (and by that, AI Agents) and introduce (based on the concept described by Nvidia in 2025) the idea of expert models that can be a real answer to the issues with LLMs.
Maciej Szymkowski is a researcher in the field of Artificial Intelligence and Machine Learning. His main research areas include reasoning models, explainable and trustworthy AI as well as mathematical foundations of ML and AI. He is affiliated with T-Labs (T-Mobile Poland) where he works as a Chief AI Researcher (from July 2026). Maciej holds a Ph.D. in the field of Computer Science which was awarded by his alma mater – Bialystok University of Technology, Faculty of Computer Science (2026). The topic of his dissertation was trimodal biometrics system powered by AI (three traits from a human finger were used, Maciej build both a device and AI models to recognize a user). He was affiliated with Bialystok University of Technology, Faculty of Computer Science (2018-2026) but also with two other universities in Poland – Warsaw University of Technology (2021-2022, Faculty of Electronics and Information Technology) and with AGH in Cracow (2021-2022, Faculty of Physics and Applied Computer Science). Between 2022 and 2024 he was with Łukasiewicz Research Network – Poznań Insitute of Technology where he was Senior AI Researcher and Head of AI development team. Maciej, summarized (until now) his research work with more than 45 research papers (published in JCR journals, as chapters in the books and in conference proceedings). He also participated in several projects funded by European Commission and by National Centre for Research and Development in Poland. Maciej has also experiences in working outside of academia. He was a software developer (he started with Java as his main language but then moved to Python), data scientist, machine learning engineer, and AI engineer. Recently he was with Future Processing, Hemolens Diagnostics, and SoftServe. He is working with different companies from 2016.
18 months after LLM4S kicked off, and one year after Kannupriya Kalra and Rory Graves showed a usable version on the Scala Days Switzerland stage, LLM4S has matured into a production-ready platform. In this talk, we’ll share the journey to LLM4S 1.0, from an ambitious idea for bringing first-class generative AI capabilities to Scala and JVM to a complete platform for building production AI applications. We’ll explore what worked, what failed, what we learned from building in one of the fastest-moving areas of technology, and why Scala’s type system and functional programming foundations have proved particularly valuable in the age of LLMs and agents. Along the way, we’ll demonstrate how LLM4S has evolved to support capabilities including multi-provider LLM integration, tool calling, MCP, agent orchestration, memory, observability, streaming, speech and multimodal AI, and production deployment patterns. Expect live demos, architecture diagrams, lessons from the open-source journey, and a practical look at what it takes to move from an AI experiment to a production-ready Scala application. Whether you have followed LLM4S from the beginning or are discovering it for the first time, you’ll leave with a clear picture of what LLM4S 1.0 can do, where the ecosystem is heading, and why your next AI project can be built in Scala.
Kannupriya Kalra is the Co-Founder of the LLM4S Foundation and creator of LLM4S, an AI-native platform for Scala. She is an engineering leader based in London. She contributes to the AI and Scala open-source ecosystem through the Scala Center and Google Open Source and has led Google Summer of Code programmes for the Scala Center in 2024 and 2025. She is also collaborating with Manning Publishing in New York to create practical AI learning resources for developers. A global speaker, Kannupriya has presented on AI, Scala, and open source in more than ten countries. She is passionate about empowering developers through hands-on generative AI projects, open-source contributions, and responsible AI education. Her engineering experience includes building scalable backend systems using functional Scala at Sky in London. Rory Graves has been passionate about coding for as long as he can remember, starting with writing games in BASIC many years ago. Over a 30+ year career, he has worked across a vast range of systems, from embedded software to distributed, internet-scale bidding engines. His experience spans industries, company sizes, from two-person startups to multinational corporations, and programming paradigms. A long-time advocate of Scala, Rory has contributed to numerous open-source projects, including performance optimisations in the Scala compiler. Today, he works in a hybrid role supporting applied AI and ML research while continuing to mentor developers and share knowledge through public speaking. Outside software, Rory is a martial artist, a windmill tour guide, and a passionate mentor of programming and public speaking.
An agent wrote a change. It passed review, it passed CI, and then it took a production path down for about three hours. Nothing in the diff was wrong on its own, which is exactly why it got through, and the failure only existed in the combination of the edits. I have been shipping AI-generated code to production since I was fourteen, and this incident changed how I work more than anything else has. It took far longer to diagnose than an equivalent bug I would have written myself, and the reason was not the code. It was that I could not ask the author what they had intended, and none of the reasoning behind the change existed anywhere. Commit message, pull request, ticket: nothing. This talk is that incident, and then the three checks I have run ever since. Keeping a change small enough that reverting it is genuinely one step, rather than a careful unwind at three in the morning. Actually testing that the revert works instead of assuming it will. And moving the policy check to before an action executes rather than alerting after it has already landed. I will be honest about what each one costs in shipping speed, because none of them are free. Tool-agnostic, no live demo, nothing to install. If you are handing more of your pipeline to agents this quarter, this is the failure mode I wish someone had described to me first.
Artemis Leonardou is a software engineer and DevRel at NOFire AI. She has been building since she was 14 and now ships AI-generated code to production as a matter of routine, which is where most of what she talks about comes from. She was selected among the top 2000 global CS talents for Y Combinator's AI Startup School, has won first place in three hackathons, and her technical content has crossed 2 million views as Artemis Codes. She has spoken at React Summit and TechBiz, speaks at DevConf.US in September 2026, and leads a hands-on workshop at Open Conf in November. She is 19 and still a full-time student.
Parametric flood insurance pays without a claim. A satellite confirms water above a stated depth and the money moves. Organisations are buying it now for informal settlements across Africa and Asia, and the trigger has a gap. Standard Sentinel-1 thresholding over eight documented floods in dense neighbourhoods detects none of them, because side-looking radar over packed low-rise building never sees the ground, and turbid water sits close to dry concrete in backscatter. The instrument is blind where the policy is sold. This session closes that gap with an agent, and spends most of its time on the design work that makes such an agent safe to point at money. We'll go end to end. How a spoken report in a low-resource language becomes a calibrated measurement. How it is priced, so one report is worth a measured quantity with a ceiling set by your own model's skill. How evidence accumulates until a zone crosses the payout line. Then the constraints. A toolset with no capability to broadcast a public warning. Repeat reports from one handset adding no weight. Evals that catch the agent leading a witness. A payout computed in deterministic code a loss adjuster can rerun a year later and reproduce exactly. You'll leave with the shape of all of it, and an honest account of where it is still weak.
Kayode Adeniyi is an AI Research Engineer. With years building flood models using computer vision methods and satellite technology across Africa, and a background writing production payment infrastructure in Nigeria. He has delivered AI ecosystem assessments across Africa and Latin America for the United Nations Development Programme, won the Luminance AI Challenge at Cambridge Judge Business School, and presented on auditable AI at the Palace of Westminster.
Most eval advice assumes you can check the answer. But a lot of high-stakes LLM work has no definitive answer- like qualitative coding, thematic analysis, narrative findings. These are outputs where two expert humans will legitimately disagree, and where "correct" is a judgement rather than a match. In UK government social research, those outputs feed policy decisions, so "the model seemed confident" is not an acceptable standard of proof. This talk covers how we've approached validation in that environment: separating the parts of a pipeline that do have ground truth from the parts that never will, building statistical validation harnesses for the former, and designing human-in-the-loop review that catches failure in the latter. I'll cover what we measure, what we deliberately don't automate, and the failure modes we've hit. Attendees will leave with a practical framework for deciding which parts of a subjective pipeline can be evaluated algorithmically, which need human judgement by design, and how to defend either choice to a sceptical client.
Oliver Gooding is AI & Automation Lead at IFF Research, a UK social research agency working for central government clients including DWP, the Home Office and HMRC. He built and leads IFF's AI strategy and governance framework, and has spent the last two years shipping production LLM tooling into government research projects, from automated figure checking to LLM-assisted content analysis. He has a background in social research methodology and an MSci in Physics from Bristol.
Agent evaluation is often reduced to a final-answer score or an LLM-as-a-judge prompt. For real agentic systems, that is rarely enough. This talk presents a layered approach to evaluating agents across outcome correctness, environment state, trajectory quality, tool-use behavior, latency and cost, repeated-run reliability, and evaluator quality. It also looks at where deterministic checks work, where model-based judging is useful, and how trace-level evaluation can be combined with human calibration.
Tahreem Rasul is a Google Developer Expert in Generative AI and an AI/software engineer specializing in production GenAI systems, including agentic workflows, RAG, evaluation, and cloud-native AI applications. She regularly speaks at developer conferences and technical communities on practical AI engineering, with a particular interest in reliability, context engineering, retrieval, and evaluating agentic systems.
A customer-service agent can give a convincing answer and still take the wrong action. Once agents can call tools, change records or pass work between agents, evaluating the final response alone is not enough.
Using a published multi-agent customer-service reference implementation, this session will explore how to test what an agent does as well as what it says. We’ll walk through scenarios involving incorrect tool selection, missing information, failed actions and situations that require a human handoff. For each, we’ll examine what to check with deterministic tests, where model-based evaluation can help, and where human judgement remains necessary.
Attendees will leave with a practical approach to building scenario-based evaluations and a reusable release checklist for deciding whether an agent is ready for a controlled pilot. The examples will use AWS services, while the evaluation principles apply across platforms.
Olushola Oladipupo is an Enterprise Solutions Architect at Amazon Web Services, supporting customers across the UK and Ireland. He builds and publishes practical examples of generative AI and multi-agent systems, with a focus on evaluation, human oversight and enterprise adoption. He has spoken at AWS re:Invent, AWS Summits and AWS Community Day West Africa. His work connects technical architecture with the decisions teams need to make to adopt AI responsibly and effectively.
Generalist RAG systems fail in high-stakes domains like finance. They can't handle the intricate cross-references and conditional logic of regulatory text, leading to hallucinations and making them untrustworthy. In this talk, I will share our practical journey building a domain-specific AI for regulatory compliance. We'll cover: 1. Architecture Evolution: Moving from standard vector RAG to a Graph RAG architecture that uses a regulatory ontology to understand relationships, not just retrieve text. 2. SLM Advantage: How combining Graph RAG with specialized Small Language Models (SLMs) yields higher accuracy, 80x lower cost, and drastically improved output stability compared to generalist LLMs . 3. Ensuring Reliability: A framework for synthetic data generation, benchmarking, and continuous evaluation to make inherently probabilistic AI systems auditable and reliable for production. We will include a practical architecture walkthrough and a live demo of our approach.
Karthigeyan R J is the Founder of Comply2Reg, building domain-specific AI for banks and financial institutes in the regulatory and compliance space. He formerly worked as a GenAI Solution Expert at 4CRisk.ai (acquired by CUBE, UK - market leader in RegTech) and has hands-on experience fine-tuning models and building agentic workflows for regulated environments.
This talk presents an automated regression testing framework designed to validate the migration of a virtual assistant from a rule-based, session-oriented conversational platform to a modern, LLM-driven orchestration platform. The framework replays multi-turn conversations against both systems in parallel and produces a side-by-side comparison of response quality, latency, and fallback behaviour, scored using two complementary text similarity metrics: Jaccard similarity for lexical overlap, and cosine similarity over sentence embeddings for semantic equivalence. A three-phase test runner coordinates live conversation capture, automated input replay, and a consolidated metrics report covering per-turn detail and unanswered-question coverage. The project was developed with the assistance of an AI-powered coding agent. The work establishes a repeatable, evidence-based process for verifying conversational parity and identifying gaps before full production cutover.
Peter Fox is a mid-level Software Client Engineer in IBM Client Engineering's Financial Services squad in London, where he has spent over five years building client-facing AI pilots using the garage methodology - working across AI agents, generative AI and LLMs, AI governance, database access management, and computer vision. Each engagement typically runs for two to three weeks with a multi-disciplinary team, tackling some of the most complex business challenges faced by enterprise clients in financial services and beyond. Before specialising in AI engineering he held roles in technical AI sales - supporting higher education and life sciences clients and contributing to a sign language recognition app built on deep learning - and in cybersecurity with a focus on SOAR platforms. A proficient Python developer and professional mentor with science teaching and outreach experience at both secondary school and university level, Peter holds a PhD from Imperial College London in theoretical physics and brings strong analytical and problem solving depth to every team he works in.
Long before language models, we had AI systems that found clever ways to satisfy exactly what we asked for but not what we meant. An AI playing Tetris learned to pause the game before it lost, so it technically never lost. We laughed at the time, but the alignment problem behind it was never solved, and agents now have far more capabilities than just pausing a game. This talk is about what happens when an agent follows the request against the intent of its user: where that behaviour comes from, what it means for security, and which mitigations actually exist today.
Guilherme Santos is an ethical hacker and co-founder and CEO of Blindsight, a Zurich-based AI security company. A former Fortune 500 red teamer and zero-day hunter, he now works on adversarial attacks against AI systems and agents, including prompt injection, data poisoning and agent misuse. He is Global Ambassador for Germany at the Global Council for Responsible AI and is a regular speaker at GITEX, IT-SA, ShmooCon, and many more.
AI agents have gone from finishing your sentences to running commands, reading results, and trying again. In this talk, Viktor shares practical insights from two perspectives: managing a large mobile codebase at scale and building personal side projects (a game, an audio app, a home server via Telegram). Key takeaways & topics covered: - Single-instruction pattern: Write instructions once and reuse them across all tools (Claude Code, Cursor, Xcode's agent, local models) without vendor lock-in. - Autonomy boundaries: Clear guidelines on what agents may do unasked vs. what requires approval. - Failure logs: Recording past failures so agents stop suggesting ineffective fixes. - Real-world split: An honest evaluation of what agents genuinely do alone today versus where human engineering remains essential.
I am a Senior iOS Engineer with over 8 years of experience building high-scale, resilient mobile applications. Currently, I ship product experiments for inDrive's Intercity module, serving millions of users globally. My work focuses on complex mobile architecture, developer experience (DevEx), AI-assisted development workflows (including MCP servers and agentic tooling), and native system integrations. I am an active tech speaker, technical writer, and mentor.
AI systems are now writing hypotheses, designing experiments, and analysing results across genomics, materials science, and climate modelling, with systems like Google's Co-Scientist and ERA already published in Nature this year. But the same speed that makes these systems exciting is quietly eroding one of science's core guarantees: that a result can be checked and reproduced. This talk looks at the actual failure modes showing up in AI-driven research pipelines today, from hallucinated citations and silent data misalignment to evaluation setups that reward confident answers over correct ones, and connects them to the rigorous uncertainty quantification practices long used in fields like neuroimaging, where a single noisy scan can wreck an analysis if untreated. I'll walk through concrete techniques for building evaluation and QC layers around AI-assisted scientific pipelines, so speed doesn't come at the cost of trust. Short bio: Data Scientist with 9+ years of experience developing signal processing and machine learning methods for biological and medical imaging time series data. Currently, I focus on how LLM models and the latest AI agents can accelerate scientific progress. Holds a PhD in Neuroscience and patents for software and methods in neuroimaging analysis.
Egor Levchenko is a Machine Learning Engineer at all.health and holds a PhD in Computational Neuroscience from UCL, where he built NNDb-3T+, an open multimodal neuroimaging dataset published in Scientific Data. He previously spent three years at Neurometry developing game-based digital biomarkers for cognitive health, worked as a data scientist at Sberbank, and co-founded a neuromarketing startup. His work sits at the intersection of neuroscience, data science and applied AI, with a focus on turning complex signals into practical models.
Welcome to the Museum of Engineering and Artificial Intelligence. Lead time and deployment frequency occupy the first cabinet. Keep walking: prompt chains, context-window bragging and tokens-per-second leaderboards have their own exhibits. Eventually, even today's LLMs are behind glass.
Drawing on my book, A Brief History of Engineering… and What Comes Next, and my work with GitLab, this technical thought experiment follows the signals shaping AI: inference economics, generation architectures, reliable autonomy and models gaining tools that act.
We examine cost and latency at a stated quality threshold, task completion, retries and human intervention. Autoregressive and diffusion language models provide a concrete example of how changing the machinery changes what we should measure.
Then we extrapolate. What happens when agents coordinate at machine speed and action outruns verification? A worked scenario connects those questions to evaluations, tracing, orchestration, sandboxes and enforceable limits.
The final exhibit imagines today's autoregressive LLMs outdone by “quantum diffusers”. This explicitly speculative future asks what must survive technological change, leading to Intentware: my proposed next-generation architecture for preserving purpose, constraints and accountability. Attendees leave with signals to watch and engineering questions to apply to their own AI systems.
Neil Douek is a platform engineering practitioner, speaker and author of A Brief History of Engineering… and What Comes Next. His experience spans developer experience, DevSecOps and automation in financial services and global technology organisations, including LSEG and Fujitsu. He works with enterprise engineering teams through GitLab Professional Services and OTTRA. A Team Topologies Advocate and certified GitLab Solution Architect, Neil explores how AI changes the tools, measurements and responsibilities of software engineering.
How do you train an enterprise search agent without access to a real company’s private data? We built a synthetic company instead: employees, projects, emails, Slack conversations, documents, and calendars—all inside a stateful environment with a clock that prevents the agent from seeing the future. This talk follows our attempt to train a 4B model to search that world and produce accurate, cited daily briefs. I’ll walk through constructing verifiable tasks, designing leakage-resistant evaluations, and a counterintuitive failure: on-policy distillation drove task completion toward zero even as the training loss improved. We recovered by building a stronger starting policy through off-policy distillation, then using reinforcement learning to repair reliability. On our held-out synthetic benchmark, the final model retained 91.1% of the frontier baseline’s failure-adjusted F1 and completed 92% of episodes. You’ll leave with practical lessons for building synthetic agent environments, measuring search quality separately from completion, and debugging post-training when a better loss produces a worse agent.
Nick is co-founder and CTO of Whitney AI, making model training accessible to AI engineers without a research background. Previously at Microsoft, he post-trained on top of frontier models like GPT-4 and GPT-5 as an ML engineer in the early days of the Azure X OpenAI partnership, often before public release. He later led engineering for Azure's RL-as-a-serving offering and served as a post-training tech lead in Microsoft's Superintelligence division. His work spans evaluation design, practical post-training, and why owning your own intelligence matters in an economy increasingly shaped by a handful of labs.
A short expression can contain a powerful idea. A short prompt can produce an entire application. How do the tools we use change what we can think and build?
Drawing on experience at Morgan Stanley, Accelex and Carta, Phillip Mortimer traces the evolution of document intelligence from regular expressions to agents that check and revise their own work. Connecting programming notation with AI-assisted mathematics, he explores how to express what matters, delegate the details, and work creatively with systems that contribute ideas of their own.
Phillip Mortimer specialises in document intelligence and AI. Previously CTO and Chief Scientist at Accelex and Senior Staff Machine Learning Engineer at Carta, he also spent a decade as a quantitative developer at Morgan Stanley in London and Tokyo. He is a visiting lecturer in MLOps and LLMOps at Université Paris Dauphine-PSL. Outside work, he can often be found in the dojo training for his next kickboxing belt.
