In machine learning, we often evaluate models against an established ground truth, treating the resulting metrics as evidence of how well a model performs. But what happens when the ground truth itself depends on assumptions, competing definitions, or scientific objectives that aren't necessarily black and white? I've been exploring this problem through independent research into atmospheric blocking — large-scale atmospheric patterns associated with prolonged heatwaves and other extreme weather events. Despite decades of scientific study, there is no single universally accepted definition of atmospheric blocking. Established detection methods can disagree about whether an event occurred, when it began, or how long it lasted. This raises a question that goes beyond the reliability of training labels: before deciding whether a model is right or wrong, shouldn't we first establish what we're actually trying to achieve? A model trained to reproduce an established blocking detection method might achieve excellent accuracy against its reference labels. But does that mean it has learned something scientifically meaningful? Would a different definition be more useful for understanding extreme weather? And how can we design experiments that challenge our assumptions rather than simply reinforce them? In this talk, I'll use atmospheric blocking as a case study to explore how scientific objectives, assumptions and definitions shape the way we train and evaluate AI systems. We'll examine why apparently contradictory labels can both be defensible, how evaluation metrics can reward the wrong behaviour, and how to build validation frameworks that help us test whether our underlying assumptions are actually useful. Although the examples come from climate science, the implications extend to LLMs, AI agents and automated scientific discovery, where success is often subjective, multidimensional or difficult to measure. The challenge isn't always determining which answer is correct. Sometimes it's recognising that we've been asking the wrong question.
Veronika Ulanova is a Senior Software Engineer at PhysicsX, where she has led engineering initiatives focused on turning complex AI and machine learning capabilities into scalable, customer-facing products. Her work spans technical leadership, software architecture and the development of reusable platforms that bridge the gap between ML capabilities and production applications. Alongside her professional work, she independently develops research originating from her MSci project, applying deep learning to detect atmospheric blocking - large-scale weather patterns associated with prolonged heatwaves and other extreme weather events. Her interests lie at the intersection of scientific AI, climate science and software engineering, particularly the challenges of model evaluation, scientific uncertainty and reproducibility.
