How do you train an enterprise search agent without access to a real company’s private data? We built a synthetic company instead: employees, projects, emails, Slack conversations, documents, and calendars—all inside a stateful environment with a clock that prevents the agent from seeing the future. This talk follows our attempt to train a 4B model to search that world and produce accurate, cited daily briefs. I’ll walk through constructing verifiable tasks, designing leakage-resistant evaluations, and a counterintuitive failure: on-policy distillation drove task completion toward zero even as the training loss improved. We recovered by building a stronger starting policy through off-policy distillation, then using reinforcement learning to repair reliability. On our held-out synthetic benchmark, the final model retained 91.1% of the frontier baseline’s failure-adjusted F1 and completed 92% of episodes. You’ll leave with practical lessons for building synthetic agent environments, measuring search quality separately from completion, and debugging post-training when a better loss produces a worse agent.
Nick is co-founder and CTO of Whitney AI, making model training accessible to AI engineers without a research background. Previously at Microsoft, he post-trained on top of frontier models like GPT-4 and GPT-5 as an ML engineer in the early days of the Azure X OpenAI partnership, often before public release. He later led engineering for Azure's RL-as-a-serving offering and served as a post-training tech lead in Microsoft's Superintelligence division. His work spans evaluation design, practical post-training, and why owning your own intelligence matters in an economy increasingly shaped by a handful of labs.
