Why Google wants to win the frontier conversation | Tulsee Doshi (Head of Product for Gemini)

Episode Overview

• Google Director of Product Management Tulsee Doshi explores how the development of advanced artificial intelligence models is transforming the product management discipline, safety evaluation protocols, and organizational workflows.
• The conversation frames the evolution of AI product development from static, deterministic software engineering to dynamic, probabilistic system curation that requires both rigorous quantitative benchmarks and intuitive qualitative evaluation.
• This content is highly relevant to product managers, software engineers, and technology leaders seeking to build, evaluate, and scale resilient AI-driven applications while maintaining safety and inclusion.

Key Concepts

• AI Models as Emergent Systems ("Snape's Potions Class"): Unlike traditional software engineered with deterministic code, frontier AI models behave more like grown systems. Product managers cannot simply spec an outcome; they must input carefully curated ingredients (data, training protocols) and dynamically discover, evaluate, and shape the "magic" or unpredictable behaviors that emerge on the output side.
• The Dual Nature of AI UX: User experience in AI exists at two levels. The first is the container (traditional UI elements like search bars, drop-downs, and tabs). The second—and more critical—is the model's inner persona (its formatting, verbosity, conversational tone, and transparency), which must be actively managed as a core product feature.
• Monitoring Chain of Thought for AI Safety: Ensuring safety in advanced agentic models requires moving past static input/output filters. PMs and safety researchers must deploy secondary monitoring models that actively evaluate the model's "chain of thought" or internal reasoning steps to identify and preemptively block misalignment before an action is executed.
• The Monk Skin Tone (MST) Scale in Model Evals: Replacing the 6-point Fitzpatrick scale (a dermatological scale biased toward lighter skin), the 10-point Monk Skin Tone scale allows developers to evaluate AI model performance and product representation across diverse populations, making image-generation and camera technologies more inclusive.
• The "Harness" of an AI Model: The harness is the environment or wrapper built around the core AI model. It includes system instructions, tools, skills, context, and data access, which heavily determine the model's performance in a live production environment.
• The 60-20-20 Rule for Product Development: Rather than relying on traditional MVP frameworks, teams building on rapidly evolving AI models should design for a spectrum of readiness: 60% of the product should be stable today, 20% should use capabilities that are nearly unlocked by near-future models, and 20% should represent ambitious, aspirational long-term bets.
• Preventing "AI Slop" through Purpose Allocation: To avoid overwhelming teams with endless, low-quality, AI-generated documents, product leaders must establish a clear division of labor: AI handles data aggregation and synthesis, while humans remain the sole providers of strategic insight and creative direction.

Quotes

• At 3:16 - "The model isn't predictable... It's kind of like Snape's potions class. You can put all the ingredients in... but you also have to expect that there will be a little bit of magic, and that magic is going to be surprising and unique." - Explaining why AI product development requires discovery rather than just traditional deterministic engineering specs.
• At 5:00 - "You also need to really understand the vibes of the model... Yes, you can understand that a model has improved on a certain capability, but where does that capability really shine through? How does that showcase downstream in a product?" - Emphasizing that quantitative benchmarks (evals) must be paired with qualitative user intuition (vibes).
• At 7:27 - "If you look at Google's strategy, it really is full-stack... all the way from hardware to then the modeling to then this end-user application. And our goal ultimately is... to build models across this Pareto curve of cost to quality." - Outlining Google's competitive advantage in optimizing AI performance at every layer of the infrastructure.
• At 11:09 - "Deploying monitors that can monitor the model's chain of thought, and actually identify whether the model is misaligned with its directed actions... that's actually the kind of mitigation that gives us more confidence in deploying these models." - Explaining how advanced safety engineering looks at the reasoning path of an AI agent rather than just its final output.
• At 12:24 - "If we can actually leverage these tools to do more of [patching vulnerabilities], we can actually make sure that our infrastructure and our systems are robust." - Highlighting how AI-driven offensive and defensive cybersecurity tools are critical to global infrastructure defense.
• At 27:23 - "Six skin tones cannot capture the diversity of skin tones in the world. Not only that, [the Fitzpatrick scale] was actually built for an entirely different purpose... healthcare purposes, and so it actually skewed extremely light-skinned... We worked with Professor Monk... and we pulled that set of [10] skin tones into our evaluations... to be more intentional about how skin tones are represented." - Shifting the industry standard for inclusive computer vision evaluation.
• At 28:53 - "Your model needs to be performing well, but also how your judge rates the model also needs to be performing well for you to have an accurate sense." - Explaining that evaluating LLM-as-a-Judge systems requires assessing both the evaluator and the model itself.
• At 33:52 - "You want to actually build features that aren't necessarily always working today... so that you are building a product that works today, but you're setting yourself up to build a magical product experience as the models become better." - Discussing the 60-20-20 framework to design resilient, future-proof AI products.
• At 36:57 - "I think of a harness as like the wrapper around the model... What are the system instructions, the tools, the skills, the context, the context data that the model has access to? You want to basically give the model an environment in which to operate." - Defining the "harness" concept.

Takeaways

• Start evaluating AI models early by creating a simple, highly curated manual "micro-eval" of 20 challenging prompts to build foundational intuition before investing in automated systems.
• Build future-proof AI products by dedicating 60% of development to stable features, 20% to emerging capabilities, and 20% to long-term, aspirational bets.
• Transition the product management role from writing passive requirements documents to bringing "heft"—including functional mockups, curated datasets, and initial model evaluations.
• Mitigate employee burnout and operational friction by automating tedious administrative work with AI, freeing up cognitive space for creative, high-value tasks.
• Keep a clear division of labor when using generative AI for documentation: rely on the model for raw data aggregation and synthesis, but mandate that human team members provide the final strategic insight.