Hello PM: Anatomy of AI products — a practical guide to building with LLMs

Ankit Shukla, who has worked at Google and Meta and has taught product management to over 2,500 students, wrote this guide to address a gap he observed in how teams approach AI product development: most resources explain what LLMs can do, but few explain the product decisions that come before and after.

The guide draws a distinction that matters for scoping: AI-native products (ChatGPT, Cursor) place the LLM at the center of the interaction, while AI-augmented products (Notion AI) layer it onto an existing workflow. The approach to building each is different, and teams that treat both as the same category often end up with features that don’t fit how users actually work.

The guide is structured as six sequential areas. Discovery asks whether the problem warrants an LLM at all, and how to identify tasks where AI meaningfully changes the outcome for users. The core technology section covers how LLMs produce output and where they commonly fail, at the level a PM needs to make architecture trade-offs without a machine learning background. The builder’s toolkit discusses prompt engineering, RAG, and fine-tuning, with guidance on when each approach fits given cost and accuracy constraints.

The section on agents is particularly detailed. Shukla introduces a formula for thinking about error accumulation in multi-step systems: ten sequential operations each running at 95% accuracy yield a combined success rate of under 60%. This is a practical argument for designing systems with checkpoints rather than assuming long chains will hold.

Two frameworks stand out in the economics and evaluation section. The Human-in-the-Loop Matrix maps stakes and confidence to determine when AI output can go directly to users, when it requires review, and when humans should act first regardless of model confidence. The Cost Optimization Hierarchy offers a ranked sequence—fix prompts, route to cheaper models, add caching, trim context, batch requests—before investing in infrastructure changes.

Throughout, the guide uses specific products as illustrations: Granola’s approach to meeting notes, Cursor’s codebase indexing for multi-agent coordination, and Harvey’s legal AI product. These examples are used to illustrate principles rather than to celebrate the companies.

The guide is most useful for PMs who have been given responsibility for an AI feature or product and need a structured way to think through the decisions between “we should use AI” and “we shipped something users want.”