AI experimentation for 10X growth — Aakash Gupta talk

What the video covers

Frederic De Todaro, CPO at Kameleoon, joins Aakash Gupta for a deep conversation on how AI is reshaping product experimentation. The episode includes a live demo where De Todaro builds and launches an A/B test variation in two minutes using AI, and covers the full experimentation lifecycle — from ideation and configuration to measurement and analysis. The discussion draws on De Todaro’s 12 years of helping thousands of teams run experiments at scale.

Who it’s for

Product managers and growth professionals who run experiments regularly and want to understand specifically how AI changes each stage of the process. Also relevant for PMs who have avoided experimentation because the build cycle was too slow — this talk addresses that barrier directly.

Key takeaways

1. The build bottleneck is dead. Most product ideas never get tested because building variations takes weeks. AI coding tools have eliminated this constraint. De Todaro demonstrates building an A/B test variation by typing a natural language instruction — “change sorting to price low to high” — and having the variation ready in two minutes. The build step that previously gated experimentation is no longer the limiting factor.

2. Vibe coding changes the economics of experimentation. When building a variation costs two minutes instead of two weeks, the calculus of what is worth testing changes entirely. Teams can now test ideas that would have been dismissed as too small to justify the engineering investment. This means product sense — the ability to generate good hypotheses — becomes the real bottleneck rather than engineering capacity.

3. Measuring AI features requires different metrics. Standard A/B testing metrics do not capture the full picture for AI-powered features. De Todaro identifies three key metrics for RAG systems specifically: retrieval accuracy, response relevance, and user satisfaction. He argues that PMs need to work closely with data scientists to ensure the measurement framework matches the unique characteristics of AI features.

4. PMs still drive strategy; AI provides execution speed. De Todaro is explicit that the PM’s role in experimentation has not diminished — it has shifted. PMs bring business context that AI does not have: customer constraints, strategic priorities, regulatory requirements. The combination of PM judgment and AI execution speed is what produces better outcomes, not AI running experiments autonomously.

5. The two biggest waves in experimentation were ML and generative AI. The machine learning wave introduced personalization and automated audience segmentation. The generative AI wave is now enabling instant test creation and real-time analysis. De Todaro positions these as compounding advances: teams that adopted ML experimentation early are now in a stronger position to benefit from generative AI capabilities.

Worth watching if…

You run experiments on your product but find the build-test-learn cycle too slow, and you want to see a concrete demonstration of how AI tools can compress the experimentation timeline from weeks to minutes.