You will hear multiple real memorable stories and examples, many that don't make it to books or articles. These were chosen from over 20 years of experimentation. You'll have the data to show that the poor success rate is documented across multiple organizations; expected it!

Understand key concepts like causality, hierarchy of evidence, and key organizational tenets required for effective experimentation.

Designing metrics is hard. There is a hierarchy of metrics and perverse incentives. The most important metrics comprise of the OEC - The Overall Evaluation Criterion. We will look at good and bad examples.

Getting numbers is easy; getting numbers you can trust is hard. You'll understand common pitfalls and how to design reliable and trustworthy tests.

Learn about the cultural challenges, the humbling results (most ideas fail, pivoting, iterating, learning), institutional memory, ideation, prioritization, experimentation platforms

When building AI or machine learning models, using A/B testing and triggering to evaluate the models that were built offline based on historical data

When you can't run an A/B test, quasi-experimentation methods and the risks of observational causal studies

What are key challenges and open questions in the field

If there is something specific you want to cover, there is time allocated for topics voted by the audience to discuss

The course focuses on developing the intuition and common misunderstandings, without the details of the statistics, which you can find in many books. We cover p-values, statistical power, and triggering.

You can go as technical as you want in the Q&A and community discussions