Joe Wyer, Head of Science at Haus, discusses five common mistakes brands make when running marketing experiments. The session covers bias and precision in experimentation, the importance of pre-commitment, aligning KPIs with customer actions, the limitations of statistical significance, and the challenges of interpreting conflicting attribution and incrementality data.
Notable moments:
- [00:00:00] - Introduction and background of Joe Wyer
- [00:03:51] - Discussion on the importance of considering bias and precision in experiments
- [00:09:00] - The value of pre-commitment in experiment design and analysis
- [00:15:33] - Aligning KPIs with customer actions and the customer journey
- [00:22:25] - The problem with having an overreliance on statistical significance in decision-making
- [00:28:53] - Challenges when attribution and incrementality disagree
- [00:33:51] - Story about Jeff Bezos' commitment to causal inference at Amazon, highlighting the importance of experimentation for business success
- [00:39:13] - Q&A session begins
- [00:56:54] - Closing remarks and wrap-up
