5 Mistakes Brands Make When Running Marketing Experiments

Joe Wyer, Head of Science at Haus, discusses five common mistakes brands make when running marketing experiments. The session covers bias and precision in experimentation, the importance of pre-commitment, aligning KPIs with customer actions, the limitations of statistical significance, and the challenges of interpreting conflicting attribution and incrementality data.

Notable moments:

• [00:00:00] - Introduction and background of Joe Wyer
• [00:03:51] - Discussion on the importance of considering bias and precision in experiments
• [00:09:00] - The value of pre-commitment in experiment design and analysis
• [00:15:33] - Aligning KPIs with customer actions and the customer journey
• [00:22:25] - The problem with having an overreliance on statistical significance in decision-making
• [00:28:53] - Challenges when attribution and incrementality disagree
• [00:33:51] - Story about Jeff Bezos' commitment to causal inference at Amazon, highlighting the importance of experimentation for business success
• [00:39:13] - Q&A session begins
• [00:56:54] - Closing remarks and wrap-up