Caraway Maximizes PMax via 3-Cell Incrementality Test

Marketers have a lot of questions as Google forces brands to switch from smart shopping campaigns to their new Performance Max product. Performance Max (PMAX) campaigns bundle a number of different ad placements within the Google ecosystem, but provide marketers with very little control over where their ads show up. For example, search campaigns targeting brand terms, which historically have been managed separately from non-brand search, are now bundled all together. 

The Caraway team, among many other savvy growth teams, feared that PMax inventory might over index on branded queries, and take credit for purchases that may have happened anyway. Before excluding brand search queries from their PMax campaigns, they decided to test 2 key business questions with Haus:

1. How incremental are PMax campaigns for the Caraway business? 

2. Does including branded queries in PMax campaign help or hurt incrementality?

To answer their questions, Caraway ran a 3-cell geo lift test in the Haus platform. The United States was split into thirds using a stratified sample - ⅓ of the US received PMax with brand terms, ⅓ received PMax without brand terms, and ⅓ of the US represented the ‘holdout’ or no PMax. Having a holdout is what enables you to answer the question of incrementality (what would have happened anyway?).

The power calculator in the Haus platform ensured that a 3 week test would provide sufficient statistical power to detect lift in Caraway’s KPIs (new customer revenue and total revenue).

To account for known flaws in user level attribution, this lift was measured on Caraway’s aggregate revenue from their data warehouse in the treatment versus holdout regions.

Both strategies were successful in driving incremental lift, but to the team’s surprise, PMax with brand queries drove 1.95X more total revenue and 1.75X more new customer revenue than PMax without brand queries.  If Caraway had moved forward with eliminating branded terms from their PMax campaigns without first testing, in one month they would have foregone tens of thousands in incremental revenue to the business. 

Lastly, they also learned that the Google platform was overstating PMax with brand by 33%. They now use this incrementality factor to calibrate their day-to-day platform reporting.