Why You Need a 3rd Party Incrementality Partner

Are you stuck wondering if you should be using 3rd party incrementality studies, ad platform lift study offerings, or trying to design your own? 

At Haus, we experience marketers' need for incrementality studies every day. While we hear marketers agree that you should be running these experiments to do things like learn fast, calibrate your MMM, or validate if the modeling for MTAs is accurate, the reality is… not all incrementality studies are made equal, and not all platforms offer the same tools. We’re going to shed some light on the different forms of incrementality offerings and why it matters to your business, so that you can derive value out of your experimentation roadmap.

The basics - let’s simplify the methodologies we hear

At their core, marketing incrementality tests are as simple as measuring the difference in a response KPI (like a sale) where one group had exposure to something (like a TV ad) and one group did not (also called a holdout). 

Within ad platforms, you’ll mostly hear about two versions of incrementality studies: user based holdouts and geo based holdouts. They each have their pros and cons.

User based holdouts

• How it works: The ad platform will randomly assign a user (based on a cookie id or some other user identifier) to a holdout group or the treatment group, then decide to serve that user the ad or not. They’ll then report on the difference in conversions, (whatever your KPI is), between the two groups. That difference is your “incremental conversions.”
• Pros: It’s all natively done in the ad platform and there is very little leg work for you to do from a setup perspective to get an incrementality read.
• Cons: With third party cookies going away, and a heightened concern for user privacy, the concept of user-level actions in ad platforms is now more reliant on modeling than it is deterministic.

Geo based holdouts

• How it works: The ad platform will identify markets that appear to be similar to each other and propose a set of markets (usually DMAs) to be “held out” for comparison, and another set of markets to receive the treatment. They’ll then measure the lift in your response KPI of sales in your test markets vs. holdouts. This is sometimes referred to as “matched market testing (MMTs).”
• Pros: There’s no question about user identification between you and the ad platform, making the results easier to QA. You can evaluate more media than just digital platforms (e.g., TV, Out-of-Home, Cinema).
• Cons: Setting up the experiment is usually more manual. Sometimes you’ll question whether markets are really comparable, or representative of the mass market. Mid-experiment, if a holdout market goes off the rails, it can make you doubt if the comparison was still the right one.

Why all business questions can't be answered with ad platform experiments alone

From an incrementality program, you need consistent, repeatable, and durable solutions to make investment decisions.

Some platforms will provide both types of incrementality methods, while others will only have one method. You will also find that these tests are only available for certain campaign types and not others.

Herein lies the root of the problem. How is it possible to normalize the outcomes of different ad platform incrementality methods for your business and make unbiased investment decisions when:

1. You’re comparing user-based holdouts to matched market testing for different platforms.
2. Each platform reports out on the incrementality of their conversion pixel. They’re not responsible for de-duping conversions from the others’, making it difficult to reconcile with your 1P data.
3. You are limited to measuring only the channels that offer conversion lift products.
4. Budget requirements to achieve statistical significance may be higher than BAU, changing the question you’re answering from “were people going to convert anyway?” to “will people convert when I spend a lot more?”

It is really hard to manage an incrementality program objectively with these considerations. Companies hire in-house data science teams in order to design geo-based experiments and report on learnings themselves, especially when the marketing mix becomes more complicated.

How can we best leverage ad platform experimentation then?

If you know it’s time to run incrementality experiments, running it in the ad platforms can still help you make some good investment decisions. 

One major benefit to ad platform experimentation is their ability to report out on platform specific metrics. Think about products like Brand Lift. You get to survey people in the ad placements where people are spending their time (e.g., which of these brands have you heard of recently?) to learn something that would be much harder for you to orchestrate on your own.

Ad platforms also provide a lot of granular optimization levers where you can learn really quickly if one is better than the other. Things like - which creative performs better for this audience? Or which bid strategy returns a higher ROI? - Are just a few examples where the proof from these tests are sufficient to make a decision to inform ad platform optimization.

But as your marketing strategies become more complex and you need to understand cross-platform incrementality measurement, you need a tool like Haus.

Why Haus? It’s your growth copilot.

All the work that you’d need from a team of data scientists and engineers for cross channel geo-based experimentation is encoded in our platform. It was built by our team of scientists, economists, marketers, and product managers that have been working on this problem for years at various companies both big and small. Haus is aware of the benefits and risks across methodologies, and the business issues they can create. As a result, we use more comprehensive methods that give you the flexibility you need to experiment with confidence.

What Haus does:

• Works off of your 1st party data
• Does not require any user-level data
• Integrates with your data warehouse for automated ingestion
• Monitors data quality
• Designs experiments
• Predicts experiment certainty
• Samples treatment and control regions
• Analyzes results using frontier causal econometrics

It’s not one-size-fits-all! The Haus UI is designed to guide you in setting up experiments that are right for your business. We know as marketers that some questions you’ll need to answer are:

• How much budget do I need to understand incrementality? 
• What would happen if we spent less, or had a smaller holdout?
• What if we did this over a longer time period so I can keep spend BAU?

Get started with Haus today

We believe how you power your test is your choice, and it’s your budget to flex and we make it easy for you to understand the tradeoffs in doing so.

From Haus, you’ll get a consistent methodology across all your platforms, instant options for experiment design flexibility, and it’s all in a SaaS tool.

Piqued your interest? Book a demo with us.