The Ultimate Incrementality Platform Buyer's Checklist

“Incrementality” has quickly become the go-to measurement buzzword‌ — ‌and for good reason. It tackles the core question every marketer (and their finance team) wants to know: Is my advertising actually driving growth for my company? For brands like FanDuel, Ritual, and Newton Baby, incrementality has been a game-changer, helping them save and optimize millions in ad spend. However, not all incrementality solutions deliver the same level of accuracy and precision. So, how can you trust your measurement platform to capture incremental impact and not steer you in the wrong direction?

This guide will walk through the key factors to consider when evaluating an incrementality solution. Whether you’re actively evaluating incrementality platforms or simply curious to learn more about incrementality testing, this article is here to help ensure that you’re asking the right questions to find the best incrementality platform.

How to Choose the Right Incrementality Platform

The most important consideration areas for evaluating incrementality platforms are:

• Methodology and scientific rigor
• Experiment design
• Results and analysis

Methodology and Scientific Rigor

A scientifically rigorous experiment ensures that results are reliable and unbiased, so you can be confident in your decision-making. This is mission-critical because flawed experiments risk providing you with inaccurate and misleading data – and bad data can lead to bad decision-making.

You should look for a solution that…

Runs experiments with a holdout

Unlike traditional metrics that measure actions like clicks or impressions, incrementality focuses on causation‌ — ‌showing whether an ad led to an actual, measurable increase in sales or conversions that wouldn’t have otherwise happened.

The only way to confirm your marketing tactic is driving growth is to run an experiment with a holdout. A holdout group shows what results would occur without the campaign. By comparing the holdout group (no exposure to ads) to the test group (exposure to ads), you can isolate the true effect of the campaign.  

It’s done the same way that scientists test new medicines‌: using a control group (not exposed to the drug) ‌and a treatment group (exposed to the drug). By isolating all other variables with a holdout group, you can rest assured that your marketing effort is driving real results. If there is no holdout, it’s not measuring incrementality. 

Uses frontier methodologies to maximize experiment accuracy and precision

While analyzing test and control groups may seem easy at first, it’s‌ extremely difficult to do properly. That’s because when the goal is to eliminate as many external variables as possible‌ — essentially, you want these two groups to be identical. Otherwise, how do you know there isn’t something else skewing your results?

Incrementality products should employ innovative methods in control selection, anomaly handling, and analysis. This likely requires having experts on staff. Experts in causality, like economists and data scientists, understand the scientific rigor necessary to yield strong results and are up-to-date on the latest research.

They employ frontier methodologies and are constantly optimizing the science that powers their experiment design and modeling to give you the most accurate results possible. 

One way to gauge whether a vendor is using innovative methodologies is to ask how many PhD-level scientists are on staff.

Uses synthetic controls, instead of matched markets

Many incrementality solutions use matched markets tests, which means similar geographic areas are compared head-to-head. For example, a solution that used matched markets might match Detroit and Milwaukee, putting Detroit in a test group and Milwaukee in a holdout group because they are similar in size, some behaviors, and certain characteristics. However, this one-to-one comparison isn’t a perfect match — there are still many differences between these two locations, which means more variables that can skew experiment results. 

Synthetic controls create a “synthetic” group, which combines data from multiple markets that didn’t receive the treatment to most closely resemble the treated market’s characteristics. If we use the Detroit vs. Milwaukee example,  Detroit could instead be matched with a synthetic control that’s 60% Milwaukee, 30% Cleveland, and 10% Minneapolis, so it looks and behaves more like Detroit does. Synthetic controls drive more accurate experiments.

Runs thousands of placebo tests before experiments to ensure optimal test design 

A “placebo test” is a method used to validate experiment design and modeling strategy. For incrementality experiments, placebo tests involve looking at a period before the treatment was applied to see how closely the test and control are matched or if they would falsely detect an effect. If the model doesn’t find a significant impact during the placebo phase, it suggests that the experiment setup is robust. If the placebo test shows that there is an effect, that means that there could be issues with the experiment design. The best solutions extensively and expediently run placebo tests before the experiment launch to minimize the time to insights.

Experiment Design

Designing experiments should be headache-free. There was a time that designing experiments from the ground up could take weeks, but with the right vendor, it's now possible to design highly sophisticated incrementality experiments in five minutes or less. At the same time, it's crucial to invest in a solution that's flexible, secure, and future-proof.

Find a solution that…

Integrates with your data provider/warehouse for fast data onboarding

Make sure to ask about a vendor’s integrations and how long it typically takes a new customer to onboard. For example, does it offer integrations with Snowflake, Shopify, BigQuery and other data providers that are essential to your business? If there isn’t an easy way to integrate with your data providers/warehouses, then it could radically prolong the time it takes to get your first experiment off the ground. 

Doesn’t need pixels, cookies, or personally identifiable information (PII)

Incrementality testing should be privacy-durable and rooted in the data that you have easily accessible: app downloads, revenue, orders, etc. Vendors that rely on pixels or third-party cookies are vulnerable to changes in privacy policies — you want a future-proof solution. 

Can enable you to design experiments in minutes

Historically, one of the biggest gripes with incrementality experiments is that they take a long time to design and launch (which makes sense — you want to ensure you are taking all the steps necessary to design an experiment that will give you results you can trust). However, there are incrementality platforms that synthesize several weeks' worth of data scientists’ efforts into experimentation software that allows you to set up a test in a matter of minutes. This is an extreme time and resource saver, enabling you to test, learn, and optimize quickly and efficiently. 

Offers holdouts at the DMA and state level

While many solutions offer holdouts at the state level, Designated Market Areas (DMAs) provide a finer level of targeting, allowing for a more precise control group. DMAs are defined by media consumption patterns instead of state lines and tend to capture more localized consumer behavior. State holdouts, on the other hand, cover larger and more diverse populations, creating challenges in controlling external variables. 

For example, New Jersey has two DMAs — “New York” for those who live in North Jersey (closer to New York City) and “Philadelphia” for those who live in South Jersey. These major cities, both of which aren’t even part of New Jersey, create many differences that wouldn’t have been accounted for if holdouts were only offered at the state level. 

Offers 2- and 3-cell experiments

2-cell experiments — those with just a test and holdout — are the most common designs for incrementality experiments. However, 3-cell tests — which include a holdout and two test groups — can yield powerful insights. 

With 3-cell tests, you can run more complex experiments and determine if you could be spending more (or less) and still be efficient. Jones Road Beauty ran a 3-cell experiment to see if they could spend more on YouTube, and they found that they could spend 2x more and see 2.6x new customer orders. FanDuel ran a similar 3-cell experiment on YouTube and saw that spending more didn’t equate to more activations.

Enables you to balance experiment speed vs. precision

When designing an experiment, there are often tradeoffs to be made between factors like speed and precision. A longer test or a larger holdout group might improve the confidence level around results (i.e. provide more “power” to the experiment). However, those tests have a higher opportunity cost, and you have to decide what is right for your business. 

Let’s say you're creating an experiment, and to run a test that has a power of 95%, you have to run an experiment for 4 weeks with a 30% holdout. However, having such a large holdout for a month may be too much of a risk for your business. You want the option to control the power of experiments to meet your needs. For example, if you decide to decrease your power to 90%, your experiment length and holdout could decrease significantly while still getting strong data insights.  

Provides recommendations for adjusting your ad spend during testing periods

When running holdout tests, you want a solution that provides guidance around adjusting your weekly ad spend based on the size of the holdout group. This can ensure that you maintain consistent spend-pressure across the regions where you’re running ads so to not skew your results. 

Results and Analytics

Experiment results should be clear, actionable, and answer important questions like how ads holistically impact sales channels. 

To be certain your results are truly actionable, look for solutions that…

Provide a holistic view of how ads impact KPIs and sales channels

When you’re running incrementality experiments, there’s a treasure trove of information to be gleaned. Even if your primary KPI is new customers, you still might be interested in how a campaign impacts total orders. Or, if you are looking at how incremental Instagram is, you may want to know how it impacts your Amazon sales. That’s why it’s important to invest in a solution that enables you to track multiple KPIs and sales channels during the same experiment. 

Offer post-treatment windows

Customers rarely see an ad and then immediately purchase. Especially for high-ticket items, it might take longer to see if an ad impacted a customer’s purchase. That’s why it’s important to look for solutions that offer post-treatment windows that show how ads continue to impact performance after the experiment ends. 

As an example, Hexclad was able to use post-treatment windows to see how critical an extended measurement view is to account for the entire multi-week journey customers take to make an expensive purchase. 

Calibrates platform metrics for daily incrementality reporting

Growth marketers often rely on platform-reported metrics — but these metrics rarely tell the full story. That’s why it’s important to find a testing solution that adjusts your platform reporting based on experiment results, allowing you to reallocate your budget to the channels and tactics that have the greatest incremental impact. Look for solutions that automatically update metrics like incremental return on ad spend (iROAS) and cost per incremental acquisition (CPIA) daily to power agile marketing decisions.

Choosing the right incrementality platform is about more than onboarding a new tool — it’s about building a foundation for confident decision-making. With a robust incrementality solution in place, marketers can get to the heart of their advertising’s true impact, enabling smarter spending and driving sustainable growth. By ensuring that your chosen platform meets rigorous standards in methodology, experiment design, and actionable insights, you’re setting your team up for success.