The Ultimate Guide to Marketing Incrementality

Whether you’re trimming your marketing budget or increasing your spend, to make an informed decision you have to test which channels are the most beneficial for your business. Traditional attribution models are many marketers’ go-to measurement method. The problem? These approaches only tell part of the story. 

Last-click attribution gives all the credit of a conversion to just a single touchpoint, without being certain that it’s the one and only touchpoint that led to the purchase. Multi-touch attribution models are built on user-level tracking which is fundamentally breaking down as privacy laws surge. 

Marketers can be led astray with these attribution models and are left gambling with their budgets.

Thankfully, there’s a better way. 

Incrementality testing allows you to vet every step of your funnel to ensure you truly know where your marketing dollars are best spent. In this article, we’ll dig into what incrementality is, why you should use it, and how to run incrementality experiments in three steps. 

What is Marketing Incrementality?

Incrementality refers to the increase in a desired marketing outcome resulting from isolating a single marketing activity, like a new Facebook campaign. 

Incrementality experiments utilize the scientific method to get accurate results. A data scientist starts by identifying two groups of consumers that are characteristically similar. The treatment group receives the marketing treatment you want to test, while the holdout group does not receive that treatment at all. The two groups are run for a specified length of time, during which the conversions for each group are gathered. Then, the analyst compares total conversions in each group to determine if the tested approach (treatment group) drove a lift in conversions compared to the holdout. 

Say you’re a marketing manager for Starbucks, and you want to test the impact of a holiday campaign. You run an incrementality experiment comparing a group who did not receive the holiday campaign with one who did. The percentage of how many more conversions were observed in the treatment group is the incrementality of the new campaign.

Why Measure Campaigns with Incrementality?

Incrementality has several benefits over its counterparts like marketing mix modeling
(MMM) and multi-touch attribution. It’s more than just an accurate way to gauge marketing efforts; It's also a versatile method, since it can establish causation for both digital and non-digital media.

Establishes Causation of a Single Variable  

Incrementality experiments let marketers isolate variables and observe the impact of a single marketing tactic, so they can reliably gauge what factors are driving conversions.

Consider our Starbucks example again. A number of customers who received the holiday campaign will buy the coffee regardless of the marketing activity. Maybe there’s a Starbucks drive-through on their way to the office, or maybe they’re addicted to a drink flavor only Starbucks carries. In marketing mix modeling or multi-touch attribution, these customers would be included as conversions driven by the campaign even though the campaign itself was not actually responsible for driving the sale. 

Incrementality, on the other hand, uses a holdout group to remove these customers from your attributed conversions. If you ran your experiment and found that the holdout group had 2,000 purchases in a week while the treatment group had 2,700, you can assume that 2,000 of the 2,700 from the treatment group were customers who would have converted regardless of the campaign (assuming the treatment and holdout groups were the same size), while the additional 700 were incremental due to the campaign — comprising the true “incremental lift”. 

Isolating just a single variable (assuming the experiment controls are kept as constant as possible – more on this below) establishes causation. In other models, in contrast, you are left trying to find correlations between many contributing factors. 

Want to learn more about why attribution tests can’t stack up to incrementality experiments in today’s marketing environment? Visit our blog that compares attribution vs. incrementality.

Measures Both Online and Offline Media 

Incrementality experiments can be used for both digital and non-digital measurement, while multi-touch attribution can only truly measure digital channels. 

This may not seem like a big deal, but it is in today’s hyper-connected society. According to McKinsey, customers see a product on three to five different channels before making a purchase. Your experiment will have a blind spot if you’re only measuring digital channels without comparisons to channels like direct mail or OTT. 

In contrast, incrementality experiments follow the same process regardless of whether the marketing tactic is online or offline. Ensuring all other variables are held constant allows you to measure all marketing the same way. This not only applies to online and offline media but can also be used to compare upper and lower funnel tactics as well as brand vs performance media

Allows for Accurate Cross-Channel Comparison

Comparing performance metrics across platforms assumes that each platform's method of measuring conversions is the same – which isn’t true. Each platform’s pixel and tracking methods vary significantly. Even Facebook’s conversion lift product uses a different methodology than Google’s incrementality offering so the two results cannot be accurately compared to one another. 

Finally, some incrementality approaches, like geo-experiments, create a standard unit of measurement regardless of the platforms you’re comparing. It lets you compare two different channels apples to apples.

How to Run an Incrementality Experiment 

While incrementality experiments seem simple in design, actually running them well takes a handful of skilled data scientists and economists to ensure the results are accurate and precise.

If you’re looking for an easier way, consider outsourcing your experiment to a partner like Haus. Haus handles the time and labor intensive process of designing, running, and analyzing your experiment, allowing you to go straight to your results. Otherwise, read on to learn the steps you’ll need to take to create an accurate and informative incrementality experiment.

1. Design the experiment 

Determine what marketing variable you want to test. Then, set your two audience groups (treatment and holdout), the variables you will maintain constant, the test variable you are measuring the impact of, your key performance indicator (KPI), and the runtime of the experiment to determine the experiment’s power. 

Identify and set controls 

Your controls are all the factors of your experiment that will remain the same, regardless of group. Basically, it’s every factor that could affect your experiment other than the one concept you’re testing. 

To create successful controls, separate your audience into two groups that are characteristically indistinguishable from one another. Alternatively, you can select two groups that are large enough in size to buffer any outliers or variations in personal preferences.

Haus, for example, creates accuracy in its testing environment by creating stratified market variation and leveraging synthetic control methods.

Determine your one variable 

Your variable is the single condition you’ll change between your two groups to test its impact. One group (the holdout) will receive only the “business as usual” (BAU) standard approach, while the other (the treatment) will receive the BAU, plus this added variable.

While incrementality is often used to determine the true value of a particular marketing channel, you can also gain more granular insights than just channel level incrementality. You could test if a particular buying type is more or less incremental or if static or video ads drive more conversions for you, for example. 

The Primary KPI

This metric represents the primary action you’re looking to drive—such as a lead form submission or a sale. 

Say, for example, your experiment is trying to determine the best marketing channel to increase the number of sign-ups for your company’s subscription product. The primary KPI for this experiment would be subscriber acquisition.
Ask yourself what metric will most clearly reflect the strategic goals of your marketing efforts, and then ensure your organization is collecting – or can collect –  that KPI by the end of the experiment. 

Experiment Runtime 

Finally, you’ll need to determine the right length of time to run your experiment in order to get reliable results. Consider the following factors to come to a conclusion. 

• Experiment cost: How much are you planning on spending in this experiment? Typically, the longer you run an experiment, the more money you will inevitably spend. On the flipside, the more you spend, the higher the experiment power is, resulting in a higher confidence level more quickly.
• Opportunity cost: Running an incrementality experiment requires you to “sacrifice” a certain portion of your audience by not exposing them to a marketing tactic that could turn out to be extremely influential. You must determine what portion of your audience you’re willing to “under market” to, and for how long.
• Data accuracy: Longer experiments will yield more accurate results, since the length of time can eventually filter out any extenuating circumstances or outlying data but again, you must weigh this against the opportunity cost.
• Testing channel: The channel you’re measuring will need to be considered in making a duration decision. Offline or print channels often take more time to report on because people don’t typically take action on these marketing tactics as quickly as they would clicking through a facebook ad or a google search ad.
• KPI expected volume: Understanding the standard volumes of your KPI helps you to properly calculate your experiment’s power, which should also be considered in runtime

While deciding on an experiment timeline alone can be tricky, it’s made simple when you partner with Haus. Our platform includes a calculator that shows the trade-off between the factors listed above and how powered your test will be, so you can confidently choose the duration and holdout percentage that’s best for your business. 

2. Launch Your Test

After finishing the experiment design and setup, it’s showtime. 

Ensure you have a solid strategy for collecting KPI data from your holdout and treatment groups. Then, deploy your new strategy to your treatment group, while continuing to track your holdout groups’ behavior as well. How you release your new test strategy will depend on the channel in question. If it’s a Facebook ad, it may be as simple as setting a geo-location for your treatment group and going live. If it’s an out-of-home test with physical billboards, on the other hand, you’ll have to decide on the target geos and partner with an agency on execution.

3. Analyze your results 

When the experiment period is finished, take your KPI data for each group and plug it into the standard incrementality rate formula. 

[(Variable group KPI - Holdout group KPI) / Variable group KPI] x 100% = the incrementality rate (as a percentage). 

Let’s use the subscription sign-up example here again. Say your variable group with the new marketing approach saw 500 new sign ups in a week, while the holdout group had 200 new sign ups. 

500 new sign ups from the treatment group - 200 new sign ups from the holdout group = 300 signups from the treatment group attributed to the new marketing approach. 

You know now that 300 of the 500 total treatment group sign-ups are because of your new tactic. But most marketers measure incrementality as a percentage of total signups within the treatment group. 

300 incremental sign ups / 500 total sign ups in the variable group = 0.6 x 100% = 60% incrementality for the new approach or lift percentage.

Once you’ve finished this calculation, you’ll have a number telling you what percent of all conversions over your time period can be definitively credited to your marketing treatment. Use your results to see which marketing channels are paying off for your brand (and which ones aren't).

Channel-by-Channel Comparison 

To do a channel-by-channel marketing analysis, you’ll need to repeat an incrementality experiment with each of your marketing channels, applying the same methodology each time. This allows you to compare each channel’s effectiveness against the others.

Say, for example, that your company has invested marketing budget in Facebook ads, Pinterest sponsored pins, billboards, and OTT commercials to increase subscription sign-ups. You run an incrementality experiment on each channel and determine the following channel lift percentages: 

• Facebook ads: 8%
• Pinterest sponsored pins: 6%
• OTT: 4%
• Billboards: 1% 

These results show you that the best “bang for your buck” in ad spend comes from Facebook ads, followed by pinterest sponsored pins, OTT, and, finally, billboards. Note that these percentages won’t add up to 100% like they would in a multi-touch attribution model, because they aren’t all being measured at the same time. 

Beyond lift percentages, marketers also consider the cost per incremental acquisition to determine if a channel is worth the investment. 

CPIA (cost per incremental acquisition) or CPI (cost per incremental) = Total campaign spend / incremental conversions 

It is possible that a channel could have a lower lift percentage but be so inexpensive that the cost per incremental acquisition is still worthwhile. 

For example, we often see that brand search is not a very incremental tactic, but it's also very cheap. The CPI is sometimes so low, that brands still find efficient acquisition from this tactic and decide to keep investing it in at low spend volumes. 

It’s important to consider both lift percentages and CPIs for each experiment you run.  

Reporting Metrics Calibration 

Instead of repeating experiments on every single marketing tactic, some people use incrementality to calibrate their ad platform reporting. Incrementality can de-bias these reports by either confirming or discrediting platform attribution. Say, for example, you’re trying to decide if you should spend more on Google search or not. Google’s attribution claims that their ads are responsible for 2,000 of your conversions, but you’re not convinced. 

You can run a short incrementality experiment on Google search ads and compare your results to Google’s platform attributed results. If they’re similar, you’ve confirmed that Google’s attribution is on point. If, however, Google search ads drove fewer than 2,000 incremental conversions, Google's platform reporting may be inaccurate. This approach gives marketers more accurate data to decision on

Regardless of whether you decide to fully commit to incrementality, work with a partner like Haus to accelerate incrementality testing, or adopt a hybrid model of calibrating platform or MMM with incrementality, one thing is certain. Incorporating this approach into your measurement approach will make for a much more complete and accurate picture of your marketing efforts.

Measure Incrementality Easily with Haus 

More than any other measurement approach, when incrementality is done right, it provides a true look at the effects of your marketing. But running well-designed experiments frequently can be complicated and time-consuming. Without an analytics partner like Haus, marketers must do all the heavy lifting of experiment design & analysis on their own.

Thankfully, there’s an easier way. 

Haus’s experiments platform automates all the heavy lifting of experiment design and analysis. Our cutting-edge software brings you thorough, accurate incrementality results in weeks, not months, so you can make better, quicker investment decisions and stop guessing.