Incrementality School, E6: How to Foster a Culture of Incrementality Experimentation

“A fool with a tool is still a fool.”  -Grady Booch

If you’ve read any of our previous blog posts in our Incrementality School series, you might come to the conclusion that media measurement and incrementality is incredibly hard. Well… yes (I mean, we wrote a whole blog series to help).

At Haus, we provide what we believe to be an essential tool that makes it much easier for many brands. But as renowned engineer Mr. Booch said so eloquently above, having the right measurement toolkit for your business is only meaningful insofar as your team’s ability to use that tool. 

Having spent my career helping companies to make sense of the impact of their media, one of the most important factors in a team’s ability to effectively utilize media measurement tools is having a true “culture of experimentation.” 

So, what is a culture of experimentation?

A 2020 article from Harvard Business Review breaks it down pretty nicely (TLDR: cultivate curiosity, prioritize data over opinion, and democratic experimentation). In this blog post, I present a complementary framework that can help you assess just how strong your organization’s culture of incrementality experimentation is.

I’m of the belief that the world can often be broken down into a simple 2x2 matrix, and in this instance, I posit that a strong culture of incrementality experimentation comes down to your organization’s level of measurement savviness and path to actionability from your experiments. 

A framework for assessing your “culture of experimentation”

Let’s break this down a bit further.

Measurement Savviness

Marketing measurement’s constant state of flux over the past few years has been well-documented, and 2025 should continue to see greater evolution (see Rob Webb’s useful overview from his Growthish newsletter). In order to gain a competitive advantage, marketers have to have a deep understanding of the pros and cons of each measurement tool available (e.g. Incrementality, MTA, Media Mix Modeling, in-platform attribution), and a discerning eye to understand the various methodologies that underlie each tool, particularly experimentation solutions.  

Here are a few signals that I’ve found to be key indicators of an organization’s measurement savviness:

• Level of advanced measurement knowledge: A very savvy organization understands the key differences of each of the areas of media measurement (incrementality, MTA, media mix modeling, in-platform attribution, etc.), and how you might prioritize and utilize each tool in conjunction with another. For example, a savvy marketer would utilize incrementality to fill in gaps in their click-based MTA solution to understand the halo effects of an upper-funnel YouTube or TikTok campaign has on offline sales. They also know incrementality experiment results should constantly calibrate their MMM, if they use one. On the flip side, a more nascent marketer might rely solely on last-click attribution as its north star, and might not be ready to jump right into building an incrementality experimentation program. 

• Level of methodological understanding: This one is for all my data scientists and economists out there who know that not all incrementality experiments are created equal. Less savvy organizations see incrementality testing and experimentation as a commodity, and are less likely to distinguish between various types of testing (i.e. A/B experimentation, incrementality hold-outs) and methodologies (i.e. quasi-experiments, matched market testing, synthetic control, difference-in-differences, etc.). The methodologies and test types you choose can have real implications for your business, and, most importantly, how confident you can be in making multi-million dollar media allocation decisions based on experiment results. You certainly don’t need to be a PhD to run effective experiments, but it helps to have folks on your side to help you make sense of your options.

• Focus on data-driven decision making: This is pretty self-explanatory. What level of importance does your organizational place on making sound data-driven business decisions? There’s two sides to this. For example, some marketers rely heavily on the gut intuition of their founders, or let their creative teams lead the way. Data is rarely present. On the flipside, some marketers rely too heavily on data, often paralyzing decision-making if the data isn’t perfect and all pointing toward the same direction. Companies with a strong culture of experimentation know the balance. They grasp the concept of an “opportunity cost of testing” (i.e. the potential hit to your business by not advertising to the holdout group), are comfortable using a mix of art and science, and embrace experiment results may be directional vs. perfect, all while choosing to move forward in pursuit of a more data-driven decision. 

Path to Actionability

Experimentation without action is just an academic exercise. As much as it would be nice to revert to simpler times sitting in our Intro to Stats lecture hall (or Zoom calls), a marketer’s job has real stakes and requires us to, you know, actually act. A clear “path to actionability” ensures that experiment owners have the authority and resources to implement changes based on the insights generated. This empowerment ensures that valuable learnings translate into tangible improvements across your organization.

• Clearly articulated use cases: As with any journey, the path is easier when you know the endpoint and the guardrails before you start. Having clearly articulated research questions and use cases for your incrementality experimentation that are – very important – also directly tied to your critical business objectives will be key. In my experience, organizations are much more likely to act on experiment insights and see improvements if they’ve spent time thinking about their experimentation roadmap for the year and are prioritized by their expected business impact.

• Variety of use cases: Somewhat related to the point above, but having a variety of use cases in mind for your incrementality experiments is also critical. Companies with a strong culture of experimentation know that sound incrementality tests could help you understand the impact that any of your key marketing channels on a variety of your business KPIs, from online sales to offline sales, from conversions to upper funnel awareness metrics, and across a wide range of time periods (from short-term to longer-term). On top of that, these savvy marketers know that incrementality test results should be continually reevaluated based on seasonality, changing consumer and macroeconomic trends, and any other factors that might render previous results less applicable.

• Speed from insight-to-action: Simply put, this is an assessment of how well your organization is to quickly action on the results of an experiment. I’ve often seen that organizations not in a position to act quickly are held back for a variety of reasons. For example, experiment owners lack internal political capital to drive change, or different parts of the organization have misaligned incentives (e.g. brand teams prioritizing upper-funnel KPIs, performance teams prioritizing online conversions), or the experiment owner is able to act, but not in a meaningful way tied to what the C-Suite cares about. On the flipside, organizations that have very strong alignment between the day-to-day experiment owners and the business decision-makers often act quickly and decisively on the most critical questions pressing their business.

Sticking the experimentation landing

These two factors – Measurement Savviness and Path to Actionability – define the strength of an organization’s culture of incrementality experimentation. 

If your organization falls short in either of these, know that you aren’t alone. It is a constantly evolving space and the bullseye keeps moving based on data privacy and regulatory changes, consumer preferences, etc. 

That said, using this framework can help diagnose how you move forward to build that strong culture. If you’re falling behind on the measurement savviness elements, it’s a sign that your organization has an opportunity for more education on the various types of measurement solutions and methodologies that exist, and how each tool might be useful for your particular needs. If it’s an unclear path to actionability that is holding you back, it’ll be important to take a step back and align on the top use cases and research questions your organization needs to address, and/or find a way to empower your experiment owners to be able to truly drive change within your organization.

And, well, if you’re nailing both – congratulations! Keep experimenting, keep learning, and keep growing.