What you measure is what you get

Share on email
Share on facebook
Share on twitter
Share on linkedin

If you optimize for installs, you will get more installs. Or at least more installs will be attributed to your ad spend. 

If you optimize for registrations, you will get more registrations. Or at least more registrations will be attributed to your ad spend. 

If you optimize for in app purchases (IAP), you will get more in app purchases.  Or at least more in app purchases will be attributed to your ad spend. 

I think you can probably see where this post is going. 

If you optimize for incremental revenue, that is revenue above what you would have received without advertising efforts. Then you will get more incremental revenue. 

Not to say that installs/registrations/IAPs aren’t important, they are. What’s important is that your efforts drive activity that increase these metrics above what they would have been without it. 

All that really matters in digital advertising is if your impression was able to impact revenue. The problem is it’s difficult to measure.

This is inherently the problem with the entire digital advertising ecosystem. The more we zero in on the metrics that are easily measurable, the more likely we are to cannibalize activity that would have happened anyways. 

AirBnB’s recent news

Airbnb recently announced that they are pulling the majority of their performance ad spend and focusing on brand effort. Executives stated that during the pandemic, as they pulled back significantly on marketing spend, their site traffic remained at 95% year over year. (Brand vs. performance is a distinction I have never liked. All advertising is performance advertising, if it’s not performing, why would you do it?)

Here is what I think happened: 

They weren’t effectively measuring their performance marketing

The only metric that really matters in advertising is incremental revenue. AirBnB wasn’t measuring this until 2020 (if they were they wouldn’t have wasted $662M worth of advertising budget). Based on the fact that their entire performance marketing effort was optimized to the wrong metrics, it doesn’t surprise me that this was the case. 

We can assume that AirBnB was optimizing towards booking value. This is fine, AirBnB wants their new and current customers to make more bookings, that’s how their business works. But as they seeked to reduced their cost per booking metric, their campaigns were optimized towards activity that produced a lower cost per booking likely because this booking was going to happen regardless if an in market ad made an impression or not. 

Scale this out over the long term and you advertising efforts have been optimized towards current and new customers that are going to make a purchase anyways! 

The effect was compounded by the size of their brand

As a company becomes larger it becomes more and more difficult to measure incremental efforts from your marketing campaigns.


Look at it this way. If you’re a brand new ecommerce company and you have no brand awareness, not site traffic, nothing. Then every sale you get through your ad spend is incremental. It wouldn’t have happened without the ads. 

But as you start growing and word of mouth starts taking off then any additional ad spend needs to be measured for it’s ability to drive incremental. It becomes more opaque, but still manageable. 

As business size increase the difficulty to measure incrementality becomes more important, but so does the difficulty

Not only did AirBnB fail to measure incrementality from their performance marketing efforts, they started doing it when they were already an established brand. So of course turning of performance advertising spend that has been optimized to the wrong metric for over a decade is going to show a negligible impact on overall revenue!

So what is the solution? 

Test incrementality early and often

There is no one-size-fits all solution for doing these tests. Each type of business is going to conduct these tests differently depending on various different factors. 

Early stage app businesses

When you’re still in start-up mode it still makes sense to periodically halt all advertising spend and monitor the impact on overall revenue. It’s a simple A/B test to see what real impact your efforts are having.

I can hear the gasps of most advertisers (and their Facebook reps). ‘Pausing is the worst thing you can do. The algorithm will never recover.’

I’ve come to believe that this is somewhat of a myth. Facebook and other platforms are incentivized to get advertisers to spend so by nature they don’t like it when campaigns are paused. 

Personally I would prefer to risk a small reduction in ‘perceived’ performance in order to assess if the campaign is actually performing. 

Mid size apps seeing traction

As your app business grows you’ll likely need to consider two different paths to help you understand the effectiveness of your campaigns. 

  1. You can evaluate using an unbiased third party to help you understand incrementality. Most mobile measurement partners have some solution for incremental measurement. There are also companies like INCRMNTAL who do only incremental measurement, not marketing attribution.
  2. You could consider starting to build an in house data team to help understand your data. 
  3. A combination of both. 

Large scale app businesses

Large app companies can and should have established in house marketing teams (and this includes marketing analytics). 

Although they can continue to rely on third party solutions to help measure campaign incrementality it’s important that their in house resources have the ability to leverage their first party data to measure incrementality.

In addition, customer privacy is (for good reason) becoming more important. Reducing granular data sharing with third parties is important and thus putting pressure on advertisers to build in-house capabilities to keep their customer’s data safe. 


As marketers we’ve become so obsessed with clicks, conversion rates, and events, we miss the point. Did that impression create additional revenue? Would that purchase have happened without that impression? 

What you measure is what you get. 

Feature Image (as seen in blog preview and on social media) Diana Polekhina on Unsplash

Find this article interesting?

Get notified when new articles are published.