|
|
This edition of TMAI Premium is free for Standard Edition subscribers.
Recent TMAI Premium editions shared the incredible data collection < > data output matrix to help foundationally transform your analytics practice, advice on dramatic changes required in Social Marketing with the social graph replaced by the content graph, and the powerful benefits accrued when Analysis Ninja's use suspicion as an ally.
You can upgrade to TMAI Premium here. All Premium revenue is donated to charity. | | TMAI #347: Solving Incrementality: Hurdles & Distractions. P1.
| You get a report that shows you earned $10 million in Revenue from 100 Conversions.
Awesome.
These 100 Conversions in the Google Analytics (GA) dashboard are of course last-click.
You are a sweet and curious person. You whip up a custom report to separate the last-click Conversions by Owned, Earned, and Paid Media, and you see this breakdown:
Owned: 20. Earned: 20. Paid: 60.
As an experienced Analyst, you know that the last-click report shows imprecise allocation across all three. Maybe, Owned should really be 40 and Paid and Earned are stealing credit. Bad, bad, bad!As you are aware of my wonderful Attribution Ladder of Awesomeness, you dive into GA's Attribution section and apply the Data-Driven Attribution model. You now see a more accurate distribution of credit:
Owned: 25. Earned: 40. Paid: 35. At the end of attribution analysis, you still have the same 100 Conversion you started with in your last-click report. They are just credited a little differently.
This is good for your CMO (and indeed everyone in Marketing).
This is not good enough for your CFO.
Her/his words to you after you present attribution results:
Thank you.
It is better to have a more accurate reflection of who gets credit among Marketing’s activity.
But, how many of the 100 Conversions would we have gotten even if we did no Marketing?
A smart CFO does not want attribution. A smart CFO wants incrementality. | Programming Note: I'm off on a new professional adventure. I've joined Croud as their global Chief Strategy Officer, with responsibility for strategy, product, data and M&A. Croud delivers innovative marketing strategies by connecting data, tech, and creativity. I am so excited about the team, recent acquisitions of Born Social and Verb Brands. Wish me luck! | The Invaluable CFO Incrementality Lens.A smart CMO is interested in the allocation of marketing budget across all marketing possibilities. A smart CFO is interested in the optimal allocation of capital across all company needs. Attribution vs. Incrementality.
It is important to emphasize that at some level as Marketers (or Sales) we realize that some number of conversions we are claiming as driven by us would have happened anyway.
Word of mouth, product quality, competitive dimensions, retail presence, pricing and promotions strategy, seasonality, and a dozen more non-Marketing all drive conversions (in fact they drive a vast majority of the conversions/revenue). Your Analytics team, your Agency, is currently reporting these as driven by Marketing.
Nudged by your CFO, you go back to your cubicle and measure incrementality using strategies you’ve learned as a TMAI Premium subscriber (#232, #233, #260, #333, #334).
It takes effort, and three months, but you do come back with an initial answer for your CFO.
New view of reality:
Conversions Claimed by Marketing: 100.
Conversions Actually driven by Marketing: 27.
Incrementality: 27%.
Bonus lesson: Attribution is not incrementality.
This sucks. This is fantastic.
It is not a great conclusion that a vast majority of the value Marketing is claiming to have delivered would have been realized anyway.It super sucks that your Cost Per Conversion reports will get destroyed. You spent $7 mil earning those 100 Conversion, hence Cost Per Conversion was $7mil/100. Now the Cost Per Incremental Conversion is $7mil/27. Just imagine what that does to Marketing’s profitability. Ouch.
It is fantastic that 27% of the Conversions would have never happened without you! It is truly a gift to have the outstanding opportunity to A. identify the pattern across those 27% and leverage them to move to 29% and then 31% and on and on, and B. ruthlessly stop the chunks of wasted Marketing that it in the 73%. A + B is so much fun to pull off, and such a wonderful outcome for the business.
This will not shock you: Marketers welcome the first set of incrementality reports just as much as colonoscopies. Channel Silo or xStack or Portfolio Incrementality. Initially, you will see 2% or 3% incrementality, it can feel crushing.I’ve patterned my brain to love and welcome incrementality. I want to know. Then, I can work with my peers, my remarkable Agency, and vendor partners to get on the path of creating unique business value. We can then proudly go to the CFO of the company and say: XY% of the business impact was exclusively driven by Marketing!
Result: A professional life filled with purpose. Nirvana. | The Fly in the Ointment.
It won’t take very long for you to discover that very few professionals have the I want to know attitude.
There is almost always vehement, and metaphorically violent, opposition to measuring incrementality – if not right from the beginning, then as soon as the initial results appear.
The Marketing ecosystem (teams, ad-platforms, Agencies, every one) already functions on the basis of Claimed Conversions (100). Less credit feels not so good, or an outright threat.
Instinctively, the ecosystem will push back against finding out 100 is actually 27 Actual Conversions (and remember, I’m being kind because incrementality will range between 0% and 20ish%).
As soon as the first incrementality results are delivered… There will be politics. Whispering campaigns will be unleashed against the Analytics teams. Wild theories about how models and experiments are flawed will be planted at every turn. Even your boss will want you to hide the results / neuter them / “rationalize” them / validate experiments with experiments. There will be a precipitous decline in the amount of love flowing your way for your smart, accurate, and excellent analysis. :(
Every Marketing VP demands to know what Marketing’s incrementality is. The reality is: They really don’t want to know. Understand this reality.
Often, you will be powerless to change it.
Reasons might include:
You are in a junior position.
You are senior but don’t have much influence.
You would rather not be the tall poppy that gets chopped first.
Your manager/director/CMO has built a culture where accountability is optional.
You want to avoid jeopardizing your upcoming maybe job promotion.
Your culture has tons of pressure to conform, none to innovate.
You might not have many allies. (Ex: Finance, Strategy teams.)
At this point, you get to decide… 1. Give in. 2. Do what is right.
If you choose #1, this is the point at which you look up from this email, hit the little x on your browsers top right, and go look for roses to smell.
If you choose #2, I'm here to help. I'll identify the main hurdles and distractions that will be thrown your way, along with specific advice on how to deal with them. It is always hard to do the right thing, but you sleep better.
[Note: Occasionally your CFO is really leaning-into Marketing’s ROI, or the economy is in a recession, or the business is doing really poorly… Then, you might have allies, Finance, who will make the above choice for you. And, that is awesome! It feels good to have friends in high places.]
If you are still reading, you've chosen #2... Let's learn the common hurdles/distractions that will be thrown your way by the anti-incrementality crowd… And how to address them. Two massive hurdles today, six more in the next two Premium editions of TMAI. | H1: The “lost revenue” objection.
Channel Silo incrementality (TMAI Premium #333, #334) is seeking the answer to this question:
How many of the 1,000 Conversions claimed as driven by Bing Paid Search would have happened even if we spent no money on Bing Paid Search?
Both Bing (platform) and Paid Search (tactic) are used here as an example. Just replace both with Google/PPC, Facebook/In-Feed or Hulu/6secs or TV/60secs or Email or Billboards, etc., and my advice below still applies.
The most common hurdle/distraction thrown your way, via your internal Paid Search team, and/or your Agency, and/or Bing itself might be:
To measure incrementality, we can easily run an experiment. But. During the experiment, you will “lose” the conversions from the control market/s. Why don't you just use Attribution?
[Note: If someone is knowingly recommending you to measure attribution as a replacement of incrementality, they are not your friend.]
To address this hurdle/distraction, paint the big picture…
Say today you have 1,000 Claimed Conversions reported from Bing. To find out how many of these Claimed 1,000 are incremental, you will use Conversion Lift Studies in Bing. As a part of that, of all the people who qualify to see your ads, a small percent will not see your ad (control group) and others will see the ad (test group).
It is true that you will not get any Claimed Conversions you otherwise might have from that small percentage of people in the control group. Very true.
But.
At the end of this CLS, you’ll get the gift of discovering that Bing’s Paid Search incrementality is, say, 15% (perhaps even less).
So.
Claimed Bing Conversions: 1,000. Actual Bing Conversions: 150.
The hurdle being thrown your way to stop incrementality measurement is due to the number of “lost” Claimed Conversions from the control group.
Respectfully, help them understand that the focus should be on the “lost” Actual and not "lost" Claimed conversions from the control group.
Then, quickly, compute the tiny loss:
Formula: Size of Control (say 20%) times % Incrementality times Claimed Conversions
Actual, not Claimed, Lost = 0.2 * 0.15 * 1,000 = Negligible "lost."
It is well worth discovering whatever you are doing with Bing PPC is a lot less cool than being implied in your reports to Sr. Leadership. AND, your PPC practice needs a massive overhaul.If you spend tons and tons on Bing PPC, you might not need as big a control. In that case…
Actual Lost = 0.02 * 0.15 * 1,000 = Less than negligible.
You are spending tons and tons, but you have never measured incrementality of Bing PPC, it is almost certain that it is extremely low. In that case…Actual Lost = 0.02 * 0.02 * 1,000 = Minute tiny small negligible.
Obviously, your numbers will be unique to you.
When the “lost” revenue hurdle is being thrown on your path as a BIG DEAL… Do the math.
The real impact will almost always be a minimal price on the path to a material transformation of your Marketing’s effectiveness.
Have a polite discussion with those who say incrementality measurement will result in “lost” revenue. Help them see the big picture.
[Note: I’m using CLS as a measurement strategy. It is offered by Google, Facebook, all major platforms to identify channel-silo incrementality. You might run a large-scale geo experiment, or a related testing strategy. No matter the methodology, everything above applies.] | H2: The “but you are only measuring a piece / short-term” objection.
Here’s how to understand this one:The goose is laying square eggs. Eggs are supposed to be oval. Useless goose! It is true that among the basket of incrementality measurement strategies, some can only measure incrementality of a piece (ex: Channel Silo), others can only measure the impact over the short-term (ex: three-month duration of the experiment).
My perspective: It better to know your incrementality is in a silo/short-term, than not knowing it at all.
Definitely, it will not be perfect. It’ll miss something, or the picture won’t be complete.
But, if you are claiming 100% incrementality today and the reality is 17%... Consider how poorly you are using your marketing budget by missing a part of the picture.
Maybe it is 20% and not 17%. We are claiming 100%!!
As the wise Athena said: Never let the perfect be the enemy of good enough.
Like Zeus, listen to Athena. M’kay?
Grab every chance you can to measure incrementality. Over any time, for any campaign, on any channel. Every signal will make you more effective.Keep accumulating more signal and better signal over time. My recommended plan:
1. Use Conversion Lift Studies (CLS) to measure Channel Silo incrementality on any channel where CLS is available (ex: G, FB). You can change duration, size, coverage of CLS. Use it to make every other week or monthly decisions about changes to G, FB campaign execution to improve incrementality.
2. Use geo experiments (MMTs) to measure xStack incrementality (ex: incrementality across Google AND YouTube AND GDN or FB AND Insta). This is entirely manual, you have full control around duration, size, coverage. Use the results, which will typically arrive each quarter, to modify the budgets allocated across channels. Leverage #1 to understand results and awesomely improve incrementality.
3. Use media mix models (FB’s Robyn, or one based on machine learning) to measure portfolio incrementality. This will cover almost all Marketing you do: brand plus paid plus retail plus organic plus all. Use the results, which will be annual at the start (and then every six months) to decide how much budget to allocate to Marketing. Leverage #2 to allocate it, and #1 to execute.
BOOM!You are measuring almost the entire portfolio, over the long-term, medium term, and short-term. You are making incrementality decisions bi-weekly/monthly, quarterly, annually.
Happy birthday.
The problem is: This takes a lot of intelligent work, well-paid experienced Analysts, and patience.
Most companies don’t have these elements (or have never measured incrementality and don’t know they desperately need this investment to have deliriously happy customers and unhealthy profits!). [Obviously, now you have the option of hiring Croud!]
If that is your reality… It is ok. Deal with this hurdle by listening to what Athena said above. You can still win loads and loads.
[Note: For this objection, ponder this as an answer: If you are unable to demonstrate any incrementality in the short-term, how does that failure get converted into long-term success? Perhaps there is an answer, ask the question.] | A ton more advice.
I hope this edition of TMAI has helped you deeply appreciate just how crucial solving incrementality is, and how complicated it can be (not because of measurement, but because of culture).
My goal is to deliver a step change in your impact on your employer by being able to solve both those sides.
There is a lot more to learn.
In the next two TMAI Premium editions, subscribe here, we will continue our transformational journey on incrementality. I’ll cover six more complex hurdles/objection that will be erected on the path to glory. And, from my successes and failures, share how to overcome them effectively. | | Bottom line.
As measurement becomes more sophisticated, as we deploy every smarter machine learning algorithms, Marketing’s incrementality (or lack thereof) will inevitably come into sharper focus.
If you want your career to head up and to the right, read every single world of this edition of TMAI (and the next two) carefully and multiple times. Understand the nuances, learn to see the hidden layers.
Solving incrementality is, literally, adding value to a business that nothing else is. But, solving incrementality is a cultural problem. It is not natural for Analysts to be good at that. Which is why I’m spending so many, many words on it.
See you next week.
Avinash.
| Committed to investing in your professional growth?
Upgrade to TMAI Premium here - it is published 50x / year. | |
|
|
|
|
|
