Pitching a Data Strategy? Here’s How to Ensure the C-Suite Says “Yes.”

Are you a CMO who thinks accurate attribution is a pipe dream? Or a customer experience director who has to hack together data to create something resembling a customer journey? This article is for you. We’ll show you how to: Assess your company’s current approach to data. Map your data strategy against overall business goals. […]

The post Pitching a Data Strategy? Here’s How to Ensure the C-Suite Says “Yes.” appeared first on CXL.

Are you a CMO who thinks accurate attribution is a pipe dream? Or a customer experience director who has to hack together data to create something resembling a customer journey?

This article is for you. We’ll show you how to:

  1. Assess your company’s current approach to data.
  2. Map your data strategy against overall business goals.
  3. Build a data roadmap to deliver on the strategy.

Knowing what you’re up against

If I had a dollar every time I heard a business say they were “data driven,” I’d be rich.

While most companies are drowning in data, few have the strategy or cultural maturity to use it for accurate measurement, let alone to make decisions or drive business goals. And while two-thirds of executives think they need a data strategy, only a third are getting it done.

Despite these shortcomings, executives in your company might be wary of signing off on a data strategy. From where they’re standing, they’ve already spent a ton of money buying analytics and dashboarding tools, each of which promised to be the holy grail. So why do they need a data strategy? 

Here’s the problem. They’ve spent money on tactical elements that serve only isolated parts of the business. You need to think beyond the data point you need right now and create a strategy that answers wider business goals.

It’s tempting to develop a strategy that works just for your department. But your customers don’t interact just with your department. Your sales team has data in their CRM. Customer service reps have data on customer conversations. You need all that data to make better decisions. 

If, by contrast, you’re in a business where short-termism runs rampant, these scenarios should sound familiar:

  1. You have massive amounts of data, but different teams have access to different bits, copying and retaining the same data in their own ways.
  2. You can’t get an overview of a customer’s journey because the data you need exists in 10 different tools (that don’t speak to one another).
  3. People ask for specific data points to support the idea they’ve already had, rather than investigating the data first to inform a decision.

Those who cobble things together may get some early wins. But under the hood, things are getting messy, and data gaps will expose the fact that you haven’t thought strategically about your needs. 

If you’re ready to move beyond those short-term tactics, here are the three steps to do it. 

Step 1: Assess your company’s current approach to data.

We’ve created a set of 19 questions to kick off this process, which will help you evaluate the current data maturity within your business. We consider four critical areas for planning your data strategy:

  1. Strategy and culture;
  2. People and skills;
  3. Technology and tools;
  4. Methodology and process.

The scores gathered at this stage inform the roadmap in Step 3. You can also use these scores as a benchmark to reassess your business throughout the delivery of your strategy to quantify progress.

Step 2: Map your data strategy against overall business goals. 

If your company has fuzzy goals like “become customer-centric,” you need to pin them down to something quantifiable. 

My go-to approach for setting goals is OKRs (Objectives, Key Results) because they marry vague objectives to measurable results. OKRs also use a hierarchical structure, with company, team, and personal levels.

This goal-setting structure really helps when it comes to writing the data requirements in the next step. 

Example OKRs:

  • Company objective: Customers love our product.
  • Key results: NPS score increases to 30 by the end of Q3. Customer lifetime value (LTV) increases by 15% by the end of the year. 
  • Product team objective: Build features our customer want.
  • Key result: 65% of customers use new product features at least once a week by the end of Q3. 
  • Product manager objective: Evaluate new feature ideas.
  • Key result: Test four new feature ideas and identify which ones customers use most.
sample okrs at company, team, and individual level.

Creating a goal tree with data requirements 

The goal tree is a document that shows the C-suite how data ties to the business value they want to achieve.

Map the cascading OKRs at a company, team, and personal level. You may choose to work down only to a team level if your company has more than 30 employees. (The goal tree becomes hard to manage after that point.)

To ensure you don’t miss any requirements, talk to individual team members and summarize their requirements at the team level.

example of a goal tree.

Adding the data requirements

Work through each objective and key result with the relevant team:

  • What data do they need to achieve and measure what’s written?
  • Which KPIs will measure their performance? 

Avoid writing a technical solution at this point (e.g., “we need an API to join website data with Intercom into a dashboard”) and instead state the need (e.g. “we need to see customer website behavior data alongside Intercom data”).

The how should be led by your tech team.  

Questions to consider when writing requirements

1. What data do you need to inspire ideas and hypotheses? Say your objective is to increase sales revenue from your existing customer base, and your key result is to generate an additional $500k from these customers by the end of Q2.  

One of your requirements might be the ability to see the behavior of returning customers throughout the customer journey. From this data, you might identify that customers abandon their carts at a high rate, despite buying previously.

This data has helped identify a problem. To inspire a hypothesis, you may need the capacity to record and codify qualitative customer exit poll responses (i.e. your requirements). This data might show that customers abandon their carts because they’re looking for promo codes and then get distracted.

example of promo code.

Your hypothesis might be that if you remove the promo code field at the checkout, cart abandonment will go down, and revenue from existing customers will increase.

2. What data do you need to validate ideas? Which tactics might the team use to achieve their objectives? Continuing with our example, you might need the ability to test your hypothesis by showing half of existing customers the current experience, and half a test version (without the promo code).

Or, maybe you’ll try a cart abandonment–triggered email that requires measuring open rates, click-through rates, and conversions. 

3. What data do you need to report on key results? In this example, you’ll need statistically significant results to understand if you can rely on test data to make a decision. You also need metrics on cart abandonment and purchase rates of those who saw your test idea and those who didn’t.

Plus, you’ll want to measure the amount of additional revenue generated from the test.

Data-informed business decisions in practice

We recently worked with an ecommerce business that realized their subscription customers were worth more than one-off buyers. But they couldn’t be sure because they couldn’t track LTV and average order value consistently.

Even if they did have that data, they couldn’t act on it—they were unable to segment customers by LTV. The ability to segment would mean they could test and target valuable customers and provide marketing messages relevant to their stage in the journey. 

This led to us remodel their entire tracking. We started with offline attribution for recurring orders. Now, we’re wrapping up a data lake that stores data across all customer touchpoints on most channels, from acquisition to re-engagement.

This has not only helped inform hypotheses for experimentation—and considerably increased their conversion rates—but it’s also guiding wider business decisions, such as focusing on a subscription model vs. one-off sales, their product offering, and their sales approach and messaging.  

Step 3: Build a data roadmap to deliver on the strategy. 

To build your roadmap, you’ll need to work with your tech team to understand what and how things need to be done to deliver the requirements you’ve set out.

Ideally, you want to deliver these data solutions in sprints. This allows the business to:

  1. Start seeing the value of a holistic data strategy early on;
  2. Iron out any problems in your plans or working methods.

The roadmap should contain fairly detailed elements such as “implement a data warehouse” in the near term (e.g., a three-month window), with activities getting broader/fuzzier the further out in time they go.

As your team delivers elements of the roadmap, you can assess the success of what’s been delivered and add more detail on a rolling basis to the next three months of planned activity.

Define your starting point  

Use the answers from the maturity audit in Step 1. Working alongside your tech team, catalog what your current technical capabilities, tool stack, skills, and culture can achieve from your list of requirements in the goal tree. 

Work out what goes where in the roadmap

Now that you understand your starting point, you’ll need to prioritize the rest of the requirements. Here are six factors to rate by importance:

  1. Data security and governance activities should occur throughout the roadmap. Specific activities to comply with security or governance should be the highest priority. 
  2. Front-load activities that are easiest to implement and tied to the biggest wins. Consider any costs involved weighted against the business value associated with the requirements. Review and mine your existing analytics setup—often, there’s unused but needed, valuable data. Early wins can help gain support for your work and get the team familiar with new practices (e.g., agile). 
  3. Dependencies. Understand what has to happen first for something else to happen later (e.g., check the reliability of the data before you start personalization). 
  4. Factor in staff availability and consider other internal projects that might conflict with your roadmap. 
  5. Consider the budget process at your company. Do you have to wait until next April when budgets get renewed?
hindsight, insight, foresight diagram.

A strategy is more than just a plan

Gartner has a model to help illustrate how activities in your roadmap move your business toward data maturity:

  1. Hindsight: What happened?
  2. Insight: Why did it happen?
  3. Foresight: How can we make it happen?
cro with hindsight, insight, foresight analytics.

Access to the right data helps businesses move through these levels, but so too does cultural change and education. 

Get your CEO and managers to ask teams to present data-based insights rather than gut feelings to reinforce the correct use of data. Help others learn from your own work by modeling how to make decisions using data. You could even share insights on your business and customers in weekly company newsletters. 

The goal tree also helps in this area—teams rely on data to inform and measure their key objectives. These may be part of their performance review; if not, consider suggesting this to your HR team to incentivize the desired behaviors around data usage. 

from information to optimization.

People and skills

Educating individuals on how to use data to make decisions is crucial for a strategy to do more than just tick the “data” box. Airbnb created an internal “data university” with a curriculum tailored to their tool stack and business cases.

As a result, they saw weekly active users of internal data science tools rise from 30% to 45%, and 500 employees had taken at least one class. You don’t need to go to this extreme, but you do need to add training and key hires into your roadmap to ensure people know how to use data.

Process and methodology 

Plans are great, but requirements will change. Someone will need to track additional metrics. A new social media network will get popular, and suddenly you have a new source of data to factor in. 

Within the strategy you’ve created, set out internal processes for handling new requests and questions going forward. For example, when a team shops for a new tool, create a checklist of things the tech team needs, such as an open API or other way to support automated data export/import. 


The above steps, exercises, and questions will help you develop a data strategy that’s clearly linked to wider business goals and avoids the all-too-common, sporadic approach of most businesses.

To increase your leverage within the C-suite, involve multiple teams from across your business—more departments will see and advocate for your proposal.

If you need a partner to help define and implement your data strategy, get in touch

The post Pitching a Data Strategy? Here’s How to Ensure the C-Suite Says “Yes.” appeared first on CXL.

Strategic B2B Digital Marketing in a Pandemic: Mountain Climbing & Forrest Gump

In times of challenge, many people instinctually think short-term. Climbers feet from the summit of Everest, for instance, are not paragons of long-term, strategic thinking. Their sole focus is putting one foot in front of the other and getting enough …

In times of challenge, many people instinctually think short-term. Climbers feet from the summit of Everest, for instance, are not paragons of long-term, strategic thinking. Their sole focus is putting one foot in front of the other and getting enough oxygen to avoid collapse, as part of a fight or flight response.

Mid-pandemic, it is understandable that business decision makers would resemble the climber and pivot to survival mode by cutting overhead, deferring capital expenses, and maximizing shareholder value in the short-term. The juxtaposition of a strong stock market and double-digit unemployment would seem to reinforce this idea. COVID-19, however, is not a flashpoint event that has thrust business leaders into a short-term orientation. For digital marketers, the twin forces of COVID-19 and a prevailing “win now” approach have been a mixed bag. On one hand, CPG merchants benefit from tailwinds born of convenience and necessity, whereas major expenses are deferred, drifting from essentials into treats, postponables, or expendables. For merchants selling the latter, the first impulse is to cut back spend for brand building or new customer acquisition. After all, each ad dollar spent during the pandemic is not coming back through the door as a customer any time soon, and advertising dollars can be funneled elsewhere within the business to generate short-term “wins.”

Digital B2B marketers, however, would argue that approach is a mistake, especially in a time of crisis. Just because purchases are being deferred during an economic downturn does not mean they won’t eventually take place - the sales cycle is simply lengthening.

Since a B2B customer may be buying a cloud server solution and not toothpaste, the time between impression and sale can take months. There is painstaking research, comparisons of quotes and specs, buy-in from stakeholders, and adapting existing systems to accommodate an update. This explains why lead gen is king within B2B: business prospects are starting relatively far up-funnel and are rarely making spur of the moment purchase decisions.

Putting that idea in action, advertisers of all stripes can boost investment and steal share of voice in a soft marketplace now, with an eye on late 2021 or early 2022. In doing so, they can digitally establish cost-effective relationships with prospects and adapt nurture strategy to a longer time frame. This can pay dividends as economic recovery begins, to the tune of 4-5x growth in profit, market share, and market penetration.

Here are four ways to execute on that idea, taken from the B2B playbook:

1. Serve future-minded content that gets to the point

By now, most brands have changed with the times, replacing outdated copy and creative that worked during the end of 2019 and Q1 of 2020. However, acknowledging the climate while avoiding being trite or bland can be a tightrope walk.

For a positive example of striking this balance, think about how auto brands like GM are offering 0% financing for lengthened time frames or the option to defer monthly payments. Offering such incentives is a short-term risk, but if the barrier to purchase is lowered, sellers can widen the top of their funnels, convey flexibility and understanding, and build long-term trust and brand preference. This is especially true as a consideration set shrinks in a thinner ad market.

2. Mix up messaging for non-converting site visitors

Just because someone isn’t yet willing to buy or provide information doesn’t mean they don’t have needs that should be addressed. After all, they visited a website for a reason. Consider adapting creative and driving a secondary KPI that provides utility to your customer after their first visit, like a whitepaper or content download. If possible, factor these sub-KPI into your bidding strategy.

3. Keep marketing and sales attached at the hip 

Your digital marketing operation can act as a survey of what’s top of mind for your potential customers. As your sales team gets in touch with prospects that don’t qualify, they will begin to gather objections which can be incorporated into your marketing message to address points of hesitation proactively and empathize with customer sentiment.

4. Relax standards for disqualification 

It is not “no” forever – it is “no” right now based on circumstances that are beyond peoples’ control. Give people more time than usual and consider accounting for that ambiguity in your CRM platform.

To borrow from a Forrest Gump clip, digital advertising during an economic downturn has its similarities to shrimping during a hurricane. By capitalizing on an emptier marketplace, showing a meaningful message to people who are looking for one, and maintaining that message through the entire sales process, advertisers can position themselves to weather the storm of COVID-19 and reap the benefits years down the line.

Want to learn more? Check out our other B2B focused blogs here.

CCPA enforcement starts now and most companies aren’t ready

Unprepared for CCPA? Here is what you need to know and do — now.

The post CCPA enforcement starts now and most companies aren’t ready appeared first on Marketing Land.

The California Consumer Privacy Act (CCPA) went into effect on January 1, 2020 with a six-month enforcement grace period. That end date is now here.

The basics. As a refresher, CCPA explicitly applies to companies that qualify under one or more of the following statutory criteria:  

  • Have gross annual revenues in excess of $25 million;
  • Possess the personal information of 50,000 or more consumers, households, or devices; or
  • Earn more than half of their annual revenue from selling consumers’ personal information

A number of categories of businesses are explicitly exempted from CCPA compliance, including certain industries covered by federal regulations. However, most publishers will need to be ready to enable U.S. consumers to opt-out of third-party data transfers and demonstrate compliance to regulators in the event of an investigation or complaint.

Attorney Aaron Tantleff, a partner at law firm Foley & Lardner, offers a sliver of hope that CCPA may not apply to everyone, while cautioning that the law has few geographic boundaries. “We have spoken with many clients that have called in a panic to discover that CCPA does not apply. The applicability of the CCPA, like the GDPR, is not limited to only those organizations based in California. It may apply to organizations that lack any physical presence in the State.”

Broad application to businesses globally. As a practical matter the statute will broadly apply to most commercial enterprises, whether or not they explicitly target California residents.  For example, an early analysis of the legislation by the IAPP says:

Companies may pass [the personal information of 50,000 consumers] threshold more quickly than anticipated because the scope of personal information is broad. Most companies operate websites and inevitably capture IP addresses. Notably, companies need to comply regardless of whether the website targeted businesses or individual customers in California given that the term “consumer” is defined to mean any “resident.” Even individual bloggers and relatively small businesses outside California may find it difficult to ensure that they do not receive personal information of more than 50,000 California resident visitors to their website annually, simply from having it be passively accessible from there, and, within California, most retailers, fitness studios, music venues and other businesses will meet this threshold.

Risks of non-compliance. The California Attorney general can impose financial penalties up to $2,500 for non-willful violations and $7,500 for intentional violations. But these numbers can multiple quickly if thousands or millions of users are implicated. In most cases there will be no liability where the violation is “cured” within 30 days of receiving notice. There is also a private or individual right of action when personal information is wrongfully disclosed under CCPA. (The first CCPA class action lawsuit [.pdf] was filed in February against Hanna Andersson and Salesforce.)

According a recent Ethyca survey of 218 general counsels of technology companies, 56% said they were “unprepared for new privacy regulations coming in around the globe,” which includes CCPA. During the months leading up to the enforcement deadline, 43% of respondents said they had deprioritized privacy preparedness because of COVID-19. The survey also found that lack of resources or cost was the greatest challenge in complying.

What to do now. “For businesses still looking to button up on compliance, the essential — and only — first step is to figure out the personal data you possess and where it lives,” says Cillian Kieran, CEO of Ethyca. “After you’ve built a data map that has a thorough and complete record of the data you hold, and where it lives, you can worry about putting the structures in place to address various compliance tasks. But it all starts with the map.”

Attorney Tantleff adds, “Document everything. By now, organizations should have a robust set of security measures in place. However, under the CCPA, an organization must demonstrate that it has implemented reasonable security measures designed to protect personal information based upon the nature and sensitivity of that information.”

According to Lisa Rapp, VP of Data Ethics at LiveRamp, “No company should try to do this on their own. The best thing to do is to obtain as much information as possible by reading what industry leaders are saying, staying up-to-date on the materials that groups like the IAPP and IAB are putting out, and reaching out to prominent law firms that deal with data privacy to gain their legal counsel and interpretations of the law.”

Julie Rubash, VP of Legal at Nativo, recommends that publishers read the Attorney General’s final regulations “to ensure that current [privacy] plans are in line with the Attorney General’s interpretation.” She adds that “Tools like the IAB CCPA Framework are a step in the right direction to prepare for an inquiry and limit revenue disruptions. Publishers that leverage the IAB CCPA Compliance Framework tool and sign the limited service provider agreement are unlikely to experience a significant impact to their business models.”

Abby Matchett, Enterprise Analytics Lead at Bounteous, says, “Because CCPA takes a much broader view of personal data than Europe’s GDPR guidelines, most companies must undertake a significant internal inventory of any data that may be linked, directly or indirectly, with a consumer or household. Conducting such an inventory places a heavy burden on IT organizations, legal departments, and data analysts who may already be dedicated to other internal priorities. Overcoming this obstacle is one of the first steps towards compliance but is often the most challenging to coordinate and fully document.”

Matchett further explains, “If you are concerned that you may not have time to build a home-grown digital solution for this purpose, consider reaching out to third-party Cookie Consent Manager software companies that specialize in maintaining CCPA & GDPR ready solutions. Some common Consent Managers include TrustArc, OneTrust, and Quantcast, among others.”

Here comes CPRA. Even as many companies are struggling to comply with CCPA, a new November California ballot initiative could impose even tougher privacy rules if passed. According to the Future of Privacy Forum’s Katelyn Ringrose, “While companies may have begun, and in some cases, finalized strong compliance programs and efforts addressing the CCPA—the California Privacy Rights Act (CPRA), recently certified for the 2020 ballot, could have an enactment date as early as 2023, placing additional obligations on covered entities. The CPRA would create a sensitive data classification, place additional obligations on processors, and require the establishment of a California Privacy Protection Agency.”

Why we care. Large numbers of consumers have expressed concerns about how their data are being handled online. But there’s evidence that “privacy forward” companies are seeing both brand and financial benefits, in terms of greater consumer trust and even stronger revenue growth.

It’s foolish to delay taking the necessary steps to prioritize privacy and data security. As Tom O’Regan, CEO of Madison Logic put it, “Ultimately, complying with the CCPA controls will be far less expensive than penalties from non-compliance.”

Thursday’s Live with Search Engine Land will be a special CCPA and privacy discussion featuring Lisa Rapp, VP Data Ethics, LiveRamp, Abby Matchett, Enterprise Analytics Lead, Bounteous, Katelyn Ringrose, attorney, Future of Privacy Forum.

It starts at 1:00 p.m. EDT and will allow up to 100 people into the meeting to experience the discussion live and ask questions. If you’re a digital marketer you can’t afford to miss this. Sign up here.

The post CCPA enforcement starts now and most companies aren’t ready appeared first on Marketing Land.

Are You Listening? Crafting an Agile Response Strategy using Social Listening

During a global, local, or brand crisis, it’s important to understand how consumers are reacting to and behaving in an unpredictable landscape. One way to monitor this in real-time is by tracking online conversations through social listening. Social li…

During a global, local, or brand crisis, it’s important to understand how consumers are reacting to and behaving in an unpredictable landscape. One way to monitor this in real-time is by tracking online conversations through social listening. Social listening provides a unique way to look beyond obvious consumer needs by exploring shifts in topics, tone, and sentiment. This information is important for digital marketers to consider when crafting an approach across all media channels, especially when formulating a crisis response strategy.

The Dirty Secret to Ranking #1 in Google (Part 3 of 3)

Years ago, one might reasonably separate the elements of Google’s results into distinct entities: Google News, Books, Videos, Images, Local… But today it’s near-impossible. The list of elements Google might show for a given query are …

Years ago, one might reasonably separate the elements of Google’s results into distinct entities: Google News, Books, Videos, Images, Local… But today it’s near-impossible. The list of elements Google might show for a given query are so vast and varied that at the macro-level there’s really only three kinds of search results that matter: 1) Google-owned properties and answers (where…

What to look at when considering a Digital Asset Management platform

What do all DAMs do, and what are the points of differentiation between them?

The post What to look at when considering a Digital Asset Management platform appeared first on Marketing Land.

When you’re making a decision about a Digital Asset Management partner, consider the following eight areas.

File formats and handling

One area of differentiation involves the varying abilities to manage a variety of file formats. Though most players say they support the most popular video, image and audio formats, if your workflow requires the use of a specialized format you will want to ensure the vendors you’re considering can fully support that format.

User permissions management

The content production supply chain can be long and complicated, involving many departments, agencies, freelancers and more. The ability to provide flexible permissions so that the right people have access to the right assets –– and only the right assets –– can be very valuable.

Search and metadata

A DAM provider’s capabilities with regard to metadata and search are key to one of the most important benefits of a digital asset management system –– the ability to find assets after they’ve been created and filed away. Most providers now use artificial intelligence, either proprietary or through a partnership, for image and video recognition and tagging.

Workflow management

DAM systems differ in the extent of their workflow management capabilities. Some allow collaboration through @ tagging, while others have more full-fledged project management offerings. This functionality can help marketing teams, along with outside creative resources, communicate about changes while an asset is in the development phase or being updated.

Later in the process, they can allow for approvals to be obtained from brand managers, execs and the legal team, while some systems also facilitate asset distribution. These capabilities may be built into the core platform or be offered as an add-on or integration.

Reports and analytics

Analytics capabilities are what allow marketing leaders to trace the return on the investment made in the development of digital media.


Most DAMs are offered as SaaS and can be accessed from modern browsers on a variety of platforms, but some have developed native apps for mobile or other platforms.

Data storage and security

The majority of DAM providers have partnered with Amazon Web Services or Google to host their software and their clients’ assets, and so depend on their partners’ geographical distribution, regular backups and adherence to security protocols. However, some players offer clients a variety of options for data hosting, something that’s likely to be appreciated by enterprises that operate in markets with strict data governance regulations.


Since a DAM system is meant to be the central “single source of truth” repository for all of a brand’s assets, a key factor for a successful deployment will be whether or not it integrates well with the other tools in your martech stack.

Vendors differ greatly in terms of the number and types of integrations they offer. Some are also beginning to specialize in serving a specific sector with unique integration needs, such as online retailers.

Learn more about Digital Asset Management platforms and get guidance on how to make a decision. Download our Martech Intelligence Report now!

The post What to look at when considering a Digital Asset Management platform appeared first on Marketing Land.

It’s like having 10 different remote controls for 10 different TVs

This NPR interview with Danielle Ofri, author of a new book on medical errors (and their prevention), had some interesting insight into how human factors play out during a pandemic. Her new book is “When We Do Harm,” and I was most interested in these excerpts from the interview: “…we got many donated ventilators. Many … Continue reading It’s like having 10 different remote controls for 10 different TVs

This NPR interview with Danielle Ofri, author of a new book on medical errors (and their prevention), had some interesting insight into how human factors play out during a pandemic.

Her new book is “When We Do Harm,” and I was most interested in these excerpts from the interview:

“…we got many donated ventilators. Many hospitals got that, and we needed them. … But it’s like having 10 different remote controls for 10 different TVs. It takes some time to figure that out. And we definitely saw things go wrong as people struggled to figure out how this remote control works from that one.”

“We had many patients being transferred from overloaded hospitals. And when patients come in a batch of 10 or 20, 30, 40, it is really a setup for things going wrong. So you have to be extremely careful in keeping the patients distinguished. We have to have a system set up to accept the transfers … [and] take the time to carefully sort patients out, especially if every patient comes with the same diagnosis, it is easy to mix patients up.”

And my favorite, even though it isn’t necessarily COVID-19 related:

“For example, … [with] a patient with diabetes … it won’t let me just put “diabetes.” It has to pick out one of the 50 possible variations of on- or off- insulin — with kidney problems, with neurologic problems and to what degree, in what stage — which are important, but I know that it’s there for billing. And each time I’m about to write about it, these 25 different things pop up and I have to address them right now. But of course, I’m not thinking about the billing diagnosis. I want to think about the diabetes. But this gets in the way of my train of thought. And it distracts me. And so I lose what I’m doing if I have to attend to these many things. And that’s really kind of the theme of medical records in the electronic form is that they’re made to be simple for billing and they’re not as logical, or they don’t think in the same logical way that clinicians do.”

Experimentation ROI: How To Think Beyond Uplift

Ian Freed was devastated. He was at the helm of a product experiment that cost his company $170 million dollars. He had built a team of 1000 employees and launched one of the most anticipated products in 2014. But this product was termed as a ‘fiasco’ and a ‘debacle’ by the media and was shelved…

Ian Freed was devastated. He was at the helm of a product experiment that cost his company $170 million dollars. He had built a team of 1000 employees and launched one of the most anticipated products in 2014. But this product was termed as a ‘fiasco’ and a ‘debacle’ by the media and was shelved within 13 months.

The CEO of Ian’s company called him and said, “You can’t, for one minute, feel bad about the Fire phone. Promise me you won’t lose a minute of sleep.”[1] 

The CEO here is Jeff Bezos, and the failed product was the Fire Phone.

Before I narrate the consequences of this expensive, failed experiment, let’s take a pause and understand what ROI means. 

What is ROI?

Simply put, ROI or Return on investment tells you whether you’re getting your money’s worth from your marketing initiatives. 

When the ROI goes out of control, it should set alarm bells ringing. 

Therefore, coming back to our story, I would be losing sleep over the Fire phone if I were Jeff Bezos. Why wasn’t he? The answer, and I quote him, came in a 2018 letter to Amazon’s shareholders:  

“As a company grows, everything needs to scale, including the size of your failed experiments. If the size of your failures isn’t growing, you’re not going to be inventing at a size that can actually move the needle.”

Amazon’s CEO Jeff Bezos announcing Fire smartphone
Amazon’s CEO Jeff Bezos announcing Fire smartphone Amazon conference
Image Source[1]

Great story, right?

Unfortunately though, if your manager isn’t Bezos, and your company doesn’t have deep pockets like Amazon, you’re most likely in trouble. 

Your manager might ask you questions like “Did the variation win?” or “Did this experiment move the needle enough to put it in my town hall presentation?” If the answer to these questions is NO, the experimentation exercise is termed futile or having no ROI. 

Why so, you ask? This is because there is so much literature around us which declares uplift (in conversion rates or revenues) as the only metric to measure the success of experimentation. Everyone expects an uplift from experiment, and then terms the uplift as either ‘present’ or ‘absent’ – there is no in-between. 

This approach reduces experimentation to a point in a checklist which companies want to tick off. This is because they see experimentation as the goose, which will suddenly lay golden eggs (read: uplift). 

The truth is that though the industry has conditioned us to think this way, this should not be the reasoning behind any experiment.  

Why do we experiment?

Essentially, experiments help in achieving the following objectives: 

1. Making better decisions

Read that again. It’s not ‘correct’ or ‘right’; it’s making ‘better’ decisions. Experimentation aids data-driven decision making, which is better than based on gut or instinct. Without an experiment, there is no way to objectively denounce an idea.

2. Reducing ambiguities and risk of losing business

It’s always better to test your hypotheses than doing a full rollout of your new website or product features. This, in turn, minimizes the risk of impacting all your current business metrics – whether it’s the conversion rate on your website or feature adoption rates.

3. Prioritizing and learning what works and what does not

Building an experimentation roadmap helps you prioritize and differentiate between the must-haves and good-to-haves. Some experiments may not give you an uplift but will provide you tonnes of learnings for your next one.

cartoon illustration on Hippo- Highest Paid Person's Opinion
An illustration on Hippo: Highest Paid Person’s Opinion
Image Source[2]

The problem arises when ‘HiPPOs’ decide which experiment to run first, and their primary expectation is a revenue uplift. The fundamental purpose of experimentation to make better decisions goes for a toss. 

Is expecting revenue uplift so wrong?

Imagine a situation where you, as a CRO practitioner, have been running an experimentation program for a quarter or 6 months with multiple experiments across different pages of your website. Most of these tests have yielded statistically significant winners, and you are elated. 

However, when you look at the overall conversion rate of your website, it is almost the same as a quarter before. What do you tell your manager? How do you justify the investments you made in the CRO product which you championed to buy? Does it mean your optimization initiatives are worthless? (Story of your life? Shoot me an email or tag me on Twitter if I nailed it.)

Imagine the same situation–with a quarter spent in testing and continuous optimization–but this time your overall revenue has shot up by 3%. You are elated as this is your moment. But as soon as you enter your CEO’s cabin/zoom room, you see the head of SEO, head of performance marketing, and others already showing how they are responsible for this uplift. 

It makes you wonder and doubt how you can isolate the impact of the CRO experiments from other variables and proclaim this as a victory for you and your team.

(Sidenote: we organized a webinar to try and unknot the mystery behind attribution challenges with experimentation.)

It is hard to accurately calculate the ROI of your experimentation program because of issues in forecasting, dealing with multiple variables, and running multiple experiments. CRO practitioners deal in averages, and what might seem an impact ‘wave’ for a specific segment of users might just be a ripple in the ocean when you look at global goals.

And you’re not alone. Almost 53% of CRO professionals[2] cannot calculate ROI of experimentation.

pie chart of CRO professionals not able to calculate roi of experimentation
A pie chart showing percentages of the CRO professionals calculating ROI of experimentation. Image Source[3]

Instead of putting resources behind chasing that one ROI number, I would advise using those resources to increase the testing velocity. This is because the ROI on experimentation is amplified by the customer insights you gain with each experiment you perform. That’s what Bezos did with Ian Freed. 

The voice recognition software of the Fire phone could follow commands and fetch information from the cloud. This feature was talked about and loved by the customers. It made Bezos curious, and he put Freed on a project to build a team and technology to respond to voice commands. Four months later, Echo was born. Jeff Bezos looked back at the incident and remarked: 

“While the Fire phone was a failure, we were able to take our learnings (as well as the developers) and accelerate our efforts building Echo and Alexa.”

So what is the impact of experimentation?

illustration highlighting the impact of experimentation on roi
Impact of experimentation

1. Innovation/Breakthrough

Think about the players who have disrupted industries – Netflix, Amazon, Booking, etc. The things that they have in common are a company-wide culture of experimentation and unmatched testing velocity that allows them to fail fast and move on to the next innovation. 

“If you have to kiss a lot of frogs to find a prince, find more frogs and kiss them faster and faster.” ~Mike Moran

2. Reducing the risk of new launches

Most people are hesitant to test out bold customer experience (CX) changes because of the effort they put into making these changes. Because what if the change leads to loss of business! Experimentation is the tool that helps mitigate the risk of wasted resources—time and money by helping take ship/no-ship decisions—at the same time inspiring people to not be daunted by testing bold changes.

3. North star of all experimentation should be customer experience

Ultimately, every experiment is taking you closer to the CX best suited for your end customer. If there is a winner, you know what your customer likes, and if the experiment has no winners, it means the customers like the status quo.

So, your mission should be to provide the best CX because it will ultimately impact your bottom line. Focus on your long term vision; a few banners and pop-ups might give you an uplift in the short run, but it might hamper the CX and hit your revenues in the long term.


To sum up, we need to reimagine experimentation ROI beyond just the revenue impact. Don’t ignore the money it makes for you, but don’t make that a priority. The compass of your experimentation efforts should move from ‘experimentation for better revenue/conversions’ to ‘experimentation for better decisions.’ 

Set up your CRO teams as the learning hubs for your business. The main goal for these teams should be to provide customer intelligence to anyone who asks for it. Aiming for higher velocity will move the key metrics for you faster and deliver growth; endless analyses of the atomic impact of each experiment you run won’t. 

PS: Watch this webinar if you’re interested in further understanding the value of experiments that fail in getting revenue uplifts.