PureCars launches an attribution platform just for car dealerships

The auto dealer-focused online marketing platform says this new solution is the first to solve dealers’ unique problem of attribution.

The post PureCars launches an attribution platform just for car dealerships appeared first on Marketing Land.

Buying a car these days often involves extensive online research before visiting a physical dealership.

That’s a key reason why attribution has been difficult for auto dealers, says car marketing platform PureCars, which released a sales attribution/business intelligence platform this week that it says is the first one to fully track the path from online research to dealership visit.

Called Signal Pro, it is complementary to PureCars’ existing Smart Advertising marketing automation platform for dealerships, which enables multi-channel campaigns. CEO Sam Mylrea said clients can use Signal Pro in conjunction with Smart Advertising, or separately.

The challenge. Mylrea said that car purchases present a unique attribution problem, because buyers typically spend so much of the pre-purchase time conducting research online, with a physical trip only in the last stage of test driving, final negotiation and purchase. In other words, the funnel dynamic is particular to this industry.

A solution. Signal Pro gets a feed from the inventory management system of its clients, about 3,000 dealerships in the US. It tracks user activity at dealer websites, engagement activity with ads run by its client dealerships, and, via third-party data services, engagement and impression data on car-related ads that are run by others, like auto makers.

When a user goes to a dealer web site or views one of the ads, PureCars is able to match the user to a cross-device graph, which employs IP addresses and other indicators to match anonymous users to persistent identifiers, like phone numbers. Those identifiers then help match the customer who walks into a physical showroom and buys or leases a car to an ad engagement or a site visit.

Previously, Mylrea said, his company could track ad campaign engagement, as well as specific attribution signals like a user filling out a contact form or a user calling a specific phone number. Now, he said, Signal Pro enables dealers to link the online research to the physical purchase, and also predict where each customer is in the buying cycle. Mylrea claims Signal Pro can accurately tie about 80 percent of dealership purchases to the relevant marketing or advertising impetus.

Why you should care. While other attribution services similarly offer multi-point attribution, including linkages between online marketing and offline purchases, Mylrea said Signal Pro is the first specifically tied into dealer inventories and designed for auto purchases.

Given that cars are generally consumers’ second-biggest purchase after homes, knowing which ad spend or site content impacted sales can be a valuable piece of the marketing puzzle.

This story first appeared on MarTech Today. For more on marketing technology, click here.

The post PureCars launches an attribution platform just for car dealerships appeared first on Marketing Land.

Court Ruling Strengthens Brands’ Mission Against Unauthorized Online Sellers

As a brand selling in an online marketplace, you may find yourself facing unauthorized sellers on the marketplace who are undercutting your price,… > Read More
The post Court Ruling Strengthens Brands’ Mission Against Unauthorized Online Sellers appe…

As a brand selling in an online marketplace, you may find yourself facing unauthorized sellers on the marketplace who are undercutting your price,... > Read More

The post Court Ruling Strengthens Brands’ Mission Against Unauthorized Online Sellers appeared first on Retail Performance Marketing Blog - CPC Strategy.

The Requirement for an Omnichannel Digital Experience for Brands

This is a guest post by Ryne Misso, Director of Marketing, Numerator. Before the omnichannel age, delivering a consistent, predictable experience to consumers… > Read More
The post The Requirement for an Omnichannel Digital Experience for Brands appe…

This is a guest post by Ryne Misso, Director of Marketing, Numerator. Before the omnichannel age, delivering a consistent, predictable experience to consumers... > Read More

The post The Requirement for an Omnichannel Digital Experience for Brands appeared first on Retail Performance Marketing Blog - CPC Strategy.

Unlock multi-touch attribution with CRM campaign tracking

If you’re using a CRM to house leads then you likely have the campaign tracking tools for middle funnel activity even if you have an offline transaction point.

The post Unlock multi-touch attribution with CRM campaign tracking appeared first on Marketing Land.

display chart graphic
Image credit: www.Curata.com

Brands with an offline transaction point often struggle to measure the full customer journey from acquisition source through to revenue. Often times, if revenue can be attributed back to something, it’s to the marketing channel responsible for the lead. This measurement is usually implemented with either a first or last touch attribution model for channel attribution and completely leaves out the rest of the customer journey.

The missing visibility and measurement is post lead acquisition. Once a lead enters our systems, how do we measure the effectiveness of our campaigns and lead treatments? Even better, how do we attribute revenue to content pieces and treatments applied to that lead in-funnel?

The answer may be something we already have access to. If you’re using a CRM to house leads then you likely have the tools to track middle of funnel activity at your fingertips with Campaign Tracking.

What is campaign tracking?

Campaign tracking within a CRM is technically an entity or object that tracks a variety of information about an event, mailing, emailing, or other marketing initiatives. It’s basically a container that houses all of the components of a campaign across channels and treatments.
Leads and contacts can be members of one or more CRM campaigns allowing visibility into the effectiveness of both single and multiple campaign influence across all levels of our funnel from lead to cash.

campaign graphic
Image credit: www.Salesforce.com

What are the benefits of CRM campaign tracking?

Depending on the CRM and how it has been architected and implemented, campaign tracking provides a significantly deeper level of insights and measurement including the ability to

  • Tie marketing activities to our sales pipeline
  • Compare the effectiveness of different marketing initiatives and their influence on each other
  • Measure mid-funnel activities alongside marketing channel attribution
  • Measure the effectiveness of content post lead acquisition
  • Inform the sales team of historical marketing activities via the contact record
  • Roll-up similar lead sources into a single object
  • Connect online & offline activities
  • Enable holistic ROI reporting
  • Preserve data integrity & maintain hygiene
  • Enables multi-touch attribution modeling within your funnel

Attribution and CRM campaign tracking

Now that measurement is enabled at such a granular level in-funnel we can see marketing activity influence across the funnel from lead to customer. This is where attribution really gets complex! Similar to the first-touch, last-touch, multi-touch debates on marketing channel attribution. Now we have these same debates in-funnel when attributing back to campaigns within our CRM.

Do we give credit to the campaign that initially acquired the lead?

Do we give credit to the campaign the lead responded to before they converted to an opportunity?

Do we give credit to the campaign that influenced the lead right before they converted to a customer?

First-touch attribution model

First-touch is pretty self-explanatory. In this attribution model, all credit is given to the very first action taken by the user that created the lead record.

attribute graphic
Images credit: www.Curata.com

The first-touch attribution has its advantages, it’s super easy to implement. The lead is tagged using a custom field and that field rides on the record all the way through the funnel to closed won. However, this model leaves so much of the story out neglecting to consider all other interactions the user had or actions the user took beyond that initial entry into the database.

Last-touch attribution model

Last-touch attribution is the opposite of first-touch. Instead of giving all credit to the first action the user took, we’re giving it to the last step the user took. Also easy to implement by merely over-riding that custom field.

display graphic

Last-touch is fantastic for measuring the effectiveness of campaigns targeting the bottom of the funnel geared directly toward driving a purchase decision. However, if we only look at what ultimately turned into a sale, we really lack insights into what levers to pull to get them to that point and we are leaving a ton of opportunity on the table. We are also boxing ourselves into diminishing returns and expensive tactics and are unable to scale our marketing programs.

Multi-touch attribution model

Multi-touch models are more complex, recording all interactions and giving credit to all touch points in the journey. They provide the clearest picture of attribution and provide the most insights regarding what levers to pull across the funnel to improve velocity and efficiency of our marketing investments. To implement a multi-touch attribution model within your funnel you have to utilize CRM campaign tracking. This is the biggest benefit of the campaign tracking tools within your CRM.

CRM campaign tracking reports

Once campaign tracking is properly set up and working within your CRM and when used in conjunction with a clean and granular lead source strategy, CRM campaign tracking opens up rich and robust reporting options. Your data will tell a very different story!

Here are a few sample reports that can be generated when both campaign tracking and a clean and granular lead source strategy are applied within Salesforce.

display graphic chart

I’d love to hear how you are utilizing campaign tracking within your CRM and what kind of new insights you’ve been able to pull. My guess is that once you were able to get to this level of insights, marketing resources were moved around to concentrate on what you didn’t even know was working within your marketing program.

The post Unlock multi-touch attribution with CRM campaign tracking appeared first on Marketing Land.

What Happens When Data Meets Creative (and How to Make it a Reality at Your Company)

There are quite a few people out there that just don’t *get* creative. They don’t understand the way in which we work or make decisions. And, indeed, creative teams are known to be cost centers rather than revenue generators. To certain execs, creatives are simply the sneaker-wearing hipsters who are brought in to make things […]

The post What Happens When Data Meets Creative (and How to Make it a Reality at Your Company) appeared first on Brooks Bell.

There are quite a few people out there that just don’t *get* creative. They don’t understand the way in which we work or make decisions. And, indeed, creative teams are known to be cost centers rather than revenue generators. To certain execs, creatives are simply the sneaker-wearing hipsters who are brought in to make things look pretty or sound good.

While this is a far cry from reality, it’s also not that hard to understand why. As creatives, we understand the value of good creative work. Proving that value, however, can be difficult. So here are a few tips for proving the ROI of your creative team and incorporating data within your creative process.

Tip #1: Know and Speak the Language of Business.

Smart creative work requires an objective-based approach. Objective-based creative is driven by data—often in the form of user feedback, website analytics, and strategic business goals. As a designer or copywriter, your job is to gather and digest this data and apply it to your work.  

When pitching your concepts to your stakeholders, most aren’t going to accept work that just “looks” better. It’s important that you are able to articulate the business problem, your target audience and the objective-based reasoning behind your decisions. This ensures that your work is influenced by hard data and research, rather than just design preferences.

On the other side, it’s important to train your stakeholders in the art of objective-based feedback. That is, feedback in the context of whether or not your work is effective in addressing the objective at hand. Doing this takes time, practice and a lot of patience, but the payoff is huge. Your executives will feel more confident after seeing that your creative team is aligned and hyper-focused on providing measurable value.

Tip #2: Use Testing to Eradicate B.S. in the Creative Process.

Brooks Bell was founded on the idea that you can eliminate creative guesswork by applying the scientific method. But at many companies, creative and UX teams rarely engage with testing teams. While this might make sense from the perspective of your org chart, few realize just how much collaboration between these functions could positively impact a business.

A few years back, my team and I were brought in to work with one of our retail clients. Looking at their website data, our analysts realized that a large majority of people were abandoning the express checkout form for the full checkout form. This seemed counterintuitive to us: less friction is always better, right? Why would anyone prefer to fill out the long form!?

In order to develop a strategy to test, we needed more data—so we turned to user research. We polled a select group of users about their purchasing experience and uncovered some potential reasons for their behavior.

We discovered that many users preferred to use alternative or saved payment methods, yet the account login and gift card payment options were only available in the full checkout experience. We ran a test adding these options to the express checkout flow, which resulted in a 5% lift. When implemented, this test translated to a $5M increase in revenue.

The impact of this was significant—and not just from a revenue perspective. Through this process, we were able to identify other areas where users could be experiencing anxiety. It also prevented us from over-designing in the future. For this company’s customers, a simple and clear message and a less cluttered experience were enough to quell their anxiety.

For data-starved creatives, these types of insights can be extremely valuable and can greatly influence the company’s overall design aesthetic.

Tip #3: Be Sure You Recruit Relevant User Groups for Discovery Research

This tip is for you if—upon presenting the results of your user research—you’ve ever been asked “why did you talk to [audience group]?” or the alternative: ”why didn’t you talk to [audience group]?”

Sure, conducting guerilla research on random mall-goers or your coworkers at lunchtime will get you basic usability feedback. But if you want actionable insights, you need to not only research the group that’s generating the most business for your company, but also the group that’s most impacted by the problem you’re trying to solve.

If return users drive the majority of your revenue, don’t research new users. Similarly, don’t ask someone to look at your mobile design if they don’t fit the demographics of the segment you’re trying to reach.

Here at Brooks Bell, we believe it’s important for our clients to be closely involved in the process of selecting user segments for research. This not only manages the scope of the project and ensures maximum impact, but it also helps to avoid the frustrating line of questioning I mentioned above.

Tip #4: Embrace Survey-Based Research

If you’re well-versed in usability testing, you know that elaborate usability tests are a waste of resources and you really can get the best results from testing no more than 5 users.  But to an executive, that number 5 can seem awfully small. And no matter how many times you reference or point them to this blog post, they still might just not buy it.

This is where survey-based research comes in. We’ve had tremendous success in conducting survey-based research for our clients, and find it is often better received by executives.

Executives respond well to survey research for a couple of reasons: You can survey a larger population of people. It’s fast—most of the time we get responses back within a day or two. And finally, depending on the types of questions you ask, it’s largely quantifiable.

While surveys are different from usability tests, oftentimes, you can use survey results to back up your usability test results.

Finally, it’s important that you also become the master of your research domains and empower yourself to dig in on your own.  For this, pivot tables are a great tool. Pivot tables unlock the magic of Excel by allowing us to take all of our survey results and slice and dice them any way we want… filtering answers by segments, averaging, counting, and creating data visualizations all without ever having to talk to an analyst.

How many of you thought you’d leave this post adding Excel to your list of preferred programs? 😉

Tip #5: Don’t Hoard Your Ideas – Bring Others Into the Creative Process

It’s every designer’s tale of despair: you spend tons of time on a project—putting in extra hours to make sure every pixel has been pushed into the perfect position, every line kerned and leaded—only to have your work completely shat on upon unveiling it.

Trust me on this one: hoarding your ideas and excluding other from your design process really only sets you up for disappointment, depression and frustration.  

So stop with the big reveal and instead invite others into the design process. Voice your ideas in a collaborative way. Position yourself as a guide within a creative process in which the objective is to build something collaboratively. Without a doubt, you’ll find you’ll get things approved faster and more frequently.

 


Interested in learning how Brooks Bell can help empower your creative and UX teams with data? Learn more about our services or contact us today.

The post What Happens When Data Meets Creative (and How to Make it a Reality at Your Company) appeared first on Brooks Bell.

Use predictive personalization to drive increased conversion rates

Every day, every month, every quarter, marketers are tasked with a conundrum: create web sites and messages that resonate with target audiences. It’s not a rare request. In fact, it’s a fundamental principle of marketing. Why is it a conundrum? Because you’re being asked to make one size fit all of your visitors. Think about […]

The post Use predictive personalization to drive increased conversion rates appeared first on Marketing Land.

Every day, every month, every quarter, marketers are tasked with a conundrum: create web sites and messages that resonate with target audiences. It’s not a rare request. In fact, it’s a fundamental principle of marketing. Why is it a conundrum? Because you’re being asked to make one size fit all of your visitors. Think about your site. Who are the different segments of visitors? What are their different needs and motivations when visiting your site?

Predictive personalization systems use machine learning to automatically choose and deliver the experiences most likely to drive each site visitor to convert.

This white paper from Intellimize covers:

  • How predictive personalization works.
  • The advantages of predictive personalization vs. A/B testing.
  • How predictive personalization works in practice.

Visit Digital Marketing Depot to download “An Introduction to Predictive Personalization.”

The post Use predictive personalization to drive increased conversion rates appeared first on Marketing Land.

Pinterest Holiday Advertising [2018 Guide]

In the event you are still trying to figure out which products shoppers are craving this holiday season (and where you should be… > Read More
The post Pinterest Holiday Advertising [2018 Guide] appeared first on Retail Performance Marketing Blog – CP…

In the event you are still trying to figure out which products shoppers are craving this holiday season (and where you should be... > Read More

The post Pinterest Holiday Advertising [2018 Guide] appeared first on Retail Performance Marketing Blog - CPC Strategy.

B2C Tips For Turning One-Time Holiday Shoppers Into Lifetime Customers

The holidays are a bad time for a lot of things. From traveling to saving money to making sure you don’t eat your weight in gingerbread, it’s hard to enter the new year unscathed. Even in the marketing world, the holidays are probably the most stressfu…

b2c-holiday-tips-introduction

The holidays are a bad time for a lot of things. From traveling to saving money to making sure you don’t eat your weight in gingerbread, it’s hard to enter the new year unscathed. Even in the marketing world, the holidays are probably the most stressful time of the year. You’re trying to ship multiple ecommerce campaigns, aggregate all of your data for year-end reports, and figure out who your Secret Santa in the office might be. But there are plenty of bright spots that stand out in the midst of the chaos. In particular, the holidays are the perfect...

The post B2C Tips For Turning One-Time Holiday Shoppers Into Lifetime Customers appeared first on The Daily Egg.

Hold-Out Groups: Gold Standard for Testing—or False Idol?

You run an A/B test on the call-to-action for a pop-up. You have a process, implement it correctly, find a statistically significant winner, and roll out the winning copy sitewide. Your test answered every question except one: Is the winning version better than no pop-up at all? A hold-out group can deliver the answer, but, […]

The post Hold-Out Groups: Gold Standard for Testing—or False Idol? appeared first on CXL.

You run an A/B test on the call-to-action for a pop-up. You have a process, implement it correctly, find a statistically significant winner, and roll out the winning copy sitewide.

Your test answered every question except one: Is the winning version better than no pop-up at all?

A hold-out group can deliver the answer, but, like everything, it comes at a cost.

What are hold-out groups?

A hold-out group is a form of cross-validation that extracts, or “holds out,” one set of users from testing. You can run holdouts for A/B tests and other marketing efforts, like drip email campaigns in which a percentage of users receives no email at all.

After completion of a test and implementation of the winning version, the hold-out group remains for weeks, months, or, in rare cases, years. In doing so, the holdout attempts to quantify “lift”—the increase in revenue compared to doing nothing.

For example, a “10% off” coupon (delivered through a pop-up or email campaign) may generate 15% more sales than a current “$10 off a $100 purchase” coupon. However, without a holdout, you don’t know how many consumers would’ve bought without any coupon at all—a winning test may still reduce profits.

Most often, however, holdouts are used not to measure lift from a single test but lift for an entire experimentation program. Because holdouts require siphoning off a statistically relevant portion of an audience, they make sense only for sites with massive amounts of traffic.

The difference between a hold-out and a control group

Imagine you want to test a headline on a product page. The version on the left (Control) is the current version, while the experimental version (Variation A) is on the right:

ab test apple cables

Assume, by some miracle, that Variation A performs better, and you implement it for all visitors. That’s the standard process for an A/B split test—50% see each version during the test, and 100% see the winning version after the test completes.

However, if you continue to show some visitors the control version, that control group becomes the holdout. In other tests, the control may not “transition” from control to holdout. Instead, it can be a separate segment omitted from the start—like the email campaign in which a percentage of subscribers receive nothing.

Because a holdout can estimate the value of a marketing effort beyond a relative improvement between two versions, some consider it “the gold standard” in testing.

Why hold-out groups are “the gold standard”

For many, holdouts are a gold standard for testing because they measure the value not just of a test but of a testing program.

And while the value of testing may be apparent to those involved in it, individual test results do not aggregate into ROI calculations made in the C-Suite. There, the considerations extend beyond website KPIs:

  • Does it make sense to employ a team of data scientists or email marketers?
  • If we fired the entire team tomorrow, what would happen?

Holdouts also have the potential to assess experimentation’s impact on customer lifetime value. While a short-term split test may record an increase in clicks, form fills, or sales, it doesn’t capture the long-term effects:

  • Do pop-ups and sticky bars increase email leads but, over time, reduce return visitors?
  • Does a coupon program ultimately decrease purchases of non-discounted items?

Some effects may take months or years to materialize, accumulating confounding factors daily. Thus, when it comes to measuring the long-term impact of tests, how long is long enough?

Defining the scope for hold-out groups

How long should you maintain a hold-out group? Without a defined window, you could make ludicrous comparisons, like decades-long holdouts to measure your current site against its hand-coded version from the late 1990s.

The decisions in the extreme are laughable, but as the gap narrows—five years, three years, one year, six months—they get harder.

Look-back windows and baselines for holdouts

How much time should pass before you update the “baseline” version of your site for a hold-out group? “It depends on your goals,” CXL Founder Peep Laja explained. “You could leave it unchanged for three years, but if you want to measure the annual ROI, then you’d do yearly cycles.

What about the degree of site change? “When it’s functionality, there’s a sense of permanence,” Cory Underwood, a Senior Programmer Analyst at L.L. Bean, told me. “When it’s messaging, you get into how effective and for how long it will be effective.

Underwood continued:

There are times when you would want to get a longer read. You can see this in personalization. You target some segment with a completely different experience on the range of “never” to “always.” Say it won and you flip it to always. Six months later, is it still driving the return?

A hold-out group offers an answer. (So, too, Laja noted, could re-running your A/B test.) But you wouldn’t get an apples-to-apples comparison unless you accounted for seasonality between the two time periods.

In that way, a hold-out group is uniquely rewarding and challenging: It may mitigate seasonality in a completed A/B test but reintroduce it when comparing the hold-out group to the winner.

Omnichannel retailers like L.L. Bean manage further complexity: Demonstrating that website changes have a long-term positive impact on on-site behavior and offline activity. The added variables can extend the timeline for holdouts. Underwood has run hold-out groups for as long as two years (an anomaly, he conceded).

For test types and timelines that merit a hold-out group, implementation has its own considerations.

Implementing hold-out groups for tests

The implementation of holdouts isn’t formulaic. Superficially, it involves dividing your audience into one additional segment. (Hold-out segments often range from 1 to 10% of the total audience.) For example:

Control: Audience 1 (47.5%)

Variation A: Audience 2 (47.5%)

Hold-out: Audience 3 (5%)

Many A/B testing tools allow users to adjust weights to serve (or not serve) versions of a test to an audience. But not every test can take advantage of segmentation via testing platforms.

As Underwood explained, a decision to roll out tests on the client side (using a testing tool) versus server side (through a CDN) hinges on two considerations:

  1. The scale of change. Large-scale DOM manipulations deployed via client-side rollouts risk a slow and glitchy user experience. The greater the difference between versions of the site involved in a test (like a hold-out that preserves an entirely different homepage design), the more that server-side delivery makes sense.
  2. The specificity of targeting. Testing tools connect user data with CRM data for more granular targeting; server-side segmentation is limited to broader attributes of anonymous users, such as location and device type, making it difficult to test changes for a narrowly targeted audience.

At a certain scale—say, for Pinterest’s quarter-billion monthly users—building a custom platform can expedite testing and integrate more effectively with in-house tools.

pinterest testing platform model
Pinterest built its own A/B testing platform to support more than 1,000 simultaneous tests. (Image source)

Perhaps most importantly, profitable implementation depends on knowing when a hold-out group improves a website—and when it’s a costly veneer to hide mistrust in the testing process.

When holdouts work

1. For large-scale changes

To the site. The more expensive a change will be to implement, the greater the justification to use a hold-out group before implementation.

After-the-fact holdouts for a non-reversible change make little sense. But advance testing to validate the long-term effect does. “As the risk goes up, the likelihood [of a holdout] also goes up,” summarized Underwood.

Often, Underwood said, marketing teams request holdouts to validate proposals for extensive site changes. A holdout that confirms the long-term value of their plans is persuasive to those who sign-off on the investment.

To team priorities. John Egan, the Head of Growth Traffic Engineering at Pinterest, agrees with Underwood—a test that implicates larger changes deserves greater (or, at least, longer) scrutiny, which a holdout delivers.

But site development costs aren’t the only costs to consider. As Egan explained, holdouts also make sense when “there is an experiment that was a massive win and, as a result, will potentially cause a shift in the team’s strategy to really double down on that area.”

In those circumstances, according to Egan, a holdout typically lasts three to six months. That length is “enough time for us to be confident that this new strategy or tactic does indeed drive long-term results and doesn’t drive a short-term spike but long-term is net-negative.”

2. To measure the untrackable

Egan acknowledged that, while holdouts are standard at Pinterest, “we only run holdout tests for a small percentage of experiments.”

For Pinterest, the primary use case is to:

measure the impact of something that is difficult to fully measure just through tracking. For instance, we will run periodic holdouts where we turn off emails/notifications to a small number of users for a week or a month to see how much engagement emails/notifications drive and their impact on users’ long-term retention.

Egan detailed such an instance on Medium. His team wanted to test the impact of adding a badge number to push notifications. Their initial A/B test revealed that a badge number increased daily active users by 7% and boosted key engagement metrics.

pinterest badge numbers
Badge numbers drove a near-term lift, but would that lift endure? Egan’s team used a hold-out group to find out. (Image source)

Still, Egan wondered, “Is badging effective long-term, or does user fatigue eventually set in and make users immune to it?” To find out, Pinterest created a 1% hold-out group while rolling out the change to the other 99% of users.

The result? The initial 7% lift faded to 2.5% over the course of a year—still positive but less dramatic than short-term results forecasted. (A subsequent change to the platform elevated the lift back to 4%.)

pinterest test results
The badging group continued to outperform the hold-out group after more than a year, albeit less dramatically than initial test results showed. (Image source)

The takeaway for Egan was clear: “In general, holdout groups should be used anytime there is a question about the long-term impact of a feature.”

3. To feed machine learning algorithms

Today, a Google search on “hold-out groups” is more likely to yield information for training machine learning algorithms than validating A/B tests. The two topics are not mutually exclusive.

As Egan explained, holdouts for machine learning algorithms, “gather unbiased training data for the algorithm and ensure the machine learning algorithm is continuing to perform as expected.”

In this case, a hold-out is an outlier regarding look-back windows: “The holdouts for machine learning algorithms run forever.

These use-cases make sense, but all come with costs, which can quickly multiply:

  • Teams spend time identifying a hold-out segment.
  • Teams spend time maintaining the hold-out version of the website.
  • A portion of the audience doesn’t see a site change that tested better.

In some cases, the justification for a hold-out group derives not from a commitment to rigorous testing but from methodological mistrust.

When holdouts skirt the larger issue

Tim Stewart, a consultant for SiteSpect and trsdigital, is usually “setting up testing programs or rescuing them.” The latter, he noted, is more common.

As a consultant, he often meets directly with the C-Suite, a privilege many in-house optimization teams don’t enjoy. That access has made him a skeptic of using holdouts: “With holdouts, the answer to ‘Why?’ seems to be ‘We don’t trust our tests.’”

Stewart isn’t a full-blown contrarian. As he told me, he recognizes the benefits of hold-out groups to identify drop-offs from the novelty effect, monitor the cumulative effect of testing, and other rationales detailed previously.

But too often, Stewart continued, holdouts support statistically what teams fail to support relationally—the legitimacy of their process:

I understand what [CEOs] want. But testing does not give you an answer. It gives you a probability that the decision you make is in the right direction. Each one individually is only so useful. But if you structure a set of questions, the nth cumulative effect of learning and avoiding risk is worthwhile. That’s the faith-based part of it.

In other words, a valid testing process diminishes the need for holdouts. Running those tests, Stewart said, is:

a lot of money and effort and caveats [that] defers any kind of responsibility of explaining it to the business. For proving the business value, you should be proving it in other ways.

That’s especially true given the opportunity costs.

The opportunity costs of holdouts

Testing resources are limited, and using resources for holdouts slows the rate of testing. As Amazon’s Jeff Bezos declared, “Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day.”

Opportunity costs can rise exponentially due to the complexity of managing hold-out groups, which businesses often underestimate.

Stewart has an analogy: Imagine a pond. Toss a large paving stone into the pond. How hard would it be to measure the size and effect of the ripples? Not too hard.

ripple in pond

Now imagine throwing handfuls of pebbles into the ocean. What effect does each pebble have on the waves that pass by? What about at high tide or low? Or during a hurricane? Pebbles in the ocean, Stewart suggested, are a more accurate metaphor for holdouts, which must factor in everything from offline marketing campaigns to macroeconomic changes.

Can a hold-out group still provide an answer? Yes. But at what cost? As Stewart asked: What’s the ROI of statistical certainty measured to three decimal places instead of two if your control isn’t much of a control?

At a certain point, too, you need to include yet another variable: The impact on ROI from using holdouts to measure ROI. And, still, all this assumes that creating a hold-out group is feasible.

The illusion of feasibility

There is no true hold-out,” Stewart contended. “Even on a control, there are some people who come in on different devices.” (Not to mention, Edgar Špongolts, our Director of Optimization at CXL, added, users with VPNs and Incognito browsers.)

Holdouts exacerbate the challenges of multi-device measurement: The longer a test runs, the more likely it is that someone deletes a cookie and ends up crossing from a “no test” to “test” segment. And every effort to limit sample pollution increases the costs—which slows the rollout of other tests.

Say you want to go down the rabbit hole to determine the ROI of a testing program—cost is no factor. As Stewart outlined, you’d need to do more than just hold out a segment of visitors from an updated site.

You’d need to withhold all test results from a parallel marketing team and, since websites are never static, allow them to make changes to the hold-out version based on gut instinct. Stewart has presented executives with that very scenario:

What we actually have to have is a holdout that includes all of our bad ideas and our good ideas. It’s not holding an audience—it’s running a site without the people who are making the changes seeing any of the test results. Why would we do that?! My point exactly.

Stewart doesn’t make his argument to eschew all use of holdouts. Instead, he aims to expose the misguided motivations that often call for it. Every test result offers probability, not certainty, and using hold-out groups under the false pretense that they’re immune to the ambiguities that plague other tests is naive—and wasteful.

A holdout doesn’t free analysts from dialogue with management, nor should management use a hold-out result to “catch out” teams or agencies when, from time to time, a test result fails to deliver on its initial promise.

“It’s not really about the math,” Stewart concluded. “It’s about the people.”

Conclusion

“Can you do it easily, cheaply, and with enough of your audience?” asked Stewart. Underwood and Egan have done it, but not because of testing efficiency alone.

Both have earned the autonomy to deploy holdouts sparingly. Their body of work—test after test whose results, months and years down the road, continue to fall within the bounds their initial projections—built company-wide faith in their process.

Top-down trust in the testing process focuses the use of holdouts on their proper tasks:

  • Unearthing the easily reversible false positives that short-term tests periodically bury.
  • Confirming the long-term value of a high-cost change before investing the resources.

The post Hold-Out Groups: Gold Standard for Testing—or False Idol? appeared first on CXL.

8 Affiliate Marketing Trends in 2019

As with any industry, trends in affiliate marketing are constantly in flux. And if you want to ensure you’re using your dollars wisely… > Read More
The post 8 Affiliate Marketing Trends in 2019 appeared first on Retail Performance Marketing Blog – CP…

As with any industry, trends in affiliate marketing are constantly in flux. And if you want to ensure you’re using your dollars wisely... > Read More

The post 8 Affiliate Marketing Trends in 2019 appeared first on Retail Performance Marketing Blog - CPC Strategy.