UX Writing and Content Strategy: What’s The Difference?

Today, we are overrun with ads. They are everywhere – intrusive, loud, flashy, and annoying.  So how can companies ensure an effective and customer-centered marketing campaign? The answer lies on the surface. Strong user experience and relevant co…

Today, we are overrun with ads. They are everywhere – intrusive, loud, flashy, and annoying.  So how can companies ensure an effective and customer-centered marketing campaign? The answer lies on the surface. Strong user experience and relevant content techniques help users get past all the noise and give them the insights they need to solve […]

The post UX Writing and Content Strategy: What's The Difference? first appeared on Loop11.

Where do the best marketers turn for advice and inspiration?

What worked in SEO, content, and growth just a few months ago may not be effective today. Making things even more challenging, there’s so much noise. Is that top-ranked content on Google actually the best thing out there? Or is it the same “me too” content? We identified top marketers based on some good-but-imperfect criteria […]

The post Where do the best marketers turn for advice and inspiration? appeared first on CXL.

What worked in SEO, content, and growth just a few months ago may not be effective today. Making things even more challenging, there’s so much noise. Is that top-ranked content on Google actually the best thing out there? Or is it the same “me too” content?

We identified top marketers based on some good-but-imperfect criteria (e.g., mentions on marketing sites, social media presence, recent presentations, etc.).

Then, we used that expert seed list to gather opinions on which people, sites, and books all marketers should listen to, read, or watch.

Our sample wasn’t a perfect representation of the marketing industry, and people on the list who knew about CXL were more likely to respond. Like every research project, there were limitations.

But among the 50 respondents, we saw some striking patterns on how the best “sharpen the saw”—and how you can, too. Here are six takeaways from the three open-ended questions we asked.

Which marketers do you learn the most from? 

One of the best parts about being a marketer today is the ability to engage directly with some of the smartest minds today. You can invest in courses, books, and seminars, sure, but the amount of high-quality content available in blog posts, industry groups, and on social media gives you instant access to level up your skills. 

So what did we learn? 

1. There are a lot of fish in the sea.

All marketers have heard of Seth Godin. But the household names of marketing didn’t dominate our list. 

In fact, we got 135 different names from a total of 150 responses. Many were names that you don’t see on the same old “these people have tens of thousands of Twitter followers” lists, such as Steven Tristan Young, Michelle Morgan, and Kameron Jenkins

Investing the time to find those “hidden gems” gives you a competitive advantage over those who follow only the well-known names. When was the last time you actively searched for new marketers to follow and learn from?

One effective way to discover under-the-radar marketers is by looking at who the big names follow on Twitter. Many times, successful marketers follow a small amount of people, making each follow count. 

2. Follow the practitioners   

By and large, traditional “influencers” and “thought leaders” were rarely mentioned. Entrepreneur and motivational speaker Gary Vaynerchuk for example, came up just once.

The bulk of the responses included individuals who are in the weeds doing the work—or have gotten out of the weeds through years of great results:

  • Benji Hyam, co-founder of Grow and Convert, doesn’t just write about content marketing; he built a content marketing agency by getting his clients results.
  • Web Smith, founder of 2PM, isn’t just tweeting about DTC strategy; he co-founded the popular clothing brand Mizzen+Main and is an investor and advisor to dozens of top-performing companies.
  • Lily Ray is a sought-after speaker because she’s overseen and executed strategies for household brands as Director of SEO at Path Interactive.

If you’re an analyst, you may spend most of your time seeking out tactical information. But paying attention to how marketers have moved up the ranks—even if you love the individual contributor life—shows how to advance your career. You may become more aware of the skillsets you don’t yet have or better understand what managers and executives need and value.

You may also want to expand your list beyond well-known marketers. As one survey respondent shared, “I follow the ones no one knows about. Usually it’s the in-house folks.”

Compared to freelancers and consultants, many in-house practitioners neglect their online brand because it doesn’t directly affect their ability to generate business. One way to find in-house practitioners is to search Linkedin for companies you respect, then filter the list of employees for those in your field. 

Which marketing websites do you pay the most attention to?

Following world-class marketers online is one thing, but what websites did our survey participants keep an eye on regularly? And what were the learnings?

3. Go deep, not wide.

Our survey data was anonymous, but it was easy to guess respondents’ niches. Why? Many listed multiple websites, all covering the same topic—there was a clear focus for what they read regularly. 

For example, one response listed Sterling Skhy, Whitespark, and SearchLab Digital—all local SEO sites. Another shared First Round, SaaStr, and Paul Graham’s blog. (Take a guess what they work on!) 

There’s certainly a benefit to reading a wide range of marketing publications, but don’t go wide at the expense of going deep. Even top, T-shaped marketers focus first on the space they “own.” 

4. Top marketers rely more on people, not websites, to curate content.

Another interesting theme: Lots of responses listed “Twitter” and “newsletters” rather than any specific site:

  • “I mostly read newsletters now: Kevin Indig, Justin Mares, and CXL.”
  • “I don’t really consume content on websites. Maybe Medium from time to time.”
  • “I don’t read blogs anymore, just filtered Twitter and LinkedIn feeds, plus email content.”
  • “Ironically, I don’t follow specific websites. I follow smart people on Twitter and Linkedin and pay attention to what they point me to.”

Several newsletters, in contrast, came up more than once: Morning Brew, The Information, and the SEO for the Rest of Us newsletter. (A bit surprisingly, there were no mentions of Facebook or Slack groups.)

Sparktoro’s Trending page, which elevates popular marketing content on Twitter, got a number of mentions, too. The shift away from blogs and toward individual (or algorithmic) recommendations reinforces a trend that Superpath’s Jimmy Daly wrote about on Animalz years ago:

“Here’s what a publication mindset looks like in practice:

  • Topics are horizontally integrated, meaning that content creators cover a broad range of topics rather than the full range of depth.
  • Posts are published on a strict schedule, so it’s hard to make time for content that requires additional time and energy.
  • Content serves an audience, therefore timeliness is prioritized.

And here’s why those things are problematic:

  • Depth is almost always more useful to readers than breadth.
  • Content efforts that require a lot of effort (think benchmark reports, data analysis, etc.) often deliver 10x the results of a post that requires less effort.
  • The huge majority of readers are not regular visitors to your site. Instead, they seek out specific articles to solve specific problems.”
Publication vs Library Approach.

Animalz took a look at “a few very successful SaaS blogs and found that, on those sites, only about 17% of visitors are returning.” If you’re still consuming content based on what a handful of established blogs show you, you may be missing out on bleeding-edge ideas.

Websites you probably know about that did come up often:

Under-the-radar newsletters you may not know about: 

Which books have influenced your work the most? 

We didn’t specifically state that the books had to be marketing related—and plenty of respondents strayed beyond the business book genre.

Others mentioned that they preferred short-form content (i.e. blogs or newsletters), which wasn’t a total shock given the padded page counts or blog post mash-ups that too many business books have become. 

Here are our takeaways: 

5. Marketers are humans, too. 

The best marketers aren’t just reading about tactics and best practices. We all stress about hitting deadlines. We want to know the best way to ask for a raise. We want to build better habits and increase our productivity. 

Most of the books focused on these topics. Books such as Atomic Habits, Deep Work, Essentialism, and Never Split the Difference were mentioned often. One survey respondent shared that the 7 Habits of Highly Effective Teens was one of the most influential books they ever read. Talk about lifelong impact. 

None of the books talked about how to improve a conversion rate. But they helped the reader become a better, more effective human, which, of course, affects everything else. 

6. Psychology—a love story.

Marketers naturally want to better understand how humans act and behave. 

Books such as Tribes, Pre-Suasion, and Drive came up frequently. Not surprisingly, so too did Robert Cialdini’s 1984 classic, Influence

Understanding human psychology will make you a better marketer and help you better understand yourself. 

If you’re looking for your next great read, here’s a reading list from some of the top marketers today:

Bonus: Reverse engineer these answers to create content that will earn the attention of top marketers.

The best marketers in the world are the best for a reason. When planning and executing your content strategy, here a few things to keep in mind to help capture their attention:

Go deep(er).

No one wants to read another blog post that rehashes content from the top three spots on Google. Great marketers want to hear your point of view, especially if you have the results to back up your claims. There’s a reason why many marketers are turning to smaller, gated sources for their information.

Source quotes from practitioners who have experience doing the thing you’re writing about (if you don’t have it yourself). Do original research. Amplify new voices. 

Highlight the human element in your field.

You don’t always have to focus on the nitty gritty tactical details to stay top of mind. Sometimes, showing your audience how to handle difficult life situations and challenges that are relevant to your industry can add the most value. 

If you write about content marketing for example, you can show how to handle the stress of managing a stable of freelance writers. Or, if you want to attract agency eyeballs, you could interview agency owners about how they’ve handled layoffs. 

Think beyond your blog.

Blogs aren’t dead, but marketers are discovering information in new ways. 

Are you sharing unique content in your newsletter? Are you engaging in Twitter chats? Do you participate in Facebook, LinkedIn, or Slack groups? Share exclusive content in online communities.  Guest author content for other popular newsletters. 

Conclusion

It’s tough to get 50 super smart marketers in the same room (even in a normal year). It’s even harder to get a minute of their time.

We asked for that, and above is what they taught us—focus on the people who are doing or have done the work; look for curators beyond the blogs; and know that becoming a great marketer is about more than mastering tactics. 

Becoming the best marketer you can be requires you to think and do things differently. We hope these insights give you some inspiration on how to improve your marketing game.

The post Where do the best marketers turn for advice and inspiration? appeared first on CXL.

How marketers can get the highest ROI out of podcasts

Planning, content and guest selection are at the foundation of a successful podcast strategy.

The post How marketers can get the highest ROI out of podcasts appeared first on Marketing Land.

The new at-home lifestyle brought about by the onset of COVID has driven a significant increase in the numbers of listeners and the numbers of programs in the podcasting and audio-streaming space, and where the channel was a secondary or tertiary option for most marketers, having a podcast strategy is now part of many marketing plans for 2021. 

“Podcasting is still a vast, open space that brands are smart to jump into,” said Lindsay Tiepkema, CEO of Casted, a podcast-centered marketing platform. “When you consider there are about 1 million podcasts, but more than 600 million blogs, you can see how much opportunity still exists for brands to own their own space,” she continued. “And listenership continues to rise dramatically, even in the midst of a pandemic, as people are actively seeking connection. Podcasts that offer that ability to connect an audience with a brand like not other form of content can and, in doing so, build trust and loyalty.” 

Research has shown that brands that advertise their products and services or business podcasts enjoy an increase on purchase intent, and major platforms are seeing significant increases. In June, podcast analytics provider Charitable tracked 825 million downloads, up from 600 million in March. 

But aside from using podcasts as an advertising channel, how can marketers use podcasts to achieve the highest ROI? One option is to recognize that the content can be leveraged beyond the podcast itself.

“Go beyond simply publishing episodes to also really wring-out that content,” said Tiepkema. “Marketers are getting into these shows they create to also pull blog posts, audio clips for social media, videos, excerpts to embed in web pages or email content and even thought leadership articles. This makes the stretched marketing team more efficient while also serving the audience more content to dig around that initial conversation with a subject matter expert.”

The podcast platform and planning

While most entertainment and social podcasts are often unscripted, or go off-script, the opposite is encouraged for marketers as they plan their podcast strategy, either as creator and host of their corporate podcast, or as a guest. 

“People are just now starting to realize that you should not just jump into starting a podcast and a podcast is not something you do for fun for companies,” said Sky Cassidy, co-host of the If You Market They Will Come podcast. Marketers failed to capitalize on the growing popularity of podcasts over the last couple of years, mainly because they didn’t have clear objectives and outcomes in mind. “You have to have a real ROI planned before you start, and it has to align with your corporate goals,” said Cassidy.

Podcast conversations spark consumer connections that lead to conversions in sales of a product or service. According to Voxnest, an audio technology solutions provider, global podcast consumption increased 42 percent during the onset of COVID between March and April. 

How can marketers maximize podcast success? Tiepkema mentions shorter, more frequent podcasts to match shorter commute times, and using statistics, facts and figures to make the content authoritive.

However, “[some] shows are publishing richer content less frequently,” said Tiepkema. “This works well for some because while they are publishing less often, like bi-monthly, they are delivering deeper content, so their audience is more likely to make the time to listen because the episodes are really that valuable.” 

Content as the cornerstone of quality podcasts

The cornerstone of any successful podcast is content. 

“You have to have something to give the audience other than a sales pitch,” said Karla Joe Helms, co-host of the If You Market They Will Come podcast. “As a marketer don’t look to try to monetize your podcast, worry more about building a consistent high-quality content outflow and go from there. People get tired of sales pitches.” 

After removing blatant sales pitches, podcast content is also dependent on high audio quality. Marketers should be willing to pay a bit more for higher audio quality platforms and tools. Always avoid podcast segments being recorded in coffee shops, moving vehicles, and at public places with spontaneous loud noises like malls and parks. 

Effective corporate podcasts educate their audience on a subject, rather than deliver a blatant sales pitch or provide pointless entertainment. One corporate activity naturally lends itself to the podcast format–the new product line or service launch/rollout. This announcement can be included in educational content that identifies the problem or need being solved by the new product or service. 

“It is the most non-captive audience you can have,” said Cassidy. “There are thousands upon thousands of podcasts out there, why would a consumer choose yours? That is the question that you have to constantly ask yourself as a marketer.” 

One way to increase the chances of your podcast being selected is to invite guests with large followings, either through their own podcasts, or on social media. “If you are depending on your podcast to grow your brand an audience, you will not get one,” said Cassidy. “Offer content that nobody else offers, and find your niche before you worry about big content. When you do, start with people, brands and entities that already have a large following. Now you have a strategy and you can be in podcasting for the long haul.”

Marketers can get the most out of their podcasts guests by allowing the guest to choose from two or three subjects so they can speak on a subject they are most comfortable with; conducting a prep call for all guests, no matter their experience or popularity level; and encouraging guests to invite their audience multiple times before, during and after podcasts.

“Energy and the audience they bring are everything for podcast guests,” said Helms. “If you nail those two down and combine it with high-quality content any marketing team can execute from there because you have already completed the hard part.” 

The post How marketers can get the highest ROI out of podcasts appeared first on Marketing Land.

Smart Data Visualizations: Quality Assessment Algorithm

The gap between a bad and good data visualization is small. The gap between a good and great data visualization is a vast chasm! The challenge is that we, and our HiPPOs, bring opinions and feelings and our perceptions of what will go viral to the conversation. This is entirely counter productive to distinguishing between […]

The post Smart Data Visualizations: Quality Assessment Algorithm appeared first on Occam’s Razor by Avinash Kaushik.

The gap between a bad and good data visualization is small.

The gap between a good and great data visualization is a vast chasm!

The challenge is that we, and our HiPPOs, bring opinions and feelings and our perceptions of what will go viral to the conversation. This is entirely counter productive to distinguishing between bad, good, and great.

What we need instead is a rock solid understanding of the updraft we face in our quest for greatness, and a standard framework that can help us dispassionately assess quality.

Let’s do that today. Learn how to seperate bad from good and good from great, and do so using examples that we can all relate to instantly.

We’ll start by looking at the two sets of humans who are at the root of the conflict of obsessions and then learn to assess how effective any data visualization is in an entirely new way. If you adopt it, I guarantee the impact on your work will be transformative.

The Conflict of Obsessions.

There are two parties involved in any data visualization.

1. Analyst/Data Visualizer.

As I’ve passionately shared frequently on this blog, we, Analysts, are all in the business of persuasion. We work against that desired outcome because when we work on creating a data visualization, here are our top-of-mind concerns/desires/perspectives:

How can I cram as much as I can into the graphic?

What can I include to ensure everyone clearly gets just how much work I did?

How much of my agenda do I need to make overt, and how much can I make covert?

Is there something I can add to increase the chances that this will go viral and result in fame and glory?

Ok. I’m only teasing.

But, as an Analyst, a Data Visualizer, I can’t say that these thoughts don’t cross my mind. :)

I’m sharing the above primarily to ensure that you know these motivations exist – and, like me, you should try to fight and resist!

The very best Data Visualizers, obsess about:

1. known and unknown variables
2. causality
3. nuance
4. visualization techniques
5. rank-ordering messages
6. simplicity, simplicity, simplicity, simplicity, simplicity, simplicity, and, just to be safe one last time, simplicity.

These are the six things that matter supremely in my work, and they should be what matter in yours.

Simplicity matters more than the rest because if I can’t distill complexity, I might as well not do the work because that is only a snowball’s chance on the sun that the audience will understand my complex visual.

Let’s look at the other set of humans involved in a data visualization equation.

2. Data Consumer.

Here are the concerns/desires/perspectives that a consumer of data visualizations has top of mind when they are presented with a set of analysis:

What’s in it for me?

How easy is it  to grasp the most important point?

What’s in it for me?

How much effort do I need to put in to understand the whole infographic?

What’s in it for me?

How can I trust that this message is from a credible Analyst/source/using sound methodology?

(Never underestimate the staggering selfishness that a Data Consumer brings with them to the table when you are showing them a table of data or a data visual. And, it is understandable because they have difficult jobs and 71 other things to worry about.)

Notice there is very little overlap between the obsessions of the Data Consumer and Data Visualizer.

If you have a choice (and you do!), let the needs of the Data Consumer drive your data visualization efforts. The only exception is when you are trying to push propaganda, then go with your agenda.

If an infographic sucks, it is usually due to the conflict between the Visualizer and the Consumer along the above dimensions.

You’ll see it vividly on display when you look at any graphic through the Consumer lens with an eye on simplicity (the Analyst dimension).

The Data Visualization Assessment Algorithm.

Algorithm might perhaps be a tad bit pompous, as applied here. I’ve developed a set of filters and lenses through which you can look at any data visualization in order to quickly assess quality.

Perhaps someone reading this blog post is going to help us all out by building a Machine Learning algorithm to assess if a Data Viz is bad, good, or great. :)

Reflecting on the aforementioned Consumer vs. Visualizer conflict of obsessions has helped me distill the evaluation of data visualizations to eight dimensions. They influence each other and the entire portfolio, yet they stand on their own.

In the format of “Obsession | [ratings scale],” here’s the data viz assessment algorithm:

1. Time to the most important insight. [Scale: Fast. Slow. KMN!]

2. The effort to understand the whole graphic. [Low. Medium. No Thank You.]

3. Trust marks. [Clear. Non-Obvious. None.]

4. Rank-ordering of key messages. [Yes. Partial. WTH!]

5. Explaining the key logic powering the graphic. [Super clear. Cloudy. Invisible.]

6. Exposing nuance. [Sweet. Some. Sour.]

7. Visualizer trying to be too clever. [No, and thank god. Yes, but it is harmless. Yes, sadly.]

8. Likely to recommend to influential leaders. [Yes! No. No way.]

I want you to explicitly notice:

I’ve put the Data Consumer first

Incentivized good behavior by the Data Visualizers, and …

… Included an outcome in the end because activity is well and dandy but it is outcomes are what matter.

My hope is to share a very specific algorithm that gets your critical thinking juices flowing. I invite your critique and suggestions on how I can make it even smarter. Please reply.

The best way to learn is to practice via real-world examples. So.. Let’s do that!

COVID What Should I be Afraid of (!) Data Visualizations.

A few weeks ago, perhaps not coincidentally, a number of different entities published visuals to help us understand what we can do safely and what’ll cause grievous harm.

I’ve collected four of these efforts – each a really different way to visualize nearly identical information. This gives us an ideal data set to apply our algorithm, and learn discerning skills along the way.

Data Visualization #1

The first graphic is from the inimitable Randall Munroe (I’m a very big xkcd fan!).
Randall has a unique way to communicate complex information (buy Thing Explainer!), and this graphic is no different. It combines seriousness, fun, and scientific accuracy.

As an approach, 2x2s work really well. They force simplicity. The color clustering above helps, you can jump to the safest or riskiest activities faster.

On the downside, it is hard to take in the whole thing. You can get lost.

I’m treating this as a very serious example, but it is important to remember that the intent above includes the goal of making us smile.

Let’s apply our algorithm and see how this graphic does with our tough, but with love, lens.

1. Time to the most important insight. [Fast. Slow. KMN!]

2. The effort to understand the whole graphic. [Low. Medium. No Thank You.]

3. Trust marks. [Clear. Non-Obvious. None.]

4. Rank-ordering of key messages. [Yes. Partial. WTH!]

5. Explaining the key logic powering the graphic. [Super clear. Cloudy. Invisible.]

6. Exposing nuance. [Sweet. Some. Sour.]

7. Visualizer trying to be too clever. [No, and thank god. Yes, but it is harmless. Yes, sadly.]

8. Likely to recommend to influential leaders. [Yes! No. No way.]

The graphic should technically get a pass on #3 as it is for fun, and possibly #5 as well. But, I’ve still graded it seriously so that all of us can practice scoring.

If the phrase big miss applies here it is perhaps #2, the effort to understand the whole graphic (or more precisely, cartoon).

Based on the algorithm’s assessment, it earns a score of 23/66.

Oh, I totally forgot to tell you… I made a little scoring system to help you truly internalize the key messages. Those who know me will not be surprised that my system has a steep grading curve (#highstandardsFTW!).

The scoring system uses a multiplier across each rating in the scale above. Additionally, since each dimension does not carry the same level of importance, there’s a multiplier for each dimension – to effectively communicate my values.

Here’s the math…

It is all fun and games until you realize there’s a score involved! :)

Important: My intent in creating the data viz assessment algorithm, and scoring sheet, is not to have you entirely agree with how I’m grading each visualization. My intent is to teach a systematic approach you can bring to these difficult and complex tasks.

I do hope you see why I’m scoring the way I am, I hope you’ll agree. But, that desire is tertiary.

Data Visualization #2

The second graphic is from the world-famous Information is Beautiful (IiB). They have some of the world’s most famous data visualizations. (The simple and effective: When Sea Levels Attack)

IiB tends to make graphics for large screens, I need to be on my beloved 27” ThinkVision monitor to read it optimally.

In this instance, you’ll notice the color palette works against the ability to read the text (teal on dark gray or slightly lighter gray on dark gray).

The spectrum from light yellow to blood red of the circles, with internal gradations, is trying to add a layer of cleverness that possibly satiates a Data Visualizer, at the cost of the Data Consumer.

Once you zoom into one part of the visual, things become readable. You do lose the full picture of any section. In this view, perhaps you’ll agree that there is a sense of randomness to what’s in the bubble (check for this in the two visuals below as well).

It was a lovely touch to add the “risk factors to consider” on the top left of the visualization which explains the logic powering the graphic.. (You can see it more clearly in the higher resolution view, the blue font on gray makes it hard above.)

I do like the subtle helpful tips like the one about condiments, below.

Let’s apply our algorithm and see how this graphic does with our tough, but with love, lens:

1. Time to the most important insight. [Fast. Slow. KMN!]

2. The effort to understand the whole graphic. [Low. Medium. No Thank You.]

3. Trust marks. [Clear. Non-Obvious. None.]

4. Rank-ordering of key messages. [Yes. Partial. WTH!]

5. Explaining the key logic powering the graphic. [Super clear. Cloudy. Invisible.]

6. Exposing nuance. [Sweet. Some. Sour.]

7. Visualizer trying to be too clever. [No, and thank god. Yes, but it is harmless. Yes, sadly.]

8. Likely to recommend to influential leaders. [Yes! No. No way.]

I was this close to choosing no way in terms of recommending this graphic to others (because I never will). In the end, IiB is such a huge entity and so famous and so many people love them… no way seemed too much against the grain.

I've come to understand that IiB has a very specific design language, texture, and philosophy that has come to define them. It possibly acts as a constraint now.

Based on the algorithm’s assessment, it earns a score of 7/66.

Here’s the math:

It is important that data this critical – for this wide a consumption (whole planet) – needs to figure out how to hit an extraordinarily high simplicity and effective comms standard.  Else, it remains an exercise in self-satisfaction by the Data Visualizer.

Data Visualization #3

The third graphic is by Professor Saskia Popescu, Dr. James P. Phillips, and Dr. Ezekiel Emanuel.

I’m a huge fan of Dr. Emanuel. He was the special advisor for health policy in the Obama administration and played an instrumental role in passing the Patient Protection and Affordable Care Act (aka. Obamacare). For this, he has my eternal gratitude on behalf of those who society and politicians don’t usually listen to in the United States.

The Covid-19 Risk Index clearly identifies the logic powering the graphic: enclosed space, crowds, duration of interaction, and forceful exhalation.

Note that IiB also had some of these factors, forceful exhalation is an addition here (unsurprising that the doctors brought that to the fore).

The colors in the graphic are related to the intensity of the risk, green is low and red is high. Simple, direct, effective.

I’m not a huge fan of a giant company logo on graphics as you see below in the "hexagon art." I believe: More white space = more peace.

Given the heartbreaking debate in the US, I did appreciate the bonus call to action up top to wear a mask.

Did you notice the trust marks at the bottom? Really nice.

As in the case with the IiB graphic, this one is meant for the large screen display. I applaud the team for making sure each segment is readable – no fancy font colors and fancy background as a demonstration of the Visualizer's smartness.

Folks in my teams know I hold a special hatred for icons. They add clutter. In this case, I do support the decision to include icons.

For example, without needing to read any text I know that working in the office carries medium/high risk, and participating in group religious services is in the recommend you please avoid category – even in the small version above and certainly in the zoomed-in version below.

Let’s apply our algorithm and see how this graphic does with our tough, but with love, lens.

1. Time to the most important insight. [Fast. Slow. KMN!]

2. The effort to understand the whole graphic. [Low. Medium. No Thank You.]

3. Trust marks. [Clear. Non-Obvious. None.]

4. Rank-ordering of key messages. [Yes. Partial. WTH!]

5. Explaining the key logic powering the graphic. [Super clear. Cloudy. Invisible.]

6. Exposing nuance. [Sweet. Some. Sour.]

7. Visualizer trying to be too clever. [No, and thank god. Yes, but it is harmless. Yes, sadly.]

8. Likely to recommend to influential leaders. [Yes! No. No way.]

This graphic went viral on the socials, and deservedly so. With CV-19 flaring up in multiple countries (sadly, we in the US are still making our way through wave one), I hope that you will use the graphic above to stay safe – and share it with your friends and family so that they can stay safe as well.

Based on the algorithm’s assessment, it earns a score of 50/66.

Here’s the math:

Clearly a graphic the Data Visualizer can be proud of, reaching a level of obsessions overlap with Data Consumer obsessions that is rare.

Data Visualization #4

The last graphic was developed by the physicians on the Texas Medical Association COVID-19 Task Force and TMA Committee on Infectious Diseases.

I love it.

It is simple. It is easy to digest. There is absolutely nothing cute about it (hurrah!). There are no circles to jump through. No expensive Data Visualizer Specialist In Fonts was hired. The graphic is not trying too hard.

It was probably designed by the Doctors in TMA. It is insanely boring. All it is is… Effective.

Just about the only lite criticism I can make is that perhaps in keeping with the (ironically) liberal posture of the state of Texas when it comes to dealing with Covid, this graphic lowers the bar for what’s risky compared to all other sources. I share that as a small red flag, but it is adjacent to the technical analysis of the data viz that we are undertaking today.

The logic powering the graphic is integrated into the core of the graphic, as becomes clear below. There is little to no effort necessary to understand the visual. Start at the top, keep going. The colors and bars help you along.

Even in this small size, it is fairly readable…

When information is laid out so clearly other things jump out at you that makes you think (an excellent trait of a great data visualization).

All of the below items are an 8 or a 9 – but consider the staggering differences.

Attending a bar is just as risky as a religious service with 500+ worshipers! And, both are a tiny bit riskier than eating a buffet!!  You were leaned-in questioning the data, being curious. A good sign.

TMA COVID Highest Risks

Let’s apply our algorithm and see how this graphic does with our tough, but with love lens:

1. Time to the most important insight. [Fast. Slow. KMN!]

2. The effort to understand the whole graphic. [Low. Medium. No Thank You.]

3. Trust marks. [Clear. Non-Obvious. None.]

4. Rank-ordering of key messages. [Yes. Partial. WTH!]

5. Explaining the key logic powering the graphic. [Super clear. Cloudy. Invisible.]

6. Exposing nuance. [Sweet. Some. Sour.]

7. Visualizer trying to be too clever. [No, and thank god. Yes, but it is harmless. Yes, sadly.]

8. Likely to recommend to influential leaders. [Yes! No. No way.]

Based on the algorithm’s assessment, it earns a score of 64/66.

Here’s the math:

The TMA graphic was the spark to write this newsletter.

The world needed a simple way to communicate effectively, in this case literally, information that can save lives.

While things are rarely that high-stakes in a business environment, I hope the TMA inspires you to ensure that you don’t lose sight of what’s important when you work on data visualizations: The understanding of data.

Bottom line.

How do you handle the conflict between your goals as a Data Visualizer (and incentives your employer creates for you) and the Data Consumer? While the answer seems obvious, it is incredibly difficult to execute. I hope you’ll use the data visualization assessment to ensure you, your team, solve for the Data Consumer first, yourself second.

If you have graphics that score above 60, I would love to see them! (If they are shareable.)

All the best.

PS: Bonus Life Lesson:

A small number would surely have noticed that the perfect score from the algorithm is 66 (all Great), and the score for it was good enough is 22 (all Could Be Optimized). That massive chasm reflects life (and my philosophy).

There are thousands of Analysts who’ll stop at good, after all it is good. Perhaps a hundred, or less, will do the hard work required to get to great. They’ll rule the (biz) world.

#nowyouknow

The post Smart Data Visualizations: Quality Assessment Algorithm appeared first on Occam's Razor by Avinash Kaushik.

A Practical Guide to Building a CRO Roadmap

If you’ve recently embarked on your CRO journey, here’s a couple of questions for you: How do you prioritize your experimentation ideas? Do you work in silos, or do you see benefit in opening up experimentation to collaboration? If you do see benefit, how do you plan to go about achieving it? How do you…

If you’ve recently embarked on your CRO journey, here’s a couple of questions for you: How do you prioritize your experimentation ideas? Do you work in silos, or do you see benefit in opening up experimentation to collaboration? If you do see benefit, how do you plan to go about achieving it? How do you plan to address resource issues in your testing plan? The answer to all these questions points to one strategic move that differentiates CRO experts from beginners – building a CRO roadmap. 

Building a sustainable CRO roadmap guides your efforts and ensures it systematically contributes towards your business goals at large. Whether you are an agency handling CRO for hundreds of clients or someone who manages CRO for your company, a roadmap will streamline your efforts and maximize throughput by avoiding redundancies and providing a clear step-by-step approach towards optimizing your site. 

Similar to a calendar, a CRO roadmap is essentially a detailed schedule that entails which experiment will be launched when, the time and resources it requires, and the expected outcome. A roadmap ensures that each tweak, change, test, and insight adds value to the next step and accordingly strengthens it to deliver improved results. With a dedicated roadmap to consult, you don’t rely on hope to get results from a few poorly planned and ill-executed experiments scattered across months.

Cro Roadmap illustration

Why do you need a CRO roadmap? 

You can think of a CRO roadmap as a step-by-step framework that you refer to for prioritization, test planning, and allocation of resources for all your CRO efforts, without which you would be completely shooting in the dark. Here are some of the major reasons you need a CRO roadmap to get started.

To switch from a fragmented to a strategic approach

If you randomly run a survey on your homepage this month and conduct a couple of tests on your product page the next month (and so on), you are not going to be able to make the most of the insights gathered or leverage the full potential of the results. To do so, you need a roadmap that dictates every process so you can feed every insight and learning into your pipeline and use it judiciously to drive more substantial results from your program as opposed to some scattered wins or losses.

Let’s say you want to improve your online store’s checkout rate. Needless to say, there are tonnes of tests you can run to optimize for the same. For instance, you could optimize the number of steps in the checkout flow, add social proof and trust badges, avoid the addition of surprise costs at the last step, and so on. Now, without a roadmap, you wouldn’t know which one to prioritize and you might just end up spending too much time running each one of them without getting the expected outcome. On the other hand, if you follow a roadmap, prioritize tests, plan and scope them out over a calendar month/quarter, you can be assured of more promising results.

To get a better hold of resource planning

Again, if you have a systemic approach to optimization, you can always plan your resources in advance, delegate projects, and overall function smoothly with little or no friction as opposed to facing a mini resource crisis every time you decide you want to run an ad-hoc test. 

Moreover, you can always learn from experience and incorporate your learnings of how you can allocate resources better to drive more significant results, efficiently. This is not possible if you follow a haphazard outlook towards optimization and don’t depend on any set framework to guide decisions. 

To improve the speed and efficiency of your CRO program

Needless to say, optimizing your digital properties methodically will only improve the efficiency of your efforts as you would be incorporating previous learnings and doubling down on what works well. Having an overarching roadmap also ensures your processes and tasks are aligned with the overall business goals, so there is minimum iteration, faster delivery, and more promising results.

For instance, if you follow a roadmap, you will know which tests you have to run in the coming month and have the liberty to start laying the groundwork (analyzing data, getting variations created, etc.) and plan your resources accordingly. On the other hand, if you are running sporadic tests, you will end up wasting time in deciding what to test next, ensuring it doesn’t overlap with another test, and planning your resources for it.

How to develop a successful CRO roadmap

An Example Of A Successful Cro Roadmap
Steps to create a CRO roadmap

Revisit your business goals

Take a step back to revisit your most pressing and current business goals so you can understand how CRO can help you achieve them. These goals will anchor your CRO program and ensure your efforts are not aimless or applied in the wrong direction. 
For example, an eCommerce company could have a business goal to increase the average order value, while for a media company, the goal could be to uplift the content consumption on their site. These will then help you deduce what your optimization goals (and their corresponding metrics) need to be.

Deduce corresponding website goals, KPIs, and target metrics from your business goals

Use your business goals to drill down upon what are some of the more tactical website goals you want your CRO program to achieve, what are the performance indicators you need to watch out for, and what would be the target metrics you need to measure corresponding to them. For instance, if increasing the average order value is your business goal, you can break it down further into:

  • Increasing upsell & cross-sell 
  • Increasing visits to product pages
  • Increasing checkout and ‘Add to cart’ rate

Now, these could be your optimization goals, each of which you can tackle using specific strategies and tests. The metrics to be measured could be revenue per customer, conversion rates, and so on. 

Flow Diagram Of Optimization Goals At Different Stages

Understand where you currently stand and establish a baseline for your key metrics

Before commencing, you will need to perform a CRO audit of your site to establish a starting point for your optimization program against which your progress can be assessed. Therefore, for all your key metrics, be sure to analyze your historical data so you can condense it into a baseline basis the trends and patterns it shows. Make sure the date range you select for this is not less than 30 days, as you would need a substantial amount of data to be able to gauge your business’ past performance against these metrics.

Formulate data-backed hypotheses taking insights from visitor behavior data

Use the data you have been gathering through various tools like heatmaps, session recordings, surveys, Google Analytics, and other user research tools to glean insights that you can turn into optimization opportunities. For instance, if you noticed high drop-offs at category pages, you could consider revamping them to highlight the CTA, reduce clutter, and make product details more appealing and apparent to customers. 


Next, craft your hypotheses based on these insights that can move your key metrics and solve for these visitor pain points. Here’s how you can formulate a solid hypothesis:

How To Build A Structured Hypothesis

Prioritize these hypotheses

Optimizing the optimization process is often just as important as the tests themselves. Prioritizing where you invest energy will give you better returns by emphasizing pages that are more important to the business

Chris Goward – Founder, WiderFunnel

Now that you have a bunch of different test hypotheses at your disposal, prioritize them so you can populate your pipeline in a way that you tackle issues that are more likely to yield maximum results first. There are several different prioritization frameworks you can follow for the same: 

Prioritization Of Your Optimization Plans

P.I.E. Framework [By Chris Goward] 

As per this model, each hypothesis is given a score of 1-10 on three factors – Potential, Importance, and Ease. This translates to the amount of potential a particular hypothesis has in improving the performance of a page, how important is it to optimize a particular page, and how easily the task can be accomplished. Once assigned scores for each of these factors, you just add them up for every hypothesis and prioritize them basis their total scores. 

Pie Framework Potential Importance Ease
Image Source: [1]

T.I.R. Model [By Bryan Eisenberg]

As per this framework, a score of 1-5 is assigned based on Time, Impact, and Resources. This means that a test that requires minimal time, has the most impact, and requires the least amount of readily available resources would be given a 5 under every factor. Once done, individual scores of each factor are multiplied, and the highest priority is given to one with the largest score.

I.C.E. Framework [By Sean Ellis]

As per this model, a score of 1-5 (with 5 being the highest) is assigned to each hypothesis based on the likely Impact it would have, the level of Confidence you have in the hypothesis, and what is the level of Ease with which it can be implemented. Once all 3 scores are added, the hypothesis with the highest score is given the highest priority, and so on.  

I C E Framework Impact Confidence Ease
Image Source: [2]

While these were a couple of the most popular frameworks, there are others you can consult to prioritize your tests so they can be picked in the descending order and fed into your pipeline. But, that’s not all. You should also categorize your hypotheses basis the final goals (the ones discussed above) they accomplish.

Collate all the information collected so far

Now that you have drilled down your business goals to tactical conversion goals you plan to achieve with CRO, and also used visitor behavior insights to craft hypotheses that can help you achieve them, you can start breaking each test down into its specific details. 

This would include your test name, description, hypothesis, observation, target page, and goal it is expected to accomplish. Share this spreadsheet with all stakeholders so that it can be enriched and evolved as your progress in your program.

Create your testing pipeline 

Create a proper schedule to plan out your experiments considering their priority order, resource bandwidth, and time required for implementation. Assign owners to each test and make sure you keep track of the progress and the results obtained so you can use your learnings constructively to enrich your pipeline. Here’s an example template you can refer to for creating a weekly A/B testing plan:

An Example Of A Testing Plan Calendar Template

Creating one such calendar would especially be useful for agencies so you can always use a standardized template for all your clients and deliver promised results systematically. 

Challenges to roadmapping 

Miscalculation of time and resources required

Creating a CRO and testing roadmap requires you to carefully plan your schedules and resources well in advance, which can sometimes be a challenge for teams that are new to experimentation. You might not be able to accurately estimate your requirements before actually getting into the process. Very often, running experiments can take extra time, and it’s not in your control to wrap them up sooner. For instance, if a test takes longer (than you had anticipated) to reach statistical significance, you couldn’t have accounted for it, and now you have to hold off the next one to ensure they do not overlap. 

The best way out of this is to always keep a buffer or stick to a conservative estimate, both for time and resources, and any other requirement you might have, to accommodate for unexpected changes that occur during experiments. 

Inaccuracy, inconsistency, or unavailability of data to inform your CRO roadmap 

If you haven’t been relying heavily on data for all decision-making, you might struggle at first with creating a data-backed roadmap. This is largely because you will most likely discover inconsistencies or inaccuracies that don’t add up, you would have data stored in silos, or you wouldn’t have comprehensive data for the entire time range and critical metrics you want to look at. 

To overcome this, use only data you know for sure is accurate and enough to inform your hypotheses and eliminate what you feel is only corrupting your roadmap. You can also prioritize your hypotheses in such a manner that those already backed by sufficient data are ones you test first and collect more data for those that need to be strengthened. 

Not evolving your strategy with changing times and scenarios 

After you have spent a whole lot of time and effort building your roadmap from scratch, it is quite natural to want to just stick to it. However, committing to one strategy with all your heart and soul and not being open to evolving it with the changing dynamics of your business or the industry will do you more harm than good. 

To stay ahead of the curve, you need to keep evolving your roadmap so you can accommodate for these changes. For example, if the festive season is coming up and it’s time for your annual sale, you would want to run a test or two catered specifically to the sale, and hence it’s important you have provision to take that decision quickly and incorporate it into your plan. 

Tools you need to build and maintain a CRO roadmap

VWO Plan 

VWO Plan allows teams to collaborate and create experimentation pipelines seamlessly and efficiently. It’s an all-in-one platform that empowers you to record observations, generate hypotheses, save ideas, and create and manage your experimentation program via a centralized dashboard. You can forget about data/idea silos and maintaining multiple documents, sheets, presentations, whiteboards, or dashboards, and rely on a single platform to ideate, run experiments, and measure their impact. 

Vwo Plan In App Screen

Jira

From the house of Atlassian, Jira is an agile project management and tracking software you can use to manage your CRO pipeline. You can easily plan and track your workflows over a kanban board to the most minute detail and collaborate with various teams to ensure you conduct your experiments efficiently and drive faster results.

Example Of A Jira Board
Image Source: [3]

Trello

Trello is another project management software that allows you to track and manage your CRO roadmap and work collaboratively and efficiently. Apart from organizing, prioritizing, and managing your pipeline, Trello enables you to boost productivity by automating redundant and manual tasks such as due date and calendar commands.

Example Of A Trello Board
Image Source: [4]

Asana

Asana is an easy-to-use project management software that provides a timeline view of your CRO pipeline and allows you to track its progress. It offers integration with all major tools such as Salesforce, Tableau, and Adobe Cloud and customizes your workflows as per your specific requirements. 

Example Of Asana Project Board
Image Source: [5]

Conclusion

Approaching CRO with a strategic roadmap ensures that every effort is tied to your overall business goal. Without one, you are most likely to rely on disintegrated efforts, which may or may not show significant results. Therefore, roadmapping is the way to go to achieve success with CRO and drive noteworthy business growth. It’s time to put your knowledge to test and embark on a strategic CRO journey by creating a roadmap for your program. On that note, Happy Optimizing! 

How to make your data sing

Stop reporting absolute numbers and put your data into context with ratios to engage your stakeholders with the smaller, but important, data points.

The post How to make your data sing appeared first on Marketing Land.

It is amazing; the horrible job many digital marketers do when reporting their work to clients. This includes both internal and external clients. Just think about how many marketing presentations and reports you’ve seen that simply contain screenshots from Google Analytics, Adobe Analytics, Adwords, Google Console, or reports from a backend ecommerce system. This isn’t the way to influence people with your data.

The biggest issue is that most marketers are not analytics people. Many marketers do not know how to collect all of the necessary data or how to leverage that data, and to a lesser degree, know how to present it in a meaningful way. Typically, this is the job of a data analyst. The same way purchasing a pound of nails, a hammer and a saw doesn’t make you a carpenter, gaining access to your analytics reporting tool does not make you a data analyst. This is why many reports contain those convoluted screenshots, and present data out of context, contributing little to no meaning. 

Data out of context

Many reports merely report the facts (the data) with a number and no context. Data out of context is just data. For example, simply making a statement that Adwords generated 5,000 sessions to a website last month is meaningless without context. The number 5000 is neither a good nor a bad data point without a reference point or a cost factor. It’s not until you add in other factors (open the box) that you can demonstrate whether or not your efforts were a success. If the previous month’s Adwords campaign only drove in 1,000 sessions, then yes without other data, 5000 sessions looks good. But what if the cost to drive those additional 4,000 sessions was 10 fold the previous month’s spend? What if the previous month, Adwords drove 5,000 sessions but at double the spend?

It is only by adding in the additional information in a meaningful way that marketers can turn their reporting from a subjective presentation into an objective presentation. In order to do this, stop reporting absolute numbers and put your data into context with ratios. For example, when assessing Cost per Session, toss in a 3rd factor (goal conversion, revenue, etc.) and create something similar to “Cost per Session : Revenue”.  This will put the data into context. For example, if every session generated costs $1 : $100 (Cost per session : revenue) vs. $2.25 : $100 (Cost per session : revenue) the effectiveness of a marketing spend becomes self-evident. In this example, it is clear the first result is superior to the second. By normalizing the denominator (creating the same denominator) the success or failure of an effort is easily demonstrated. 

Data is boring

Yes, presenting data is boring. Simply looking at a mega table of a collection of data will cause many to lose interest and tune out any message you might be trying to present. The best way to avoid this is to make your data sing!

Make your data sing

Just like in the marketing world, the easiest way to grab someone’s attention and make your message sing is with imagery. Take all that great data in your mega table, and turn it into an easy to understand graph, or when necessary, simplified data tables. Even better, (if you can) turn it into interactive graphs. During your presentation, don’t be afraid to interact with your data.  With some guidance, your audience can dive into the data they are most interested in.

Learn to use data visualization tools like Data Studio, Tableau, DOMO, Power BI and others. Leveraging these tools allows you to take boring data and not only give it meaning but to make the data sing, which will turn you into a data hero.

Interacting with your data

Back at the end of July 2019, my firm acquired an electric vehicle. We wanted to know if the expense was worth it. Did the cost savings of using electricity over gasoline justify the difference in the ownership cost of the vehicle (lease payments +/- insurance cost and maintenance costs).

Below is a typical data type report with all the boring detailed data. This is a mega table of data and only those truly interested in the details will find it interesting. If presented with this table most would likely only look at the right-hand column to see the total monthly savings. If presented with just this data, many will get bored, and will look up and start counting the holes in the ceiling tiles instead of paying attention.

The following graphs demonstrate many of the ways to make this data sing, by putting all of the data into context through interactive graphics.

The above graph (page 1 of the report) details the cost of operating the electric vehicle. The first question we were always curious about was how much it was costing us to operate per 100 km. By collecting data on how much electricity was used to charge the car, how many kilometers we drove in a given month and the cost for that electricity, we are able to calculate the operating cost. In the graph you can easily see the fluctuation in operating costs, with costs going up in winter months (cost of operating the heater in the car) and again in June & July (cost of running the AC). You can also see the impact of increases in electricity prices.

To truly evaluate the big question “Was acquiring an electric vehicle worth it?” we’d need to estimate how much gasoline would have been consumed by driving the same distance against the average cost for gas during the same months. On page 2 of the report the data is now starting to sing as the difference in the savings of electrical over gas becomes clear. The chart becomes interactive and allows the user to hover over any column to reveal the data details.

To make the data truly sing, we’d need to not just compare the operating costs, but the costs of ownership. Do the savings in the operating costs justify the price difference between the vehicles? We know that the difference in lease costs, insurance and annual maintenance is in the range of $85-$90/month

The above graph (page 3 of the report) demonstrates, the impact of plummeting gas prices and the reduced driving done during April 2020 due to the COVID-19 shutdown. In April 2020 a mere monthly savings of approximately $41 dollars was achieved. Therefore, there were no savings in owning a more expensive electric vehicle over an equivalent gas-powered vehicle (the difference in lease costs and insurance, etc. is in the range of $85-90/month). While it might not be sing, it definitely was screaming out when we saw it. 

Check out the entire report for yourself.  It is accessible here so you can view all the pages/charts. The report is interactive allowing you to hover given months to see data details or even change the reporting date range.

By embracing not only data visualization but the visualization of meaningful data, we as marketers can raise the bar and increase engagement with our audience. Think of the four pages of this report, which page talks most to you? Which way of presenting the data makes it sing for you? Odds are it was not the first table with all the detailed data.

The post How to make your data sing appeared first on Marketing Land.

Google Announces Tailored Insights, Performance Max Campaigns at Advertising Week

As part of Advertising Week, Google announced several search product updates rolling out over the next several months. The two updates we’ll focus on specifically are Tailored Insights and Performance Max campaigns.

Earlier this year Google released…

As part of Advertising Week, Google announced several search product updates rolling out over the next several months. The two updates we’ll focus on specifically are Tailored Insights and Performance Max campaigns.

Earlier this year Google released a tool called the Rising Retail Categories report to help advertisers see period-over-period snapshots of consumer behavior searching patterns. It was helpful for seeing nationwide trends across different categories, especially during a period when consumer behavior was a bit unpredictable.

Tailored Insights does exactly what the name suggests: it takes these consumer behavior trends and tailors them to your own account, making the recommendations more specific and actionable. Tailored Insights will take search trends relevant to products or keywords in your account and layer that trend data on top of the trends your related campaigns are seeing. In the example screenshot below, Google suggests where a gap might exist by comparing the increase in search demand with the increase in an advertiser’s conversions over that same period of time. Advertisers can look at more details related to that trend, including how that search or product is trending across different audiences, locations, demographics, and queries. With this tool being account-specific, it’s also integrated with recommendations that will provide budget, bid, and keyword opportunities based on the search trends identified.

Google Adweek Updates

From Google.com

Another new, interesting feature is Forecasted Trends. This provides predictions for interest that will increase in the near future, along with recommendations to take advantage of this anticipated change. There isn’t a lot of detail regarding how those trends are forecasted, but it could be a helpful in-account reminder for preparing ahead of changes in product seasonality.

We used this exact same sub-headline when blogging about the Rising Retail Categories report, and it still rings true for Tailored Insights. It’s important to dig into the trends and recommendations presented before taking action to ensure they’re relevant for your own account. A few questions to ask yourself when evaluating these trends:

1. Are the specific queries driving increased interest relevant to my business? For example, if I’m an online-only seller of large, luxury tents and some of the top trending queries are “pop-up tents” or “tents for sale near me”, there may not actually be untapped opportunity to capture.

2. Are the queries likely to have sustained interest? Increased interest in flower delivery the weeks leading up to Mother’s Day does not warrant bid pushes after Mother’s Day, for example.

3. Is my bidding method likely to naturally make these adjustments for me with my account’s performance goals in mind? Auction-time bidding systems should react to increased demand relatively quickly without any manual adjustments.

4. Will the cost required to capture incremental demand fit within my performance goals?

Performance Max campaigns will reach more Google ad inventory than any of its Smart Bidding predecessors, with eligibility to serve on the Display Network, YouTube, Gmail, Discover, Search, etc. It is meant to complement keyword-based search campaigns, and will offer various options for business goals, including online sales, new customer acquisition, and offline sales. Advertisers will provide creative and copy assets, and machine learning will optimize the campaign across channels to achieve its goals. This update isn’t surprising, given Google’s march toward machine learning and automation over the past several years (see: Responsive Search Ads, Smart Shopping Campaigns, etc.)

Performance Max campaigns are in an earlier stage than Tailored Insights, so additional information on the format is limited. Overall, it sounds like a useful option for smaller advertisers that want to take advantage of Google’s various ad types within an easily-manageable account structure. For more sophisticated advertisers, a few questions come to mind:

1. How will the interplay work between Search campaigns and Performance Max campaigns? Will keyword-based Search campaigns win out over Performance Max on queries where a keyword exists, similar to what we see with Dynamic Search Ads?

2. What visibility will advertisers get into where, when, and how Performance Max ads serve? Google’s blog post indicates that data will be available to show which creative assets and audiences perform the best, but will we get to see the search terms or websites that our ads are showing on?

It’s been an unusual year for consumers, businesses, and advertisers alike, and Google’s been busy rolling out changes to support all of those parties as they continue to navigate uncertain waters. These updates are no different. Tailored Insights is a positive addition for all advertisers, especially given the early-Q4 timing, to reach consumers based on what they need now. Performance Max campaigns could be a valuable tool for small businesses looking to expand their online presence to balance out lower in-store demand. We look forward to diving into these tools once they go live!

A/B testing is like chess

The rules of chess are easy to remember: a pawn moves one step forward, the queen can go anywhere and the end goal of the game

Hi 👋  I am Paras Chopra, founder & chairman of VWO. Hope you are finding my fortnightly posts outlining a new idea or a story on experimentation and growth useful. Here is the 4th letter.

The rules of chess are easy to remember: a pawn moves one step forward, the queen can go anywhere and the end goal of the game is to protect the king. Once you remember the rules, the game is easy to set up and fun to play.

But being easy in principle doesn’t mean it’s also easy in practice. Truly mastering chess can take decades of daily practice and requires memorizing thousands of nuances about opening moves, closing moves and opponent strategies.

A/B testing is very similar to chess in that sense.

In principle, A/B testing is simple: you have two variations, each of which gets equal traffic. You measure how they perform on various metrics. The one that performs better gets adopted permanently.

In practice, however, each word in the previous paragraph deserves a book-length treatment. Consider unpacking questions like:

1/ What is “traffic” in an experiment? 

Is it visitors, users, pageviews or something else? If it is visitors, what kinds of visitors? Should you include all visitors on the page being tested, or should you only include the visitors for whom the changes being tested are most relevant?

2/ What is “measurement” in an experiment? 

If a user landed on your page and does not convert, when do you mark it as non-conversion? What if the user converts after you’ve marked it as non-conversion? How do you accommodate refunds? If different user groups have markedly different conversion behavior, does it even make sense to group them during measurement? If you group them, how do you deal with Simpson’s paradox[1]?

3/ What types of “various metrics” should you measure? 

Should you have one metric to measure performance of variations or should you have multiple? If you’re measuring revenue, should you measure average revenue per visitor, average revenue per conversion, 90th percentile revenue, frequency of revenue, or all of them? Should you remove outliers from your data or not?

4/ What does “perform better” mean? 

Is 95% statistical significance good enough? What if it is 94%? What if the new variation is not performing significantly better but feels it should? Do you take a bet on those? What if one metric improved but another that should have improved as well actually became worse? How real is Tyman’s law which states that extreme improvements are usually due to instrumentation effort?

For the skeptic, these questions may seem like a needless pedantic exercise. But, without rigor, why bother doing A/B testing in the first place?

Nobody likes their ideas and efforts go to waste, so we latch onto any glimmer of success we see in our A/B tests. It’s relatively easy to get successful A/B tests because it presents many avenues for mis-interpretation to a motivated seeker. It’s only human to be biased.

But because of this lack of rigor in A/B testing, many organizations that get spectacular results from their A/B tests fail to see an impact on their business. Contrast this with organizations who take their experimentation seriously: Booking.com, AirBnB, Microsoft, Netflix and many other such companies with a culture of experimentation know that getting good at A/B testing takes deliberate commitment.

So, next time someone tells you that A/B testing doesn’t work, remind yourself that it’s like saying chess is a boring game just because you’re not good at it.

If you enjoyed reading my letter, do send me a note with your thoughts at paras@vwo.com. I read and reply to all emails 🙂