Online glossary of A/B testing terms and abbreviations

We are happy to present a brand new addition to our website: a comprehensive A/B testing glossary containing terms and abbreviations used testing as part of conversion rate optimization (CRO).  Definitions start from very basic things such as “A/…

We are happy to present a brand new addition to our website: a comprehensive A/B testing glossary containing terms and abbreviations used testing as part of conversion rate optimization (CRO).  Definitions start from very basic things such as “A/B test“, “mean“, “conversion rate” and “revenue per user“, go through “hypothesis“, “null hypothesis“, “standard deviation“, “p-value” […] Read More...

Mobile Video Optimization And Its Impact On Conversions

Mobile video optimization isn’t only about making videos play smoothly on smartphones of different screen sizes. Popular video hosting sites can help you to that end. Vimeo and Wistia even offer responsive embedded code so that you can upload videos on your landing pages or blogs without worrying about the container size. Even if your […]

The post Mobile Video Optimization And Its Impact On Conversions appeared first on Blog.

Mobile video optimization isn’t only about making videos play smoothly on smartphones of different screen sizes. Popular video hosting sites can help you to that end.

Vimeo and Wistia even offer responsive embedded code so that you can upload videos on your landing pages or blogs without worrying about the container size.

Even if your website is responsive, embedded videos with fixed width can give your visitors an unpleasant experience. And studies say that half of your users are less likely to engage with you if you give them a bad mobile experience. So, the next time you are planning to embed a video, go for responsive instead of “fixed width.”

Optimizing videos for mobile can be tricky. This article will not just help you fit your videos within the screen of a mobile device, but also help you improve these videos to increase conversions.

We have more video optimization hacks laid out for you below. Dig in.

And note that making videos playable on mobile is not your end goal. What matters more is conversion. Why are we even paying so much attention to mobile?

Mobile Video Optimization: Reasons

The reason we want you to be serious about mobile video optimization is because of these 2 stats:

1. Mobiles give you a wider reach compared to desktops. A study indicates that there are more mobile users now than desktop users.

Source

2. By 2021, 75% of all mobile traffic will come from video content.

Per these 2 points, you get the idea that mobile browsing is on the rise and people like to watch videos, more on the smartphone than on their desktop. Ergo thinking out a mobile-first strategy is crucial for your video marketing success.

After all, it’s much easier to watch a video on your phone. And the cherry? 92% of mobile video consumers share videos with others. That means easy marketing for you—higher views, engagement, and click-throughs!

Let us now move to how videos can be optimized to drive conversions on mobile devices:

Optimization Tip 1: A/B Test Vertical Video Ads

Most videos are designed to play in the landscape orientation. But let’s face it—we hold our phones vertically 94% of the time. So, it can be a hassle to flip your phone just to watch a video and then flip it back. Sounds like a waste of time, right? Many marketers thought so and are now A/B testing their ads with vertical videos.

Source

Vertical videos are popping up as in-app ads too. So far, we have heard a lot of success stories about use of vertical videos.

Chartboost adopted the vertical video ad format, and reported that their advertisers saw up to 20% lift in install per thousand impressions (IPM). That’s great, right?

Even a study from Facebook saw people preferring vertical video content:
– 79% of the novice vertical video consumers were in favor of the vertical video format.
– 65% of respondents applauded brands that are using vertical video for their advertising as “more innovative.”

So, prepare to contribute to this brave new world of vertical video content.

Optimization Tip 2: Use Native Video Uploads to Get More Views

Natively uploaded videos play automatically while you need to click to play videos that have been linked with other platforms. Facebook reports imply that you can achieve as much as 1055.41% higher average share rate with native videos compared to YouTube third-party video links.

So, don’t be a stranger to this native video tactic.

In one of the expert interviews, Matthew Vazquez also asserted the importance of uploading your video separately on YouTube, Vimeo, and Facebook each, and to use modified descriptions with strategic keyword density for all 3 uploads. Matthew says that “this is powerful when done right because now you’ve 3X your SEO potential.”

Source

Optimization Tip 3: Ensure That Your Landing Page Video Is Mobile-Compatible

Conversion prophets have revealed that adding an explainer video on your landing page can boost your conversion rate by
up to 23%.

Source

One of the best landing pages using an explainer video as the conversion bait was that of Dropbox. They had kept their landing page simple with one engaging video and a download button. Visitors watched the video, saw the benefit of using Dropbox, and proceeded to download it. It was a simple funnel, and the conversion rate was high. Reports say that Dropbox earned a million users and bagged a revenue of $48 million.

There are a few problems when it comes to adding a video to your landing page:

– To begin with, ensure that your landing page video is responsive. As we shared in the beginning, you can use Wistia or Vimeo’s responsive embedded code to get this done. These video-hosting platforms offer incredible analytics to help you monitor your video performance. You will get cool insights, such as at which point your viewers are dropping off, and will be able to use these to optimize the playback accordingly.
– Use a thumbnail that prompts visitors to play the video. Never put your landing page video on autoplay. That’s a no-no.
– Keep your video short; 60 to 90 seconds is the best. (Stats say 59% of viewers will watch your video to the end if it’s under a minute.) The idea is to ensure that your video isn’t too heavy and that it shouldn’t lag.
– Position the Call to Action (CTA) button next to the video. Also, ensure that the narrator ends the video with a verbal CTA message, or use text to highlight CTA on the end screen. Try doing both as well.
– Try user testing to see how your target audience interact with the video. Check if they click the video right away or if they are distracted by some other elements on your landing page.

Select these probable issues before spending on PPC campaigns and ads. After that’s done, you will have a landing page with a video that can get you the ROI.

Optimization Tip 4: Ensure that your video has a call to action at the end

Remember those “Please subscribe to our channel.” requests that video makers leave with at the end of the video? These work.

If you watch a video till the end, that means you already like it. So when the creator politely asks you to subscribe, there’s a good chance you would do it.

Call to actions are, therefore, important.

These instruct your users on the next course of action—what they should do after watching the video. So, don’t just put up your CTA message or link in the video description. Say it. Have the narrator of the video conclude your video with the call to action message.

You can also use the actor in the video to point to the CTA button at the end. (If it’s an animation video, use a hand illustration or directional cues.) You will notice that many YouTube video creators use this tactic to request the viewers to subscribe.

There’s another CTA hack. If you are using YouTube, it is its annotation and card features. You can also use these to pop up your CTA link on the screen itself. For mobile users, that makes navigation easier.

Source

Optimization Tip 5: Use typography or subtitles to get a reaction

Your videos should make sense even when muted.

With platforms like Facebook and Twitter having the muted autoplay feature, you are bound to have viewers who will look at your video for a few seconds to determine if it’s worth watching. This means that you can’t afford to have videos that rely only on audio and narration.

Your video should make sense even without the audio or at least provide a context of what’s being presented. Mute your video and see if the idea is being conveyed even without the audio, or if the visual is powerful enough to make the viewers turn the volume up or put on their earphones (if they are at a public place).

Best way—try adding captions or subtitles or use typography animation. Either will help you grab viewer attention, engage them even with a muted video, and get a reaction.

Conclusion

Making your video play on mobile devices is not your end goal. As a marketer and business owner, what matters more is conversion. By using the above optimization hacks, you can get your videos to perform better on mobile.

Just remember the distinct phases of your buyer’s journey. A video designed for customers in the awareness phase may not be appealing to your audience in the evaluation phase or those hesitating at that purchase point.

So, create several types of videos to power your customer’s decision journey.

Conduct a survey acquiring information about your demographics if you want to be painfully precise; but for the most part, develop your content such that it is not inhibited by a small screen. Your videos should have details clearly presented so that these may not get missed out if viewed on a small screen.

This is just the tip of the iceberg. If you have some cool video optimization tips, please share your story in the comments below.

The post Mobile Video Optimization And Its Impact On Conversions appeared first on Blog.

How To Build A Culture Of Experimentation

It’s one thing to run an A/B test correctly and get a meaningful uplift. It’s another thing entirely to transform your organization into one that cares and respects experimentation. This is the goal, though. You can only shake off so much additional revenue if you’re the only rogue CRO at your company. When everyone is […]

The post How To Build A Culture Of Experimentation appeared first on Blog.

It’s one thing to run an A/B test correctly and get a meaningful uplift. It’s another thing entirely to transform your organization into one that cares and respects experimentation.

This is the goal, though. You can only shake off so much additional revenue if you’re the only rogue CRO at your company. When everyone is involved in the game, that’s when you stride past the competition.

It’s not just about the tools you use, or even the skills, but also about the people involved. But organizational matters tend to be a bit complex, as anything that involves humans is. How do you build a culture of experimentation?

This article will outline 9 tips for doing so.

1. Get the Stakeholders Buy into CRO, and Establish Program Principles

First thing’s first—we need to get everybody on the same page.

It used to be more difficult to convince people of the value of conversion optimization. Now, it seems that it is more mainstream, and most people buy into the benefits.

We know from conducting State of the Industry Report that CRO is being more widely adopted, and those that are adopting it are increasingly establishing systems and guidelines for their program. All of this is good.

If you’re just getting started on your CRO journey, though, don’t fret. There are some simple and tactical ways you can start establishing a vision.

First, if you don’t have full buy-in from stakeholders, make sure you have at least one influential executive sponsor who is on your side. If you don’t have this, you won’t go far. (Programs tend to have a substantial ramp-up period before you see a good return.)

Second, write down up front, your program principles and guidelines. I like to create a “principles” document for any team I’m on (and a personal one as well), just so that we know what our operating principles are and we know how to make decisions when things are ambiguous.

Here’s an example of a principles document from my team at HubSpot (just a small section of it, but you would get the point):

Of course, we have tons of documentation from everything on how we run experiments to our goals, and more.

Andrew Anderson gave a great example of his CRO program principles in a CXL blog post:

  • All test ideas are fungible.
  • More tests does not equal more money.
  • It is always about efficiency.
  • Discovery is a part of efficiency.
  • Type 1 errors are the worst possible outcome.
  • Don’t target just for the sake of it.
  • The least efficient part of optimization is the people (with you also included).

Yours could look completely different, but just make sure you up-front script the critical plays and don’t leave any questions hanging in the air. This will help stakeholders understand what you are up to and will also help onboard new employees in your team when they get started.

2. Embrace the Power of “I Don’t Know”

With most marketing efforts, we expect a linear model. We expect that for X effort or money we put into something, we should receive Y as the output (where Y > X).

Experimentation is somewhat different. It may be more valuable to think of experimentation as building a portfolio of investments, as opposed to a machine with a predictable output (like how you’d view SEO or PPC).

According to almost every reputable source, many tests are going to fail. You’re not going to be right. Your idea is not going to outperform the control.

This is okay.

If, for every 5 tests that fail, you get 1 true winner, you’re probably already ahead. That’s because, on the 5 tests that failed to improve conversion rates, you are only “losing” money during the test period. You didn’t set them live for good, so you mitigated the risk of a suboptimal decision. (This alone is a great benefit!)

Outside of that, the one test that did win should add some compounding value over time. A 5% lift here and a 2% lift there add up; and eventually, you’ve got the rolling equivalent of a portfolio with compounding returns:

Source

A side point to the whole “embrace I don’t know” thing is that you shouldn’t seek to test things to validate only what you think is right. The best possible case is that something wins that you didn’t think would win.

That’s how Andrew Anderson frequently frames conversion optimization, saying in this post that “the truth is, in optimization, the more often we prove our own perceptions wrong, the better the results we are getting.”

Ronny Kohavi, too, makes the point that a valuable experiment is when the “absolute value of delta between expected outcome and actual outcome is large.” In other words, if you thought it would win and it wins, you haven’t learned much.

Source

3. Make It a Game

Humans like competition; competition and other elements of gamification can help increase engagement and true interest in experimentation.

How can you gamify your experimentation process? Some tools, such as GrowthHackers’ NorthStar, embed this competition right into the product with features like a leaderboard:

Source

You can create leaderboards for ideas submitted, experiments run, or even the win rate of experiments. Though, as with any choice in metric, be careful of unintended incentives.

For example, if you create a leaderboard for the win rate, it might be possible that people are disincentivized from trying out crazy, creative ideas. It’s not a certainty, but keep an eye on your operational metrics and what behaviors they encourage.

4. Adopt the Vernacular

Sometimes, a culture can be shifted by subtle uses of language.

How does your company explain strategic decision making? How do you talk about ideas? How do you propose new tactics? What words do you use?

If you’re like many companies, you talk about what is “right” or “wrong,” what you have done in the past, or what you think will work. All of this, of course, is nourishing for the hungry, hungry HiPPO (who loves talking about expert opinion).

What if, instead, you talked in terms of upside, risk mitigation, experimentation, and cost versus opportunity instead?

The world sort of opens up for those interested in experimentation. Obviously, you still have to be grounded in reality. You can’t throw insane test ideas at the wall and hope that everyone jumps on board.

But if you can propose your ideas in the context of a “what if,” something you can test out with an A/B test rather quickly, you can probably get people on board.

We see here that 40% of our users are dropping off at this stage of the funnel. We’ve done a small amount of user research and have found that our web form is probably too long. It would take us very little time to code up X, Y, and Z variants, and we’d have a definitive answer in 4 weeks. The upside is big. The risk is low. Let’s run the experiment?

It’s much harder to argue against something like this.

Most of persuasion is framing. If the person you are trying to convince feels attacked or threatened (“you think a scrappy A/B test is better than my 25 years of experience?!”), you’re not going to get far.

If you pull people into the ideation process and propose ideas as experiments with lots of upside, it’s easier to get people involved in the process. Or maybe just start throwing the words “hypothesis,” “experiment,” “statistically significant,” “risk mitigation,” and “uncertainty reduction” into all of your conversations, and hope that people follow along.

It doesn’t need to be limited to experimentation, either. You can make it normal to talk about pulling the data, cohort analyses, user research, and others. These should be normal processes for decision making that replace gut feel and opinions.

5. Evangelize Your Wins

It’s important to stop and take the time to smell the roses. When you win, celebrate! And make sure that others know about it.

It’s through this process of evangelization that you both cement the impact and results you’re creating in others’ minds as well as recruit others to become interested in running their own experiments.

How do you evangelize your wins? Many ways:

  • Have a company Wiki? Write your experiments there!
  • Send a weekly email including a roundup of the experiments.
  • Schedule a weekly experiment readout that anyone can attend.
  • If possible, write external case studies on your blog. This isn’t always possible, but can be a great way to recruit interesting candidates to your program.

I’m sure there are many other interesting and creative ways to celebrate and evangelize wins as well. Make sure to comment in the end on how your company does it.

6. Define Your Experiment Workflow/Protocol

If you want everyone to get involved with experimentation, make sure that everyone understands the rules. How does someone set up a test? Do they need to work with a centralized specialist team or can they just run it themselves? Do they need to pull development resources? If so, from where?

These all are questions that can cause hesitation, especially for new employees; and this hesitation can really hinder the pace of experimentation throughput.

This is why it’s so beneficial to have someone, or a team, owning the experimentation process.

Even if you don’t have someone in charge of the program, though, you can still build out the documentation and protocol. At the very least, you can create an “experimentation checklist” or FAQ that can answer the most common questions.

In Switch, Chip and Dan Heath wrote:

“To spark movement in a new direction, you need to provide crystal-clear guidance. That’s why scripting is important – you’ve got to think about the specific behavior that you’d want to see in a tough moment, whether the tough moment takes place in a Brazilian railroad system or late at night in your own snack-packed pantry.”

“Clarity dissolves resistance,” they say.

Luckily, there are tons of examples of testing guidelines, frameworks, and rules out there to borrow from.

7. Invest in Ongoing Education and Growth Opportunities

This is anecdotal; but I’ve found that the best organizations, those that run very mature experimentation programs, tend to invest heavily in employee development.

That means granting generous education stipends for conferences, books, courses, and internal trainings.

Different companies can have different protocols as well. Airbnb, for example, sends everyone through data school when they start at the company. HubSpot gives you a generous education allowance.

There are tons of great CRO specific programs out there nowadays, specifically through CXL Institute. Some programs I think everyone should run through:

8. Embed Subtle Triggers in Your Organization

I’ve found one of the most powerful forces in an organization is inertia. It’s exponentially harder to get people to use a new system or program than it is to incorporate new elements into the current system.

So what systems can you use to inject triggers that inspire experimentation?

For one, if you use Slack, this is certainly easy. Most products integrate with Slack—Airtable, Trello, GrowthHackers Northstar, and others—so you can easily set up notifications to appear when someone creates a test idea or launches a test.

Just seeing these messages can nudge others to contribute often. It makes the program salient overall.

Whatever triggers you can embed in your current ecosystem—even better if they’re automated—can be used to help nudge people toward contributing more test ideas and experiment throughput.

9. Remove Roadblocks

According to the Fogg Behavioral model, there are 3 components that factor into someone taking an action:

  • Motivation
  • Ability
  • Prompts/Triggers

Source

I think the ability, or the ease at which someone can accomplish something, is a lever that we tend to forget about.

Sure, you can wow stakeholders with potential uplifts and revenue projects. You can embed triggers in your organization through Slack notifications and weekly meetings so that people don’t forget about the program. But what about making it easier for everyone who wants to run a test?

That’s the approach Booking.com seems to have taken, at least according to this paper they wrote on democratizing experimentation.

Some of their tips include:

  • Establish safeguards.
  • Make sure data is trustworthy.
  • Keep a knowledge base of test results.

To summarize, do everything you can to onboard new experimenters and mitigate their potential to mess up experiments. Of course, everyone has to go through the beginner phase of A/B testing, where they’re expected to mess things up more often than not. The trick, however, is to make things less intimidating while also making it less likely that the newbie may drastically mess up the site.

If you can do that, you’ll soon have an excited crowd anxiously waiting to run their own experiments.

Conclusion

An organization with a mature testing program knows that almost all of it is dependent on a nourishing experimentation culture. One cannot operate, at scale and truly efficiently, with only one or a handful of rogue experimenters.

The program needs to be propped up by influential executive stakeholders; and everyone in the company needs to buy into the basic process of making evidence-based decisions by using research and experiments.

This article outlines some ideas I’ve seen to be effective in establishing a culture of experimentation, though it’s clearly context dependent and not limited to the items on this list.

Got any cool ideas for implementing a culture of experimentation? Make sure you let me know!

The post How To Build A Culture Of Experimentation appeared first on Blog.

The A/B Testing Guide to Surviving on a Deserted Island

The secluded and isolated deserted island setting has been used as the stage for many hypothetical explanations in economics and philosophy with the scarcity of things that can be developed as resources being a central feature. Scarcity and the need to…

The secluded and isolated deserted island setting has been used as the stage for many hypothetical explanations in economics and philosophy with the scarcity of things that can be developed as resources being a central feature. Scarcity and the need to keep risk low while aiming to improve one’s situation is what make it a […] Read More...

Inherent costs of A/B testing: limited risk results in limited gains

I’ve already done a detailed breakdown of costs & benefits in A/B testing as well as the risks and rewards and how A/B testing is essentially a risk management solution. In this short installment I’d like to focus on the trade-off betwe…

I’ve already done a detailed breakdown of costs & benefits in A/B testing as well as the risks and rewards and how A/B testing is essentially a risk management solution. In this short installment I’d like to focus on the trade-off between limiting the downside and restricting the upside which is present in all risk management […] Read More...

Results From Our Latest A/B Test: Here’s The New VWO Logo!

Over the past 8 years, we’ve made some key (and some minor) changes to the look and feel of our brand. Around this time last year, we revamped our website for the launch of VWO Conversion Optimization Platform™. As an organization that thrives on a culture of experimentation, we are always looking into data to […]

The post Results From Our Latest A/B Test: Here’s The New VWO Logo! appeared first on Blog.

Over the past 8 years, we’ve made some key (and some minor) changes to the look and feel of our brand. Around this time last year, we revamped our website for the launch of VWO Conversion Optimization Platform™.

As an organization that thrives on a culture of experimentation, we are always looking into data to discover insights for optimization. By turning our opinions into hypotheses, we test changes for almost everything which could have a significant impact on the business, and then derive the next logical step. Based on this simple framework, we recently made a minor change to the VWO logo. Before we delve further into the hypothesis behind this change, look at the logo in its full glory:

The Hypothesis: Making The Letters V, W, and O Prominent Will Improve Readability

In the beginning, our product was called Visual Website Optimizer. However, over the years, people (including us) fondly started abbreviating it to VWO. This is what the VWO logo looked like during this gradual change:

More recently, we dropped the accompanying text “Visual Website Optimizer” completely, and also started referring to our product as just “VWO.”

With this change, we realized that it would be hard for someone unfamiliar with our brand to read or understand our logo. We hypothesized that if the letters “V,” “W,” and “O” were made distinguishable, the brand name VWO would stand out more clearly.

The Test: Conducting an A/B/C Test to Choose a Winner

After the hypothesis was finalized, our design team created a new variation of the logo, per the new specification. Next, we decided to test the hypothesis by conducting extensive user testing through 5-second tests on UsabilityHub.

Five-second tests are a method of usability testing, where the participants are shown a visual for only 5 seconds, and then asked questions corresponding to it.

For our tests, we selected a sample of participants from across the globe, with varying demographics, location, and other attributes. They were showed the 3 variations of the logo—the existing one, the proposed one, and the one with VWO written as well-spaced plain text. Next, we asked the participants the question “What do you read?” to which they had to type in a response.

For the proposed logo, we got 90% of them answering “VWO”, as opposed to only 66% for the existing one. For the variation with VWO written as well-spaced text, the response was around 96%.

The Result: Reinforced Belief in the Potential of Testing

As an obvious next step, we decided to make this minor update to our logo which can now be seen to be live across all our digital properties. We’re proud of the fact that the basic tenets of experimentation continue to give direction to our efforts.

If it wasn’t for validating our initial, seemingly insignificant hypothesis, VWO wouldn’t have got a brand new identity. We strive to uphold this culture in our organization for the years to come.

What do you think of our new logo? Feel free to share your thoughts in the comments below.

The post Results From Our Latest A/B Test: Here’s The New VWO Logo! appeared first on Blog.

Representative samples and generalizability of A/B testing results

I see a nice trend in recent discussions on A/B testing: more and more people realize the need for proper statistical design and analysis which is a topic I hold dear as I’ve written dozens of articles and a few white-papers on. However, there ar…

I see a nice trend in recent discussions on A/B testing: more and more people realize the need for proper statistical design and analysis which is a topic I hold dear as I’ve written dozens of articles and a few white-papers on. However, there are cases in which statistical validity is discussed without consideration for […] Read More...

Designing successful A/B tests in Email Marketing

The process of A/B testing (a.k.a. online controlled experiments) is well-established in conversion rate optimization for all kinds of online properties and is widely used by e-commerce websites. On this blog I have already written in depth about the s…

The process of A/B testing (a.k.a. online controlled experiments) is well-established in conversion rate optimization for all kinds of online properties and is widely used by e-commerce websites. On this blog I have already written in depth about the statistics involved as well as the ROI calculations in terms of balancing risk and reward for […] Read More...

46 Conversion Rate Optimization Hacks

Conversion is the ultimate goal of a business’s online efforts. If your website has a glorious design and drives huge traffic but you’re still not getting enough leads, you need to get serious about conversion rate optimization and these 46 conversion …

Conversion is the ultimate goal of a business’s online efforts. If your website has a glorious design and drives huge traffic but you’re still not getting enough leads, you need to get serious about conversion rate optimization and these 46 conversion rate optimization hacks will help you get there. Conversion rate optimization is a systematic process...

The post 46 Conversion Rate Optimization Hacks appeared first on Conversion Sciences.

It’s The Little Things That Count With CRO

Big, bold, radical, redesign tests always move the needle the most. That is if you know your market, really really well.  But through the years, I’ve found that it’s the little things that count with CRO.  Even today I’m excited every time …

Big, bold, radical, redesign tests always move the needle the most. That is if you know your market, really really well.  But through the years, I’ve found that it’s the little things that count with CRO.  Even today I’m excited every time I find an interesting case study that highlights something that you wouldn’t have likely...

The post It’s The Little Things That Count With CRO appeared first on Conversion Sciences.