A/B Testing Website Copy With GPT-3: Opening New Doors for Experimentation Using AI

From initially helping humans out with redundant and manual tasks, to now mastering creative jobs such as making original art or composing music, AI has evolved and transformed in unprecedented ways. One such creative job that bots are surprisingly good at is writing copy! Yes, GPT-3 (Generative Pre-trained Transformer 3) is a neural-network powered AI…

From initially helping humans out with redundant and manual tasks, to now mastering creative jobs such as making original art or composing music, AI has evolved and transformed in unprecedented ways. One such creative job that bots are surprisingly good at is writing copy! Yes, GPT-3 (Generative Pre-trained Transformer 3) is a neural-network powered AI that can produce nearly flawless text relevant to the given context. Built by OpenAI[1], a San Francisco-based research lab, GPT-3 is a third-generation powerful language generator that uses machine learning to predict and produce text, almost like a human. 

If you expand on GPT, here’s what it denotes: 

Generative: Indicating that the goal of the model is to generate text by predicting one word at a time in a given sentence. 

Pre-trained: Indicating that a huge amount of data has been fed into the system to train it.

Transformer: Indicating the algorithm used by the AI model, which specializes in natural language processing, i.e., how words are used in a language and what they mean. 

AB Testing And GPT3

Once GPT-3 is fed a prompt, it generates streams of text by predicting the possibility of a sentence existing in this world. Currently, the functionality is in beta and only offered to a select group (including VWO) through an API accessible via the cloud. 

Let’s face it – copywriting is no easy task. GPT-3’s robust and flexible language model can produce short copy at scale. If you add to that the ability to test copy versions, you can get the best of both worlds. Also, some of the most commonly run tests revolve around webpage copy. So, integrating Open AI’s GPT-3 API with VWO Testing was the most natural and logical thing for us to do.

Our new feature enables you to use AI-generated copy to create variations for your website copy and deploy them without any help from IT. You can also test the AI-generated copy against the original human-written copy on your website. The next section covers how popular brands uncovered the practical implications of our new feature via a friendly contest between human-written and AI-generated copy. 

VWO’s Human vs. AI competition

In August this year, VWO hosted a friendly competition between copy written by human copywriters and that by our new feature powered by OpenAI’s GPT-3 API. We invited participants from all over the world and tested AI-generated text against human-written one for their webpages with sufficient traffic via VWO or any other testing platform they were using. 

Over 450 brands were given access to the AI copy generating feature during the course of this competition. Among the 18 shortlisted participants were Booking.com, Clark Germany GmbH, and Schneiders, to name a few. The AI feature was able to generate copies in various languages such as Spanish, German, Portuguese, etc. The participants were highly satisfied with the accuracy of the output in these languages.

All participants had to set up their tests keeping the original website copy as the control and the AI-generated one(s) as the variation. 

Results of the competition

Among the 18 tests run by the confirmed participants, 1 had an existing (or new) human written copy as the winner, 3 had AI copy as the winner, 3 were declared as a tie, 2 are still awaiting results, and 9 were inconclusive. 

Let’s take a look at some of the tests where the AI-generated copy won:

Schneiders [An eCommerce store for horse wear & equipment]

The team tested their topmost banner copy by creating a variation of the original page using the AI-backed language generator. Here’s a look at the control and variation from the test:

Human Written Copy In Ab Test On Schneiders
Control [Human-written copy]
Gpt 3 Ai Written Copy In Ab Test On Schneiders
Variation [AI-generated copy]

Once statistically significant results were achieved, the A/B test declared the variation to be the winner as it led to a 7.06% uplift in their banner clicks.

Clark Germany GmbH [An insurance agency based out of Frankfurt]

3 variations of the page headline were created using the AI copy and pit against the control. The test was run for 48 days. Following are the control and variations of the test: 

Human Written Copy In Ab Test On Clark Germany
Control [Human-written copy]
English translation: Manage Your Insurance Digitally
Gpt 3 Ai Written Copy In Ab Test On Clark Germany
Variation 1 [AI copy]
English Translation: Manage Your Insurance Digitally
Variation 2 Of The Gpt 3 Ai Written Copy In Ab Test On Clark Germany
Variation 2 [AI copy]
English Translation: Keep Track Of Your Insurances
Variation 3 Of The Gpt 3 Ai Written Copy In Ab Test On Clark Germany
Variation 3 [AI copy]
English Translation: Clark Is Your Digital Insurance Manager

Once the test reached conclusion (statistical significance >  90%), all 3 variations outperformed the control. Variation 2 resulted in the maximum uplift in their CTA clicks (15.77%), while Variation 1 and 3 resulted in an uplift of 9.13% & 7.13%, respectively.

Here’s the test that declared the human copy as the winner:

Booking.com [A global travel company]

The team at Booking.com tested the CTA on their hotel booking pages. 2 human-written copies were pitted against an AI-generated one. Following are the variations they created:

Human Written Copy In Ab Test On Booking Com
Variation 1 [Human-written copy #1]
Gpt 3 Ai Written Copy In Ab Test On Booking Com
Variation 2 [Human-written copy #2]
Variation Of The Gpt 3 Ai Written Copy In Ab Test On Booking Com
Variation 3 [AI-generated copy]

The human copy #1 won the test as it resulted in a 1.7% uplift in the CTA conversion rate.

Here’s a test that resulted in a human-AI tie:

Springworks [A SaaS company based out of India]

The team at Springworks tested their landing page headline by creating a variation using the AI-generated copy and pitting it against the original (control). Their goal was to improve clicks on the ‘Add Trivia’ CTA. The test was run for 8 days. Here’s a look at the control and variation:

Human Written Copy In Ab Test On Springworks
Control [Human-written copy]
Gpt 3 Ai Written Copy In Ab Test On Springworks
Variation [AI copy]

Since the difference in the uplift in CTA clicks between the control and variation was less than 5%, and the test results were statistically insignificant, the test was declared to be a tie. 

Let’s deep dive into the nitty-gritty of how VWO Testing and GPT-3 work together.

VWO Testing & GPT-3

We integrated Open AI’s GPT-3 API with our visual editor so that every time you decide to run a test or deploy a change, you can generate copy recommendations that you can choose to create variations out of. This means you get to cut down on time spent brainstorming on variations and alternatives by having a library of AI-generated ideas readily available at your disposal. 

Whether you are looking to optimize headlines, CTA text, product descriptions, or any other text on your site, you can quickly generate alternatives and either directly deploy them or test them against your original copy, both without any developer help. Either way, by automating this aspect of experimentation, you get to make your CRO program more efficient and agile. 

Once you open VWO’s visual editor and click on any piece of text, you will find a ‘Suggest Variations’ option in the drop-down menu. Clicking on it will display a bunch of AI-powered copy suggestions (based on the existing copy) that you can choose from.

Sounds too good to be true? Sign up for a free trial by VWO and assess the GPT-3 feature for yourself.

What the future holds for GPT-3, automated copywriting, and testing

Experts have conflicting views around the scope of GPT-3 and the extent to which humans can leverage it to automate copywriting. While some feel that the model can be trained to mimic and replace human written-copy, others argue that it lacks the ability to construct cohesive sentences, use reasoning or logic constructively, or build a narrative – something you can only expect from a human copywriter. 

It is hard to anticipate everything that might happen. We don’t think we can get everything right, certainly not up front. Still, it’s better to play around with this type of technology now while it can still be controlled and learn lessons to be applied as AI gets ever more powerful

Greg Brockman, Co-Founder & CTO, OpenAI

Whether automated copywriting will be a norm in the future is something we are yet to figure out as we explore GPT-3’s full potential. However, what we know for sure is that this innovation is going to be revolutionary when it comes to copy experimentation. 

With your new AI partner, you get to reduce the time spent on manual work as well as iterations with a copywriter. As you generate AI copy on demand within seconds and make real-time quick-fixes on your website, you can create short-form content at scale and thus take a giant leap towards increasing your experimentation velocity and evolving your CRO program


The power of GPT-3 in copy driven optimization is immense, and we’ve only touched the tip of the iceberg so far. However, some restraint is advised because we cannot equate it with the intelligence of human copywriters – not yet, at least. The real value, at least for now, lies in being able to effectively test out copy variations, while reducing the back and forth with copywriters and developers. The good thing is we can keep leveraging the power of GPT-3 to run better and faster experiments using platforms like VWO.

10 CRO Best Practices to Uplift Your Optimization Journey

Change is the only constant.  If your organization’s experimentation culture assumes this to be true, you are on the right track (and in good hands). Agility in your optimization efforts enables you to push and explore new boundaries with every experiment, which can get you nailing your CRO game.  But there is a catch! You…

Change is the only constant. 

If your organization’s experimentation culture assumes this to be true, you are on the right track (and in good hands). Agility in your optimization efforts enables you to push and explore new boundaries with every experiment, which can get you nailing your CRO game. 

But there is a catch! You cannot define CRO in one dimension. For instance, there are a number of approaches to optimize your homepage—from improving the UX to making the copy more persuasive. However, there is no single practice that will guarantee immediate results (and testing every practice that you adopt is a great habit indeed!)

CRO image best practices

In this blog post, we have shared a few common and not so common practices and expert tips to fetch you wins in your optimization efforts. Do bear in mind that your focus should be on making these best practices a constant in your strategy, and you should strive to make the most of every impression, every click, and every visitor.

Know your visitors to identify leaks in your funnel

It’s tough to coax people into taking actions that translate into a business goal when you don’t know them. Delighting them is even harder. Study your visitors’ interests, requirements, demographics, and behavior on your site well to know who they are before setting up a CRO strategy for your conversion funnel.

  • Utilize visitor behavior analysis tools such as heatmaps, session recordings, and click maps to understand their hesitation on the pages critical to your business. You might discover usability issues, which usually sit in the blind spot.
  • Run enter and exit surveys on your landing pages to dig deeper into your visitors’ psyche and identify leaks in your website that can be fixed. For example, you may trigger an NPS survey for your power users upon entry, and exit surveys to ask drop-offs the reasons they are not moving ahead in your funnel. The reasons could be many, ranging from pricing and value proposition to features offered. 

Have well-defined goals and specific hypotheses

Your goals and hypotheses for an experiment should be backed by data from your qualitative and quantitative research. Don’t fall for the ‘obvious improvements’ and ‘no brainer’ assumption traps while experimenting. Without a clearly defined hypothesis, you would not know what you are testing, why you are testing, and how to interpret the results. Goals should pave the way for your optimization journey to become better with every experiment.

Prioritize your CRO roadmap

Shoot-from-the-hip guesswork leads to random testing, which in turn leads to sluggish conversions. Without a proper roadmap in place, you end up testing every other page every month and end up deriving inconclusive results. These results neither have a clear conclusion on your visitors’ behavior, nor do they contribute to conversions. 

You must have a prioritized CRO roadmap planned, well in advance, for your experimentation efforts to yield fruitful results. 

  • Pick up the bolder, impactful, and targeted tests first to gain larger returns in the shortest time.
  • Prioritize easy to implement tests that promise a high financial impact.
Screenshot Of A Cro Roadmap
Example of a CRO roadmap

Measure micro conversions

Every step in your conversion funnel has a conversion goal. You can imagine conversions as a spectrum. On the right-hand side, you have the main goals that directly impact your business’s net profit, while on the left side, you have intermediate metrics such as the number of sign-ups your button generated or the click-through rate (CTR). These metrics, also known as micro conversions, contribute to the main conversion goal at every stage of your conversion funnel. Hence, it’s a good practice to measure them. 

If your website does not drive colossal traffic, make use of these micro conversion metrics for your subsequent experiments as they are much larger in number and can be instantly measured.

Don’t tweak your running experiment

Many testing softwares allow you to stop your running test so that you can make the changes suggested by early test data. However, CRO experts suggest that this should be avoided as it harms the data.

Ensure that you keep running your tests until they reach a statistical significance of 95%. You can calculate your test duration to estimate how long you should run a test, and also check whether your results are significant, using readily available online tools. 

QA your experiment

It is a good practice (read:hygiene) to perform a quality check on your experiment before running it. Ensure that your goals are defined and are getting tracked correctly in your testing software. It is also important that the test is being rendered correctly for your targeted audience, across browsers, devices, etc. 

Design: Prefer a Stanley hammer any day 

Always optimize your product’s core functionality—a Stanley hammer is designed for hammering. You can optimize it for hammering different kinds of nails into different materials. However, the last thing you should be optimizing it for is looks and aesthetics (unless you are a home decor brand!). 

Stanley Hammer
An example of a Stanley hammer having a core functionality of easy, one-handed nail placement.

Similarly, your product’s functionality should be the DNA of your web page design. Aesthetics are not a substitute for testing and research. So focus on the functionality of your product/services and optimize elements that speak for your core. Marketers can fall for competitor’s designs and get influenced as design impacts your emotions. 

If you already have a high-converting landing page and you still want to beautify it, ensure that your designs follow your brand style, are easy to comprehend for your audience, and easy to update. 

Document and extrapolate your learnings

Make sure that you document your learnings from the test. These learnings can be about setting up the test (so you don’t repeat errors), the hypotheses and corresponding observations, targeting, QA process and steps taken, final metrics of the test, deep-dive into segments and other qualitative knowledge gathered, etc.

Try to apply your learnings further—from getting better at testing to even using the winner attributes in other strategies, channels, segments, etc.

‘Zoom out’ to have your next test planned

CRO is an iterative process. Every test brings with it a set of learnings. Successful ones that lead to high conversion rates give your competitors a tough time and present to you many opportunities for further optimization. If your test does not win, you haven’t lost anything.

Instead, you learn from it. Have a look at your conversion funnel and fix your eye on the next thing in your optimization pipeline.

Have the right tool in your arsenal

One of the most important and often overlooked aspects of any CRO program is the tool(s) it uses. Being a scientific and logical process, CRO warrants sturdy processes, and a tool makes for the foundation. CRO programs are typically started off using existing systems (email, sheets, etc.) While this may turn out great for a start, it can very quickly get out of hand. It’s important to be mindful and onboard a tool well in advance. 

There are many tools out there – both free and paid. Choosing the right tool, at the right time is imperative. For example, VWO has all the best practices structured in one holistic platform, with so much more. Get a VWO free trial or take a demo to give your CRO program the structure it warrants. 

Conclusion

Experiments can be quite volatile in the beginning. You must understand that CRO is a marathon and not a sprint. The best practices mentioned in the blog post will have a deliberate and incremental effect to give you an initial push.

Also, the agility in your strategy and patience in your experimental temperament will take you a long way while you build upon your existing optimization efforts. 

A CRO Guide for Marketers

As a marketer, pulling online traffic is one part of your job. But all that traffic you produce via ad campaigns, SEO, and SEM efforts doesn’t guarantee conversions. As a result, it has always been nothing less of a quest for digital marketers to identify ways that can help them yield the best leads and…

As a marketer, pulling online traffic is one part of your job. But all that traffic you produce via ad campaigns, SEO, and SEM efforts doesn’t guarantee conversions. As a result, it has always been nothing less of a quest for digital marketers to identify ways that can help them yield the best leads and the best conversions.

what is cro marketing

This is where CRO marketing enters.

In this blog post, we will focus on what CRO marketing is, the best strategies to pick up and prioritize, and the tools that are available for you to leverage CRO’s true power.

What is CRO marketing?

Conversion rate optimization marketing or CRO marketing is crucial because optimization enables your marketing efforts to edge over those who do not practice CRO marketing efficiently.  

CRO marketing is a process of increasing the percentage of your website visitors taking the desired action (such as subscribing to mailing lists, signing up for a free trial, filling a form, or downloading an ebook).

These actions could vary from meeting your main business conversion goal to attaining micro conversions. For instance, visitors making an online purchase on an eCommerce store could be your primary goal, and clicking the ‘add to cart’ button, which would move them ahead in the sales funnel, could be a micro conversion.

So, how can you get more visitors to convert? The first step is not looking at the visitors but looking within your company to identify your business goals.

What are CRO marketing goals?

First thing first. You must know what you want to achieve with your optimization exercise. You need to zero in on a set of goals, and for that, you need to sit with your stakeholders and identify the critical metrics (also known as KPIs) that define your business goals. Once you have identified the goals specific to marketing, you can use the A/B testing framework recommended below to test your optimization ideas.

  1. Do your research: Utilize Google Analytics and behavioral analysis tools, such as heatmaps, session recordings, and surveys, to gather behavioral insights from your existing website traffic.
  1. Identify the top key metrics that directly impact your business conversion goals—for example, free trial sign-ups, online purchase, etc. Don’t forget to make observations of these data as they will help you compare your conversion rates pre and post experimentation.
  1. Construct hypotheses based on each of the observations and combine them wherever required. You can fill the following to create and articulate a hypothesis: “I believe______, and if I am right then _______, because_________.” The latter part of the hypothesis is the qualification of the outcome that you expect and why you think so. For example, I believe that adding customer testimonials on key pages will address the low rate of the audience making the purchase because testimonials will establish the credibility of our product/service as social proof. You can prioritize your hypotheses using the ICE model for efficient execution. 
  1. Run your experiment: You can utilize a testing system with analytics like VWO to measure the KPIs against the current baseline and desired goal. For example, to test the impact of social proof on your conversions, run an A/B test, wherein your control could be a landing page without testimonials and a variation of the same landing page with testimonials.
  1. Analyze your test results: Study your test results for any uplifts in conversion rates (including micro conversions). It’s imperative to run a test till it reaches 95% statistical significance, which yields relevant results. Also, you should run a test for a minimum duration that accounts for traffic variations for both weekend and weekdays. Do remember to document your result metrics for subsequent experimentation activities. 

In the below section, I’ve listed some common and easy-to-implement tests that can kick-off your CRO journey smoothly and are likely to give you some quick wins.

CRO strategy to get started: focus on low-hanging fruits

Your high-traffic website has enormous potential to turn your visitors into customers. You only need a direction on how to go about the optimization process. Your CRO strategies could be both site-wide and visitor level.

Here is a list of some strategies that you can implement:

  1. Rule of congruence: Congruence in marketing can be explained as alignment and uniformity of your brand voice across channels in the most intuitive way. For instance, your ads should match the image on your landing page, leaving no room for confusion in your prospects’ minds. The last thing you want is for your prospects to doubt your brand. You can determine and confirm what drives more traffic—same image on the ad and the landing page, or two different images. 
  1. Use CTAs anchored within the blog text: According to an experiment by HubSpot, they captured between 47% and 93% leads on one of their posts[1], using the text-based CTA. You can highlight your text-anchored CTAs within your blog posts or guidebooks. Test whether they work in favor of your conversions. 
Example of anchor text CTA in a blog post
Image Source: [1]
  1. Create lead flows: A lead flow in CRO defines the flow in which a lead is captured in exchange for an offer. It is a pop-up or a slide-in box that appears while you interact with a web page. The idea of this pop-up is to offer something to add value to your experience. But it can be annoying, as it hinders your focus while you are reading on a screen to solve a problem at hand. Yet, in a few cases, it has worked well. For example, the below-shown lightbox drove a humongous 1,375% more sign-ups[2] as compared to the sticky sidebar form on the landing page. 

Lightbox form:

Example of a lightbox sign up form
Image Source: [2]

Sidebar form:

Example of a sidebar form
Image source [2]
  1. Optimize your conversion path: Remove the friction from the sales process for your high-intent visitors. These are those visitors that land on your website either through paid advertisements or direct search with an intent to convert. You can remove the friction by optimizing your conversion paths for different visitors. Conversion path is a process wherein your website visitor becomes a marketing-qualified lead (MQL). For example, a few visitors might want to get down straight to work. In a B2B setup, your prospects might not want hand-holding across a typical, step-by-step buyer’s journey that you have crafted. Instead, they would need a call with a sales rep by jumping right on requesting a demo than a free trial.

Analyze visitor behavior, note these observations, and run experiments to find the best conversion path for high-intent visitors. A healthy conversion path for your website constitutes: 

  • Digestible content with a clear context: Address a specific pain point with a solution in your content.
  • Knowledge of buyers’ personas: You should know who is your ideal prospect/customer. For example, you should know their demographics, psychographics understanding, etc.
  • Enticing CTAs: Action-oriented, congruent, and clear CTAs are the most enticing ones. But again, never overestimate your assumptions. You might end up shooting yourself in the foot without backing your hypotheses with data. Utilize the behavior analysis tools to embed CTAs in the right places on your website.
  • Optimized thank you pages: The conversion path comes to a full circle with an optimized thank you page. It will fulfill the promise you made to your visitor on your landing page and allow you to push them further in their buyer’s journey.  

5. Optimizing your high-performing content: Leaving your high-performance content unattended is akin to killing your conversions slowly. For example, your high-traffic blog posts should trigger you to push these visitors into your marketing funnel. You can do so by adding CTAs pertaining to your business within the context of your content in the most logical and intuitive way. This approach is likely to yield better conversions. 

You can optimize your high-converting landing pages as well if they do not drive traffic. To do so, fix your SEO with semantic indexing and entities, which might result in an organic push from Google. Of course, this does not guarantee a high ranking on Google. However,  you can run an experiment to see if that works for your high-converting content pages. 

6. Retargeting to re-engage website visitors: Your visitors may jump off your website for many possible reasons. However, you can retarget these users by using push notifications and also by optimizing your landing page.      

  1. Web push notifications (like VWO engage): These notifications are pretty easy to build and do not need much technical skill as well. You can utilize push notifications to encourage visitors to keep coming back to your website, thereby increasing conversions.
  1. Optimizing your paid ads: You can optimize your ad content with a well-crafted copy, engaging images, videos, and a compelling offer to target the lost visitors. With behavioral insights in place, you can also utilize website personalization for retargeting. You can utilize retargeting tools such as Retargeter[3] and Adroll[4].

These strategies lay the foundation of your CRO efforts. But lack of prioritization before implementation can leave you befuddled with the results. To gain efficiency, prioritize your strategies in a way that they can be implemented logically.

Prioritize your strategies

Assign a number on a scale of 1 to 10 to every strategy that you pick for your optimization exercise, using the impact, confidence, and ease (ICE) scoring model. A higher score would mean that the strategy should be picked and implemented early on in the pipeline. 

Now that you have a list of prioritized strategies with you, ready to be implemented, you must realize that strategies alone can’t help you succeed in your optimization journey. You will also need the right set of tools. 

CRO marketing tools

Evaluate your CRO tools well before putting them to use. Here is a recommended list of tools you can utilize for your CRO efforts. 

Best practices

Your customers’ response and, of course, trends in your business goals should drive your CRO efforts. However, there are a few best practices that any CRO team should follow:

  • Always have your ideas, assumptions backed by data, and test them before making any changes to your website.
  • Utilize the ICE model to prioritize and rank your CRO strategies as discussed in the above sections.

CRO marketing: on a rescue mission for the complex journey of the modern customer

An efficient CRO strategy takes care of the complicated journey of a modern, 21st-century customer on your website. For example, a visitor’s journey on a travel website is not as straightforward as one would assume.

As shown in the below image, the flow involves several nuances that need to be taken care of while creating a website’s CRO strategy. 

Example of a user journey on a travel website
Image Source: [3]

Identify the steps where your visitors get lost and break them down into small and easy steps to move things seamlessly. Never assume that your prospects or even customers know everything. Instead, always back your observations with data, empathize with your personas, and implement changes only after testing your experimentation hypotheses.

CRO marketing is not a one-time activity but a persevering quest to be better than what you were yesterday. The best part about CRO marketing is that it does not interfere with your other existing marketing strategies; instead, it facilitates them to convert better.

So go on, embark on this exciting journey! Happy optimizing! 

A Practical Guide to Building a CRO Roadmap

If you’ve recently embarked on your CRO journey, here’s a couple of questions for you: How do you prioritize your experimentation ideas? Do you work in silos, or do you see benefit in opening up experimentation to collaboration? If you do see benefit, how do you plan to go about achieving it? How do you…

If you’ve recently embarked on your CRO journey, here’s a couple of questions for you: How do you prioritize your experimentation ideas? Do you work in silos, or do you see benefit in opening up experimentation to collaboration? If you do see benefit, how do you plan to go about achieving it? How do you plan to address resource issues in your testing plan? The answer to all these questions points to one strategic move that differentiates CRO experts from beginners – building a CRO roadmap. 

Building a sustainable CRO roadmap guides your efforts and ensures it systematically contributes towards your business goals at large. Whether you are an agency handling CRO for hundreds of clients or someone who manages CRO for your company, a roadmap will streamline your efforts and maximize throughput by avoiding redundancies and providing a clear step-by-step approach towards optimizing your site. 

Similar to a calendar, a CRO roadmap is essentially a detailed schedule that entails which experiment will be launched when, the time and resources it requires, and the expected outcome. A roadmap ensures that each tweak, change, test, and insight adds value to the next step and accordingly strengthens it to deliver improved results. With a dedicated roadmap to consult, you don’t rely on hope to get results from a few poorly planned and ill-executed experiments scattered across months.

Cro Roadmap illustration

Why do you need a CRO roadmap? 

You can think of a CRO roadmap as a step-by-step framework that you refer to for prioritization, test planning, and allocation of resources for all your CRO efforts, without which you would be completely shooting in the dark. Here are some of the major reasons you need a CRO roadmap to get started.

To switch from a fragmented to a strategic approach

If you randomly run a survey on your homepage this month and conduct a couple of tests on your product page the next month (and so on), you are not going to be able to make the most of the insights gathered or leverage the full potential of the results. To do so, you need a roadmap that dictates every process so you can feed every insight and learning into your pipeline and use it judiciously to drive more substantial results from your program as opposed to some scattered wins or losses.

Let’s say you want to improve your online store’s checkout rate. Needless to say, there are tonnes of tests you can run to optimize for the same. For instance, you could optimize the number of steps in the checkout flow, add social proof and trust badges, avoid the addition of surprise costs at the last step, and so on. Now, without a roadmap, you wouldn’t know which one to prioritize and you might just end up spending too much time running each one of them without getting the expected outcome. On the other hand, if you follow a roadmap, prioritize tests, plan and scope them out over a calendar month/quarter, you can be assured of more promising results.

To get a better hold of resource planning

Again, if you have a systemic approach to optimization, you can always plan your resources in advance, delegate projects, and overall function smoothly with little or no friction as opposed to facing a mini resource crisis every time you decide you want to run an ad-hoc test. 

Moreover, you can always learn from experience and incorporate your learnings of how you can allocate resources better to drive more significant results, efficiently. This is not possible if you follow a haphazard outlook towards optimization and don’t depend on any set framework to guide decisions. 

To improve the speed and efficiency of your CRO program

Needless to say, optimizing your digital properties methodically will only improve the efficiency of your efforts as you would be incorporating previous learnings and doubling down on what works well. Having an overarching roadmap also ensures your processes and tasks are aligned with the overall business goals, so there is minimum iteration, faster delivery, and more promising results.

For instance, if you follow a roadmap, you will know which tests you have to run in the coming month and have the liberty to start laying the groundwork (analyzing data, getting variations created, etc.) and plan your resources accordingly. On the other hand, if you are running sporadic tests, you will end up wasting time in deciding what to test next, ensuring it doesn’t overlap with another test, and planning your resources for it.

How to develop a successful CRO roadmap

An Example Of A Successful Cro Roadmap
Steps to create a CRO roadmap

Revisit your business goals

Take a step back to revisit your most pressing and current business goals so you can understand how CRO can help you achieve them. These goals will anchor your CRO program and ensure your efforts are not aimless or applied in the wrong direction. 
For example, an eCommerce company could have a business goal to increase the average order value, while for a media company, the goal could be to uplift the content consumption on their site. These will then help you deduce what your optimization goals (and their corresponding metrics) need to be.

Deduce corresponding website goals, KPIs, and target metrics from your business goals

Use your business goals to drill down upon what are some of the more tactical website goals you want your CRO program to achieve, what are the performance indicators you need to watch out for, and what would be the target metrics you need to measure corresponding to them. For instance, if increasing the average order value is your business goal, you can break it down further into:

  • Increasing upsell & cross-sell 
  • Increasing visits to product pages
  • Increasing checkout and ‘Add to cart’ rate

Now, these could be your optimization goals, each of which you can tackle using specific strategies and tests. The metrics to be measured could be revenue per customer, conversion rates, and so on. 

Flow Diagram Of Optimization Goals At Different Stages

Understand where you currently stand and establish a baseline for your key metrics

Before commencing, you will need to perform a CRO audit of your site to establish a starting point for your optimization program against which your progress can be assessed. Therefore, for all your key metrics, be sure to analyze your historical data so you can condense it into a baseline basis the trends and patterns it shows. Make sure the date range you select for this is not less than 30 days, as you would need a substantial amount of data to be able to gauge your business’ past performance against these metrics.

Formulate data-backed hypotheses taking insights from visitor behavior data

Use the data you have been gathering through various tools like heatmaps, session recordings, surveys, Google Analytics, and other user research tools to glean insights that you can turn into optimization opportunities. For instance, if you noticed high drop-offs at category pages, you could consider revamping them to highlight the CTA, reduce clutter, and make product details more appealing and apparent to customers. 


Next, craft your hypotheses based on these insights that can move your key metrics and solve for these visitor pain points. Here’s how you can formulate a solid hypothesis:

How To Build A Structured Hypothesis

Prioritize these hypotheses

Optimizing the optimization process is often just as important as the tests themselves. Prioritizing where you invest energy will give you better returns by emphasizing pages that are more important to the business

Chris Goward – Founder, WiderFunnel

Now that you have a bunch of different test hypotheses at your disposal, prioritize them so you can populate your pipeline in a way that you tackle issues that are more likely to yield maximum results first. There are several different prioritization frameworks you can follow for the same: 

Prioritization Of Your Optimization Plans

P.I.E. Framework [By Chris Goward] 

As per this model, each hypothesis is given a score of 1-10 on three factors – Potential, Importance, and Ease. This translates to the amount of potential a particular hypothesis has in improving the performance of a page, how important is it to optimize a particular page, and how easily the task can be accomplished. Once assigned scores for each of these factors, you just add them up for every hypothesis and prioritize them basis their total scores. 

Pie Framework Potential Importance Ease
Image Source: [1]

T.I.R. Model [By Bryan Eisenberg]

As per this framework, a score of 1-5 is assigned based on Time, Impact, and Resources. This means that a test that requires minimal time, has the most impact, and requires the least amount of readily available resources would be given a 5 under every factor. Once done, individual scores of each factor are multiplied, and the highest priority is given to one with the largest score.

I.C.E. Framework [By Sean Ellis]

As per this model, a score of 1-5 (with 5 being the highest) is assigned to each hypothesis based on the likely Impact it would have, the level of Confidence you have in the hypothesis, and what is the level of Ease with which it can be implemented. Once all 3 scores are added, the hypothesis with the highest score is given the highest priority, and so on.  

I C E Framework Impact Confidence Ease
Image Source: [2]

While these were a couple of the most popular frameworks, there are others you can consult to prioritize your tests so they can be picked in the descending order and fed into your pipeline. But, that’s not all. You should also categorize your hypotheses basis the final goals (the ones discussed above) they accomplish.

Collate all the information collected so far

Now that you have drilled down your business goals to tactical conversion goals you plan to achieve with CRO, and also used visitor behavior insights to craft hypotheses that can help you achieve them, you can start breaking each test down into its specific details. 

This would include your test name, description, hypothesis, observation, target page, and goal it is expected to accomplish. Share this spreadsheet with all stakeholders so that it can be enriched and evolved as your progress in your program.

Create your testing pipeline 

Create a proper schedule to plan out your experiments considering their priority order, resource bandwidth, and time required for implementation. Assign owners to each test and make sure you keep track of the progress and the results obtained so you can use your learnings constructively to enrich your pipeline. Here’s an example template you can refer to for creating a weekly A/B testing plan:

An Example Of A Testing Plan Calendar Template

Creating one such calendar would especially be useful for agencies so you can always use a standardized template for all your clients and deliver promised results systematically. 

Challenges to roadmapping 

Miscalculation of time and resources required

Creating a CRO and testing roadmap requires you to carefully plan your schedules and resources well in advance, which can sometimes be a challenge for teams that are new to experimentation. You might not be able to accurately estimate your requirements before actually getting into the process. Very often, running experiments can take extra time, and it’s not in your control to wrap them up sooner. For instance, if a test takes longer (than you had anticipated) to reach statistical significance, you couldn’t have accounted for it, and now you have to hold off the next one to ensure they do not overlap. 

The best way out of this is to always keep a buffer or stick to a conservative estimate, both for time and resources, and any other requirement you might have, to accommodate for unexpected changes that occur during experiments. 

Inaccuracy, inconsistency, or unavailability of data to inform your CRO roadmap 

If you haven’t been relying heavily on data for all decision-making, you might struggle at first with creating a data-backed roadmap. This is largely because you will most likely discover inconsistencies or inaccuracies that don’t add up, you would have data stored in silos, or you wouldn’t have comprehensive data for the entire time range and critical metrics you want to look at. 

To overcome this, use only data you know for sure is accurate and enough to inform your hypotheses and eliminate what you feel is only corrupting your roadmap. You can also prioritize your hypotheses in such a manner that those already backed by sufficient data are ones you test first and collect more data for those that need to be strengthened. 

Not evolving your strategy with changing times and scenarios 

After you have spent a whole lot of time and effort building your roadmap from scratch, it is quite natural to want to just stick to it. However, committing to one strategy with all your heart and soul and not being open to evolving it with the changing dynamics of your business or the industry will do you more harm than good. 

To stay ahead of the curve, you need to keep evolving your roadmap so you can accommodate for these changes. For example, if the festive season is coming up and it’s time for your annual sale, you would want to run a test or two catered specifically to the sale, and hence it’s important you have provision to take that decision quickly and incorporate it into your plan. 

Tools you need to build and maintain a CRO roadmap

VWO Plan 

VWO Plan allows teams to collaborate and create experimentation pipelines seamlessly and efficiently. It’s an all-in-one platform that empowers you to record observations, generate hypotheses, save ideas, and create and manage your experimentation program via a centralized dashboard. You can forget about data/idea silos and maintaining multiple documents, sheets, presentations, whiteboards, or dashboards, and rely on a single platform to ideate, run experiments, and measure their impact. 

Vwo Plan In App Screen

Jira

From the house of Atlassian, Jira is an agile project management and tracking software you can use to manage your CRO pipeline. You can easily plan and track your workflows over a kanban board to the most minute detail and collaborate with various teams to ensure you conduct your experiments efficiently and drive faster results.

Example Of A Jira Board
Image Source: [3]

Trello

Trello is another project management software that allows you to track and manage your CRO roadmap and work collaboratively and efficiently. Apart from organizing, prioritizing, and managing your pipeline, Trello enables you to boost productivity by automating redundant and manual tasks such as due date and calendar commands.

Example Of A Trello Board
Image Source: [4]

Asana

Asana is an easy-to-use project management software that provides a timeline view of your CRO pipeline and allows you to track its progress. It offers integration with all major tools such as Salesforce, Tableau, and Adobe Cloud and customizes your workflows as per your specific requirements. 

Example Of Asana Project Board
Image Source: [5]

Conclusion

Approaching CRO with a strategic roadmap ensures that every effort is tied to your overall business goal. Without one, you are most likely to rely on disintegrated efforts, which may or may not show significant results. Therefore, roadmapping is the way to go to achieve success with CRO and drive noteworthy business growth. It’s time to put your knowledge to test and embark on a strategic CRO journey by creating a roadmap for your program. On that note, Happy Optimizing! 

How Seattle Southside RTA Increased Visitor Guide Conversions

One of the most common challenges for a destination is accurately capturing a visitor’s intent to visit and ensuring that a personalized experience leads to both clicks and goal conversions. Visitor Guides, often referred to as Travel Planners, are an essential way to track intent to visit.   Like many other destinations, Seattle Southside Regional Tourism… Read More

The post How Seattle Southside RTA Increased Visitor Guide Conversions appeared first on Bound.

One of the most common challenges for a destination is accurately capturing a visitor’s intent to visit and ensuring that a personalized experience leads to both clicks and goal conversions. Visitor Guides, often referred to as Travel Planners, are an essential way to track intent to visit.  

Like many other destinations, Seattle Southside Regional Tourism Authority’s Travel Planner requests remain a critical goal in tracking website engagement.  In Fall 2019, the Seattle Southside RTA team saw a decrease in their Travel Planner request conversions, both year-over-year and quarter-over-quarter. Furthermore, the team found that the gap between their Targeted and Default audience increased, highlighting that this goal had become a more challenging conversion point for audiences.

With the intent of increasing Travel Planner conversions, the Seattle Southside RTA team decided to refresh their content with a seasonal focus on their imagery.  They first created two versions of Travel Planner content, both with gorgeous Fall scenery highlighting the region’s colorful season.  

Launching the content as an A/B test, the Seattle Southside RTA team was amazed at the increased engagement.  Within the first few weeks, this new content saw an 8% increase in Click Through Rates (CTRs) over their regular Travel Planner content.  Better yet, the team saw a 15% increase in Travel Planner conversions within the first month of the new content running.

Encouraged by this initial response, the team continued to run the seasonal content until the last few weeks of the year.  Quarter over quarter, the team saw a 26% increase in conversions, with a year-over-year increase of 43%!  Using the new Goal Dashboard, the Seattle Southside RTA team was able to further breakdown the conversion rate for each of the new content pieces allowing them to see that the new content pieces not only had higher CTRs, but also much higher conversion rates vs their original content. The team found a 79% increase in conversions for their Desktop content, as well as a 63% increase for their Mobile content.

Inspired by the results of their A/B test, Seattle Southside RTA plans on launching more tests for goal related content with seasonal imagery.  Knowing that the new Goal Dashboard allows for a deeper level of insight into their testing, the Seattle Southside RTA team is better equipped to deepen their visitor’s personalization journey to increase goal conversions and engagement.

Congratulations to Seattle Southside Regional Tourism Authority for a job well done!

 

Interested in increasing your conversions?  Personalization can help you get there.  We’d love to chat with you more about making it happen!

 

The post How Seattle Southside RTA Increased Visitor Guide Conversions appeared first on Bound.

A Well Balanced Content Personalization Diet: 3 New Years Resolutions to Increase Goal Conversions

Happy New Year, travel marketers! The beginning of January always brings its own kind of magic with resolutions and the opportunity to both reflect on the past year and look towards the next.  It’s also a time that, if I can be honest, is a little overwhelming with the pressure of setting life-changing goals. And… Read More

The post A Well Balanced Content Personalization Diet: <br/>3 New Years Resolutions to Increase Goal Conversions appeared first on Bound.

Happy New Year, travel marketers! The beginning of January always brings its own kind of magic with resolutions and the opportunity to both reflect on the past year and look towards the next.  It’s also a time that, if I can be honest, is a little overwhelming with the pressure of setting life-changing goals. And it’s not only personal goals! Working within the digital marketing space I feel that every other content piece is focused on “new year, new marketing strategy” resolutions that couldn’t be easier to implement – or so the articles read…

At Bound, we’re big believers in starting where you’re at, especially when it comes to personalization and your marketing strategy.  That’s why one of our resolutions this year is to focus on something that we know has an impact: optimizing our goal conversions

When it comes to our monthly content reports, few things give our Customer Success Managers more joy than seeing an increase in click through rates on goal related content pieces.  But as fun as these increases are to see, we are even more thrilled by increases in the goal conversions themselves. As we’ve become increasingly aware of the important relationship between clicks and conversions – and the very different stories each can highlight when they don’t align  – we’re excited to share our new Goal Dashboard and highlight three resolutions on increasing your conversions in 2020:

Read More (into your A/B tests):

When in doubt about your content, run an A/B Test!  While click through rates can certainly highlight your audience’s preferences for the imagery, copy or CTA, how do you account for the content’s impact on the actual conversion?  Within the new Goal Dashboard, you can now compare conversion rates against your campaigns, segments and pieces of content, allowing for a deeper level of insight. We recently took a closer look at an eNewsletter related A/B test we have been running with a DMO.  Month over month, we found that one content piece had consistently less clicks than the other. However, in comparing the conversion rates between the two pieces, we saw that the content piece with a lower CTR had a considerably higher conversion rate. This comparison helped us see the value of a content piece we might have otherwise removed and will help inform future A/B tests.

Exercise (your understanding of your Mobile and Desktop visitors differences):

As we’ve written about before, there are many things to take into consideration when creating content for your Desktop and Mobile visitors.  Goal conversions are no different, especially given that our Mobile visitors are often less likely to convert. Within the new Goal Dashboard, we can now dive into the conversion rates for our different segments across campaigns, allowing us to compare, for example, fly-ins served to desktop visitors and banners served to mobile audiences.  Layering in this insight can help us develop content best suited for each of our unique visitors groups.

Spend Less (time guessing how your content is performing):

Over the past few years, we’ve increasingly become fans of thoughtful “abandonment” content and the way these direct CTAs can increase conversions for visitors who have initiated, but not completed, a conversion goal.  While we often see this content with high CTRs, it can be challenging to determine how exactly this content contributes to the overall goal. Thankfully, our new Goal Dashboard takes the guesswork out of content creation and helps us see exactly which Abandonment content is best contributing to the goal. 

Our hope for your 2020 is that your conversion related content is directly increasing your goal conversions (leaving you with more time to increase engagement for your ad visitors!)   Knowing that goal conversions are a vital piece to understanding your visitors intent to travel, we’re excited that our new Goal Dashboard will bring new awareness and insight this year.  Cheers to you and your increased conversions!

Want to learn more about the Goal Dashboard or personalizing to increase your conversions?  We’d love to chat with you and hear all about your 2020 marketing resolutions!

The post A Well Balanced Content Personalization Diet: <br/>3 New Years Resolutions to Increase Goal Conversions appeared first on Bound.

How Denver Increased Engagement for Ad Visitors

With the launch of their “always on” regional “Reclaim the Weekend” ad campaign, VISIT DENVER faced the challenge of how to keep their main landing page relevant. The regional effort, which promotes visiting Denver for a long weekend, targets a wide variety of personas that change monthly. Instead of creating multiple new landing pages every… Read More

The post How Denver Increased Engagement for Ad Visitors appeared first on Bound.

With the launch of their “always on” regional “Reclaim the Weekend” ad campaign, VISIT DENVER faced the challenge of how to keep their main landing page relevant. The regional effort, which promotes visiting Denver for a long weekend, targets a wide variety of personas that change monthly. Instead of creating multiple new landing pages every month, VISIT DENVER used personalization with Bound to match the hero slideshow content to the appropriate persona.

VISIT DENVER developed and rolled out three waves of ad personalization within their first year with Bound:

Wave 1

The first step was to personalize the slideshow for visitors coming to the landing page directly from the ad. This involved not only showing the appropriate group of slides but also starting the slideshow with the content targeted to that persona. While these visitors only had a 4% increase in clicks specifically on their persona-targeted slides, overall page engagement was significantly increased. Compared to other visitors, the ad persona segments had a 53% increase in visit duration and a 45% decrease in bounce rate when entering the site through the Reclaim the Weekend landing page.

Wave 2

The second step was to use Bound’s Media Optimizer tool to personalize the slideshow for visitors who were exposed to the ad. The pixeling capabilities of Media Optimizer allowed Denver to target Reclaim page visitors who had seen, but hadn’t clicked on the ad, as well as visitors who came back to the site after their specific persona campaign ended. Not only did these pixeled visitors have great page engagement, but they also had a 100% increase in clickthrough rates on the slideshow and were 28% more likely to click specifically on the persona-targeted slides. With this information, Denver had the data needed to show that visitors were still interested in persona-specific content even if they had not clicked on the ad. 

Wave 3

The third step was to build on the learnings from the first two phases of personalization and launch a fly-in campaign. The fly-in targeted visitors exposed to the persona who had never clicked on the ad or otherwise reached the Reclaim page. Using the fly-in, Denver was able to successfully direct 2% of these visitors to the page and continued to increase website engagement. Visitors exposed to the persona fly-in had a further 23% increase in visit duration and 18% decrease in bounce rate.

By identifying visitor interests based on ads, even if those visitors never directly engaged with the ad, Denver has been able to increase views on their key ad landing page and continually increase their landing page engagement. This has increased overall site performance and has allowed Denver to optimize the experience for these high-value website visitors. 

Want to learn more about personalizing for your targeted ad visitors? 

The post How Denver Increased Engagement for Ad Visitors appeared first on Bound.

E-commerce Abandonment Rates

There are few key performance indicators that everyone focuses on for an e-commerce store: conversion rates, average order value and the number of monthly visitors. These metrics translate into money…

Please click on the title to read the full artic…

There are few key performance indicators that everyone focuses on for an e-commerce store: conversion rates, average order value and the number of monthly visitors. These metrics translate into money...

Please click on the title to read the full article!

Bayesian vs Frequentist A/B Testing – What’s the Difference?

Bayesian versus Frequentist Statisticians: the war is real
Imagine that you wake up in the one morning and you don’t remember anything from your previous life. You’ve erased all memories from…

Please click on the title to read the full articl…

Bayesian versus Frequentist Statisticians: the war is real Imagine that you wake up in the one morning and you don’t remember anything from your previous life. You’ve erased all memories from...

Please click on the title to read the full article!