What Can You Learn From Running an A/B Test for 2 Years?

We just concluded an A/B test on Analytics-Toolkit.com that has been left to run for just over 2 years. And it failed, as in failing to demonstrate a statistically significant effect based on the significance threshold it was designed for. Has it been …

We just concluded an A/B test on Analytics-Toolkit.com that has been left to run for just over 2 years. And it failed, as in failing to demonstrate a statistically significant effect based on the significance threshold it was designed for. Has it been a waste of time, though, or can we actually learn something from […] Read More...

Does Your A/B Test Pass the Sample Ratio Mismatch Check?

Most, if not all successful online businesses nowadays rely on one or more systems for conducting A/B tests in order to inform business decisions ranging from simple website or advertising campaign interventions to complex product and business model ch…

Most, if not all successful online businesses nowadays rely on one or more systems for conducting A/B tests in order to inform business decisions ranging from simple website or advertising campaign interventions to complex product and business model changes. While testing might have become a prerequisite for releasing the tiniest of changes, one type of […] Read More...

5 Reasons You Shouldn’t Be Scared to Try A/B Testing

Editor’s Note: We want to focus on giving you actionable advice for what you can pay attention to, evaluate, and implement quickly to make sure you get the most out of Q4 and the remainder of 2019. Building on the video from our post on our sessi…

Editor’s Note: We want to focus on giving you actionable advice for what you can pay attention to, evaluate, and implement quickly to make sure you get the most out of Q4 and the remainder of 2019. Building on the video from our post on our session from the A/B Testing Summit, we wanted to […]

The post 5 Reasons You Shouldn’t Be Scared to Try A/B Testing appeared first on The Daily Egg.

Client-Side Vs. Server-SideA/B Testing Tools: What’s The Difference?

Marketers of all stripes are obsessed with tools. This obsession has bred comprehensive lists of growth tools, SEO tools, and general online marketing tools. It is no different with us in conversion optimization. We nerd out on testing tools. Though no optimization program has ever hinged on which tool you used, there are important distinctions […]

The post Client-Side Vs. Server-SideA/B Testing Tools: What’s The Difference? appeared first on CXL.

Marketers of all stripes are obsessed with tools.

This obsession has bred comprehensive lists of growth tools, SEO tools, and general online marketing tools. It is no different with us in conversion optimization. We nerd out on testing tools.

Though no optimization program has ever hinged on which tool you used, there are important distinctions between A/B testing tools—from the statistics they use, their price, and more.

One thing that is often either overlooked or misunderstood is the difference between client-side and server-side testing tools.

Client-side, server-side: what’s the difference?

Here’s the basic difference between client-side and server-side testing tools:

Client-side tools create a variation of your test page by doing manipulations on your browser via clever JavaScript. Your web server sends the same version of the page, the default, as it would normally do. All the variations are reflected by the visitor’s browser. Since the changes aren’t done on your server, and are instead on the visitor’s browser, they’re called ‘client-side tools.’

Server-side tools are different in that no modification occurs at the browser level. When the visitor lands on your page, a randomly picked version of your test is sent straight from your server. Yes, this means you have to involve your developers in testing, but it also gives you robustness and flexibility.

Tim Ash was quoted in a TrustRadius report, summing up the difference between the two types of tools very well:

Tim Ash from Site Tuners

Tim Ash:

“With server-side tools, the content management systems are building in testing, maintaining multiple versions of content and serving them up based on segmentation. It offers control over deployment regardless of browser problems. Client-side tools are usually easier for the marketer. They have WYSIWYG editors and make it easy to make surface level changes without needing IT, which is a huge pain point. It’s hard to use them to restructure the registration path or the checkout, though, because you’ll touch the CMS.”

There’s a third type of tool as well, closely related to server-side capabilities. There’s only one company I know that uses this method (SiteSpect) so it’s not super common. What they use is something called the proxy server or intercept method.

Basically, the tool acts as a proxy server between the visitor’s browser and your server. It controls the site content that is presented to the visitor, so it eliminates most of the problems that occur with client-side tools, while retaining the robustness of server-side tools.

A/B Test Planning: How to Build a Process that Works

By Jaan Matti-Saul

A strong A/B testing plan will allow you to increase your revenue and learn valuable insights about your customers.

  • This field is for validation purposes and should be left unchanged.

Okay, but when should I use which?

As Paras Chopra from VWO said in a blog post, the two aren’t in opposition. Different tools for different needs and goals:

paras

Paras Chopra:

“Why are we supporting server-side testing frameworks? Aren’t they in competition with us? Not really! Our focus with Visual Website Optimizer is to make A/B and multivariate testing easier by taking technical hassles out of the picture. In fact, ideally, we want our users never to touch any code (be it HTML, JavaScript, CSS, or PHP). So, VWO is suited for people who don’t want to rely on developers or IT team for doing even the simplest kind of testing (thereby lengthening the time involved in getting the test up and running).”

Client-side tools offer ease of use, faster execution, and a WYSIWYG editor. You don’t necessarily need to take up your development team’s time to get experiments running.

That said, server-side tools have benefits as well. An article from Google claimed that server-side implementations offer more flexibility to do things such as:

  • Run experiments for websites with dynamic content.
  • Test non-UI changes that still have an affect on your objective. For example, a database query result set that is returned to a user.
  • Integrate Google Analytics experiments with your service (e.g., content management provider).
  • Manage experiments using your own optimization platform.

In summary, they offer more flexibility. They also tend to be more robust, and they eliminate a few of the weaknesses of client-side tools. Some of these potential weaknesses include the flicker effect (which we’ll talk about below), load speed, and flexibility.

As Chris Goward of WiderFunnel said in a TrustRadius report, the JavaScript approach (i.e. client-side) has a lot of benefits, but also some downsides:

Chris Goward

Chris Goward:

“They’ve solved the biggest problem, which is the technical barriers to testing. The downside is load time and site speed, and the lack of flexibility of what you can manipulate. There are cases where it will have a flicker or it loads more than it needs to. Other approaches might allow for more flexibility and speed. I don’t think we’ve hit the best solution yet as an industry.”

Pete Koomen, co-founder of Optimizely, summed it up well:

pete

Pete Koomen:
“The advantage of testing on the client side is speed and simplicity. You can test a lot of changes quickly without much initial investment. On the other hand, testing on the server side is both more work and generally more powerful.”

Though, as another way to look at it, Nate Weiss (CTO of Conductrics) says that the ‘skipping IT’ benefit of client-side tools could have a cost for some companies. He explains:

nate

Nate Weiss:

“For companies that have a rigorous QA and approval process for their web content in general (outside of testing), the “skipping IT” benefit of client-side tools has a cost: it becomes a “back door” where the usual approval processes and so on end up getting ignored, because the testing tool doesn’t know about the internal workflows that lead to normal content being published. It is then very difficult, if done at all, to keep these two “moving targets” (changes made normally, and changes made via the testing tool) in sync.”

In summary, use different tools for different uses and goals.

The flicker effect

As mentioned above, one of the weaknesses of client-side tools can be the flicker effect. What is it?

Basically, it’s when, for very small amount of time, your website visitor sees the default page before the variation loads. It’s also known as the “Flash of Original Content.”

This generally happens because of an asynchronous script with the testing tool. While it has the advantage of not slowing down the page, this type of script can cause a flicker effect once in a while.

This matters because the human eye detects, and we change our behavior when we know we’re part of an experiment. Though, there is the surprising (fluke) occasion that it helps a test. Take this example from an Unbounce blog post:

As David Hauser, co-founder of Grasshopper, said in the article:

david

David Hauser:

“Our A/B testing tool had a bug that delayed the $25 activation fee from being crossed out until a few seconds after the page loaded.

This error ended up creating a much larger uplift than having it already crossed out on load, when the bug was fixed.

The result now is that the activation fee shows, and then is crossed out after a few seconds.”

Most of the time, though, flicker effect results in statistically significant differences in test results. That’s totally affecting the test, and can skew the outcome. The goal is to reduce the flicker to 0.0001 seconds so people won’t see anything.

So, I mentioned that flicker effect is really only a problem with client-side tools. What are some ways, if there are any, that we can mitigate this?

Synchronous vs. asynchronous scripts

Client-side tools can load scripts in two distinct ways: synchronously and asynchronously. What’s the difference?

Synchronous scripts

Synchronous loading is when a testing tool’s script loads completely before the page starts to load. It calls back to the server and waits for an answer. Put another way, the scripts are loaded sequentially starting with the tag.

For example, Optimizely uses a synchronous snippet to prevent the flicker effect and a balanced CDN system to make sure the impact on the page is minimized.

Synchronous:

Asynchronous scripts

Asynchronous is when some scripts can be loaded simultaneously. They load while the testing script runs, possibly causing a flicker effect if the original content loads a split section before the variation.

Asynchronous:

You can run client-side tools like VWO and Optimizely either way. In general, page loading speed is important, so asynchronous code is usually preferred. The batched scripts allow the pages to load faster, and you can usually prevent the flicker effect with a few measures.

As Paras Chopra said in a GrowthHackers thread:

paras

Paras Chopra:

“Generally with VWO’s asynchronous code, there should not be any flickering. In edge cases or in cases where you haven’t put asynchronous code in the head of the page, flickering might happen. We’ve made our asynchronous code in such a way that the original page is hidden until the settings are loaded. (The page gets loaded in parallel and that’s why it is async but it’s not displayed).

With regards to VWO’s (or any other tool’s) synchronous code, I wouldn’t recommend it generally unless there are specific cases that cannot be fixed. Synchronous slows down the page speed.”

Alhan Keser from WiderFunnel also expressed a similar sentiment. Unless you’re running a massive site with tons of visitors, page speed is of primary importance:

alhan

Alhan Keser:

“Asynchronous loading addresses a separate issue: helping with overall site speed, not FOOC [flash of original content]. Given the speed of modern CDNs, snippets loading synchronously should be the least of your concerns. But, if you’re like our neighbors here in Vancouver, PlentyOfFish, with a bajillion users hitting their site at the same time, you may want to be considerate of what and how things load on your pages.”

Other tips to prevent flicker effect

One way to prevent the Flicker Effect that Optimizely outlined is to add a timeout on asynchronous scripts. Their default setting is one second. This means that if the A/B testing script isn’t loaded in that time, the A/B won’t be executed. If the visitor has a slow connection, that visitor won’t be included in the test.

Some other ways to prevent the flicker effect:

  • Optimize your site for speed. Make sure everything, especially the images, load fast. Move your site onto a fast web host. Use compression, optimized images, and content delivery networks.
  • Take your A/B testing tool script out of the tag manager (if you use one, like Google Tag Manager).
  • If you can, do split URL testing instead.
  • Pay attention to the HTML elements you change. If you made changes on the page itself (as opposed to doing split URL testing), the key thing to keep in mind is the amount of code that was changed. If you changed thousands of lines, you’re far more likely to see a flicker effect as opposed to changing button copy. So the scope of changes influences everything.

What client-side tools do right

Client-side tools let marketers run tests accurately without the help of developers.

That’s not a small accomplishment. Development resources are expensive, in time and cash, so enabling testing independent of product development is efficient. Optimizely put it well:

“When it’s time to take action and make changes to your site, there is no replacement for dedicated technical resources—in most organizations, these resources are hard to come by and typically demand waiting in line.”

Test faster

Sean Ellis introduced the idea of high-tempo testing as a key mechanism for growth.

The basic premise is that the faster and more efficient you can run tests, the more of them you can run, and the faster you can grow. Though their bottlenecks were mostly at the ideation and organizational levels, many optimizers find bottlenecks at the development and implementation levels. Client-side tools using visual editors solve this.

In a previous piece, Andrew Anderson wrote about the importance of speed in testing tools. He gives a few examples of tools as well:

Andrew Anderson:

“When thinking about speed, there are two different factors that matter. The first is after your initial deployment and set-up, how fast can you get a test from concept to execution to live? The goal for any program is to get most tests with at least five experiences through that concept in 30 minutes or less. That is a hard target to meet but it does express just how important it is to prioritize speed in testing and the need for general knowledge of CSS, HTML, and JavaScript by the operator.

This means that items such as templatized rules, easy site interaction, easy interface, and easy navigation are vital. It also means that tools that require complex URL targeting rules, constant IT deployment, or require manual jQuery set-up should be lowered on your priority scale.

An example of a tool that for the most part does this right is Optimizely while an example of those that doesn’t is SiteSpect.”

Andrew also noted that, while lots of tools provide easy-to-use visual interfaces to create treatments, the automatically created jQuery code is often pretty bad. This results in cross-browser compatibility issues and slowness. Always have a developer check and improve the code—or learn jQuery. It will really help.

Server-side tools: when to use?

As I’ve mentioned, a huge benefit to client-side tools is the speed and ease-of-use for not-super technical people. So what, then, are the benefits of server-side tools? Why would companies use them?

As Matt Gershoff, the CEO of Conductrics explains, server-side tools have their advantages as well. Here’s what he had to say about them:

Matt Gershoff

Matt Gershoff:

“The main advantages of server side will be:

  • Greater flexibility and control. Use test and targeting for non-visual testing targeting (call centers, CMS systems), and tight integration into marketing tech stack for single workflow (CMS, database, web servers, and so on).
  • Privacy. Keep all testing/targeting decisions internal.
  • Security. Reduce attack surface.
  • For web apps, reduce the need for ‘heavy’ pages.”

Server-side tools may also be better for mobile sites that serve dynamic content because, as Nate Weiss, CTO of Conductrics, put it, “the slower connections and less powerful browsers in many phones mean that client-side JavaScript approaches are at more of a disadvantage on mobile.”

He continues to talk about the applicability to native mobile applications:

nate

Nate Weiss:

“Some verticals continue to shift toward native mobile apps for engagement. If your company has a native mobile app or plans to in the future, selecting a tool that supports server-side testing now means that you’ll be ready to start using the same tools for testing in your apps (with developer involvement).”

Another topic of fewer conversations that Nate brought up to me was that, depending on the nature of your site and the tests you plan to run, it may be more respectful for your visitors to use server-side tools.

As Nate said, “consider which is more important: your convenience when setting tests up, or your visitor’s user experience (and impact on their mobile data plans).”

In addition, server-side tools broaden your capabilities for optimization beyond web-based-visual presentation issues, in a way. What they do is allow you to think a little different about optimization, more towards optimizing any of your transactional marketing efforts.

With client-side tools, the main topic of discussion is around the flicker effect. This is an issue, but it isn’t the only, or the largest, issue to be discussed with tools. Things like flexibility, control, and privacy affect your optimization as much or more than flicker, especially if you take the precautions to mitigate that risk.

Which type is your tool?

Here’s a chart that quickly categorizes popular tools. Let us know if we’re missing any or getting anything wrong:

Server-SideClient-Side
Conductrics (both capabilities)AB Tasty (both capabilities)
AdobeConductrics (both capabilities)
SiteSpect (both capabilities)Convert
Optimizely (both capabilities)Bound
AB Tasty (both capabilities)Kameleoon
 Monetate
 Omniconvert
 Optimizely (both capabilities)
 Oracle Maxymiser
 VWO

Source

Conclusion

I’m simplifying things, but here’s the bird’s eye difference between server-side and client-side tools:

Server-side tools render code on the server-level and send a randomized version of the page to the viewer with no modification on the visitor’s browser. Client-side tools send the same page, but JavaScript on the client’s browser manipulate the appearance on both the original and the variation.

Client-side tools use either synchronous or asynchronous code, both of which have pros and cons regarding pagespeed, flicker effect, and loading issues.

Client-side tools are easier for marketers to use, and you can often get tests up and running faster without burning development resources. However, server-side tools offer greater flexibility and control. They also protect your privacy and security better, and for web apps, reduce the need for ‘heavy pages.’

Both tools have their use cases and don’t need to be used exclusively. However, far more important than the tool you choose is the process by which you optimize. Don’t overthink it with tools.

The post Client-Side Vs. Server-SideA/B Testing Tools: What’s The Difference? appeared first on CXL.

AGILE A/B Testing Update: Custom API & Support for Non-Binomial Data

I’m happy to announce the release of two long-awaited features for our AGILE A/B Testing Calculator: Support for non-binomial metrics like average revenue per user A new custom API for sending experiment data to the calculator Below is an explana…

I’m happy to announce the release of two long-awaited features for our AGILE A/B Testing Calculator: Support for non-binomial metrics like average revenue per user A new custom API for sending experiment data to the calculator Below is an explanation of each of these new features in some detail. Support for Non-Binomial Data While our […] Read More...

Statistical Methods in Online A/B Testing – the book

The long wait is finally over! “Statistical Methods in Online A/B Testing” can now be found as a paperback and an e-book on your preferred Amazon store. Note that the Kindle edition is available for $2.99 or equivalent if you’ve alrea…

The long wait is finally over! “Statistical Methods in Online A/B Testing” can now be found as a paperback and an e-book on your preferred Amazon store. Note that the Kindle edition is available for $2.99 or equivalent if you’ve already purchased the paperback (through Kindle Matchbook). The book is a comprehensive guide to statistics […] Read More...

Going from 10 to 100 experiments per year: Building the frame

Note from the Editor: This article is part I in a series dedicated to helping you increase your online experiment…Read blog postabout:Going from 10 to 100 experiments per year: Building the frame
The post Going from 10 to 100 experiments per year: Bu…

Note from the Editor: This article is part I in a series dedicated to helping you increase your online experiment...Read blog postabout:Going from 10 to 100 experiments per year: Building the frame

The post Going from 10 to 100 experiments per year: Building the frame appeared first on WiderFunnel Conversion Optimization.

Time to Get Personal

As a destination marketer, one of your main challenges is turning your website visitors into destination visitors. Before a visitor comes to your destination they compare you their other options. During this research phase, you tailor your ads to match their interests, you utilize search engine marketing tools to make sure your advertising and social… Read More

The post Time to Get Personal appeared first on Bound.

As a destination marketer, one of your main challenges is turning your website visitors into destination visitors. Before a visitor comes to your destination they compare you their other options. During this research phase, you tailor your ads to match their interests, you utilize search engine marketing tools to make sure your advertising and social content is targeted to their search results, and you hope these visitors click through to your site to consume and engage with the top content you’ve created.

But what are the best practices in turning these online visitors into destination visitors?

Leading destination marketers from Explore Branson, Elkhart County, Indiana, and Visit Williamsburg believe website personalization is a cost effective way to turn their website visitors into destination visitors. In Time to Get Personal, these three destinations highlight some of the ways Bound’s personalization solution has helped them stand out amongst their peers and convert their online visitors into destination visitors. Some of their results include the following:

  • Explore Branson has seen a 560% increase in e-newsletter sign-ups by using a personalized pop-up targeted to different website audiences.
  • Elkhart County, Indiana used Bound’s A/B testing capabilities to increase travel guide conversions by 253%.
  • Visit Williamsburg used Bound to maximize the value of their paid media campaigns. ince targeting paid media visitors to the website with personalized landing pages, they have seen a 41% increase in time on site.

Read more in this report to learn how these destinations got these results and to see if now is the right time for you to explore personalization for your destination’s website.

Related Posts

The post Time to Get Personal appeared first on Bound.

A/B Testing with a Small Sample Size

The question “How to test if my website has a small number of users” comes up frequently when I chat to people about statistics in A/B testing, online and offline alike. There are different opinions on the topic ranging from altering the si…

The question “How to test if my website has a small number of users” comes up frequently when I chat to people about statistics in A/B testing, online and offline alike. There are different opinions on the topic ranging from altering the significance threshold, statistical power or the minimum effect of interest all the way […] Read More...

Free Guide: How to Strategize & Execute Profitable Personalization Campaigns

When I speak with our clients, it often strikes me how many of them feel overwhelmed by the very idea of personalization. Our imagination, often fueled by the marketing teams of various software companies, creates a perfect world where personalization enables every interaction to be completely custom for every individual. In this dreamland, artificial intelligence […]

The post Free Guide: How to Strategize & Execute Profitable Personalization Campaigns appeared first on Brooks Bell.

When I speak with our clients, it often strikes me how many of them feel overwhelmed by the very idea of personalization.

Our imagination, often fueled by the marketing teams of various software companies, creates a perfect world where personalization enables every interaction to be completely custom for every individual. In this dreamland, artificial intelligence and machine learning solve all our problems. All you have to do is buy a new piece of software, turn it on, and…BOOM: 1:1 personalization.

As a data scientist, I’ll let you in on a little secret: that software only provides the technological capability for personalization. Even further, the algorithms found within these tools simply assign a probability to each potential experience that maximizes the desired outcome, given the data they have access to. Suffice to say, they’re not as intelligent as you are led to believe.

If you caught our first post in this series, you already know that we define personalization a bit more broadly, as any differentiated experience that is delivered to a user based on known data about that user. This means personalization exists on a spectrum: it can be one-to-many, one-to-few, or one-to-one.

And while there are many tools that enable you to do personalization from a technical standpoint, they don’t solve for one of the main sources of anxiety around personalization: strategy

Most personalization campaigns fail because of a lack of a strategy that defines who, where and how to personalize. So I’ve put together a free downloadable guide to help you do just that. This seven-page guide is packed full of guidelines, templates and best practices to strategize and launch a successful personalization campaign, including:

  • Major considerations and things to keep in mind when developing your personalization strategy.
  • More than 30 data-driven questions about your customers to identify campaign opportunities.
  • A template for organizing and planning your personalization campaigns.
  • Guidelines for determining whether to deliver your campaigns via rule-based targeting or algorithmic targeting.

Free Download: Plan & Launch Profitable Personalization Campaigns.

The post Free Guide: How to Strategize & Execute Profitable Personalization Campaigns appeared first on Brooks Bell.