Predict A/B test results using ancient Indian astrology

Wouldn’t it be wonderful if there was a way to instantly know which version is going to win without waiting weeks or months for the data to arrive?

We at VWO have a mission to help marketing and product teams reduce the time and effort required for figuring what works best for their business and what doesn’t. In 2010, we pioneered the Do-It-Yourself visual editor for business teams for editing webpages and creating their variations for A/B tests without involving IT teams. That innovation cut the effort to launch an A/B test from weeks to hours and the A/B testing industry hasn’t been the same ever since.

But, as anyone who has run an A/B test knows, you still have to wait for weeks in order to start getting statistical significant results about which version is better.

We’ve been wondering if there’s a way to cut this wait period as well. Wouldn’t it be wonderful if there was a way to instantly know which version is going to win without waiting weeks or months for the data to arrive?

Can AI help?

The first research direction we turned to was the booming AI technology. You would have certainly heard about neural networks, machine learning, and data science. Could one such  technique help us? If neural networks can drive cars today, perhaps they can also predict which website version is better?

Over the last few years, we tried several techniques and invested a lot of money in trying to get one of the AI techniques to work. We did make some progress when a couple of months ago, we launched an AI powered website copywriter that helped marketers come up with new alternative headlines, CTAs and product descriptions to test.

But, unfortunately, similar AI-based techniques didn’t perform satisfactorily on prediction of A/B test results. It seems driving cars, beating world Go champion and detecting tumors is an easier problem than predicting which website design is going to be a better choice.

Enter IA

As we were about to almost give up on our dream project, someone pointed us to the elephant in the room: astrology. Our first reaction was to dismiss this absurd suggestion immediately. We’re a skeptical bunch and there’s no place for such pseudoscience in a serious business like ours.

But we were desperate and decided to give this totally absurd idea one last shot. We were told that millions of people across the world predict all sorts of phenomena using techniques like horoscopes, or Tarot cards. If people are using a technique to predict whether their marriage will succeed or not, can’t it be used to predict whether A version is better than the B version?

After looking at the cost-benefit equation, we decided to do a small pilot. If, as we predicted, astrology didn’t work we’ll just lose some investment and time. However, if it did work, it’ll change the marketing and user research industry forever.
So, in all earnestness we kickstarted the pilot. After doing initial research on the most predictive astrology technique, we settled on Indian Astrology (IA). Particularly, we settled on parrot astrology. What gave us confidence in it was that there’s even a wikipedia page on it.

Parrot
Photo via Vijay S (on Flickr)

In parrot astrology, an astrologer lays out multiple options in front of a caged pet parrot. When the parrot is let out of the cage, it picks one of the options as the prediction. Simple and straightforward. We loved it already.

What we did for our pilot is the following:

  • Take 100 A/B tests from our database where there was a clear, statistical significant winner (either A or B)
  • Print their screenshots
  • Recruit an Indian astrologer with a parrot and have the parrot pick between the pairs of screenshots across 100 trials (one for each A/B test)

Our null hypothesis was that the parrot wouldn’t be able to pick better than the random chance. That is, the parrot should pick the correct winner and the loser each roughly with a 50% chance.

However, to our astonishment, the parrot picked the winner 80 times out of 100. Using any standard binomial calculator (either frequentist or bayesian), you’ll quickly find out that the chances of that happening are close to nil. Indian Astrology really worked!

Pblog

We’re obviously onto something here. We soon plan to write a scientific paper on this and submit our findings in a top journal. It’s all very exciting and mysterious. How does the parrot know which version works best? Well, we don’t know that but it shouldn’t prevent us from using the technique in the real world (just the same way we still don’t know why deep learning is so effective at various kinds of problems but it’s used widely).

Get early access to parrot-powered A/B test prediction

We’re actively trying to scale the pilot with an eventual goal for making it available for all businesses in the world via simple user interface inside VWO or via an API.  The main challenge is in finding enough parrots and making sure we’re doing this ethically. Once we do that, any business in the world will be able to rapidly converge on optimal user interface without the need to do any A/B testing.

We have some slots for people who want early access to this technology. If you want to give it a shot, email us here: aprilfools@vwo.com

Wait, what?

Well, didn’t we say we’re a skeptical bunch? It turns out there’s no scientific evidence that astrology works. Moreover, we strongly condemn caging of any bird or animal for anything other than clinical or medical research (even in clinical research, we wish computational and physical models increasingly replace actual animals).

If you were excited about the prospect of prediction of A/B test results, you’re not alone. We’re excited about the same thing and even though parrot-astrology doesn’t work, we’re committed to continue making progress towards helping business teams achieve a higher probability of success in each of their A/B tests.

After launching an AI powered website copywriter last year, we recently launched an AI powered heatmap predictor. After you enter your webpage URL, we take a screenshot and use deep learning to predict the distribution of clicks on the website. This can help you spot obvious dead zones or distractors on your website even before you make it live or use a heatmap tool on it.

Ecommerce

This tool is free to use. So feel free to try it as many times on as many websites.

We’re sorry for giving you hope that you can get rid of A/B testing, but hope you had fun reading the post 🙂

Happy April Fools day!

Had a laugh? Send me a note at paras@vwo.com. I read and reply to all emails.

A/B testing the covid19 vaccine

Vaccines for covid19 were first developed a year ago and finally they’re here. What took them so long?

Hi 👋  I am Paras Chopra, founder & chairman of VWO. I am back with the 8th post in my fortnightly series outlining a new idea or a story on experimentation and growth. Hope you find it valuable.

Vaccines for covid19 were first developed a year ago and finally they’re here. What took them so long?

The answer to this question points to the most significant A/B test conducted in recent history: the clinical trials for making sure vaccines are safe and effective against the covid19 virus.

Ab V 1
From Vaccine Clinical Trials 101

These clinical trials take so long because they’re divided into three phases, with each phase testing a different aspect of the vaccine. This 3 stage design has taken shape over multiple decades with thousands of scientists informing how it should be run. 

Because human lives are at stake, clinical trials of vaccines are done very carefully. Be too hasty, and an unsafe vaccine can get released. But be too careful, and many more lives can be lost from the unchecked pandemic. 

This tradeoff leads to thoughtful design of clinical trials of vaccines and hence for the conversion optimization industry, there’s a lot to learn from it.

So, let’s see the 3 stages in which vaccines are tested and what we can learn from them for our own A/B tests.

Phase 1: Test for side-effects

Once developed in the lab and tested on animal models, in the first phase, a vaccine is tested on a small group of humans. The primary objective of this phase is to ensure safety of the vaccine and to check if there are any side effects from it.

Translating it to the A/B testing world, if you have a large change to make (say a homepage change, or a massive marketing campaign), the first thing you need to check for is not the positive impact of the change but for potential negative impact. A variation that increases your clickthrough rate but ultimately decreases your business is no good. Hence, you should always have a set of business metrics (called guardrail metrics) which you never want to see getting reduced.

Phase 2: Test for effect-size

Once phase 1 is over, researchers enroll a bigger group and divide this group into multiple groups, with each group getting a different dosage level, and one group getting no dosage at all (and instead getting a placebo). You can think of this as an A/B/C/D test where the control means no change while different variations correspond to different dosage levels. 

Volunteers belonging to different groups are then monitored over time to see whether the disease got cured or prevented, while ensuring there are no major side effects. 

The variation with minimum side effects and maximum benefits is declared a winner, and then progressed to final phase 3 trials.

Phase 3: Large scale confirmatory test

By phase 2, there’s clear evidence that the vaccine seems to work and researchers have discovered an appropriate dosage for it as well. However, the work isn’t done. 

Researchers want to be doubly sure that the vaccine will work when released to the masses, and hence in phase 3 significantly expand the number of people included in the test. Researchers actively seek out people from various previous conditions and age groups to ensure the sample they have is a good representation of the entire population. 

This larger group is divided into two groups and the final A/B test happens with one placebo and the winner dosage from the previous phase. This is the phase that takes the most amount of time and effort.

Summing up: A/B testing works!

The tradeoffs relating to human lives seen for testing vaccines aren’t likely to arrive in our businesses, but we all can learn from the rigorous manner in which scientists figure out things. Just like us, they’re humans too and want so badly for vaccines and other treatments to work. But they also know that the cost of a wrong vaccine can be devastating. 

In the same spirit, I think it’s prudent to realize that even though we want our ideas to succeed so badly, we need to be careful about the possibility of such ideas making things worse for the business. A/B testing is the only way to ensure that they don’t.

If you have a story of an idea that you really wanted to work but it didn’t, I’d love to know about it. Send me a note at paras@vwo.com. I read and reply to all emails.