How Codecademy Saw 5X Growth with Strategic Testing

Codecademy started out in 2011 as one of the first free products that taught people to code.  Since then, we have helped over 45 million learners improve their lives through programming while making major improvements to our product—driven by experimentation. Our goal is to empower the world through tech education, reaching as many learners as […]

The post How Codecademy Saw 5X Growth with Strategic Testing appeared first on CXL.

Codecademy started out in 2011 as one of the first free products that taught people to code.  Since then, we have helped over 45 million learners improve their lives through programming while making major improvements to our product—driven by experimentation. Our goal is to empower the world through tech education, reaching as many learners as possible to support our vision. 

As a subscription business, we knew that small repeatable wins could compound to earn us millions of dollars in additional revenue. As part of the growth team, we were tasked with experimenting on any part of the business that could drive an impact, so we focused on the big, key levers of our monetization flow. 

From spaghetti testing to strategic testing 

Codecademy process.

When I think back to where we were two and a half years ago, the biggest issue with our testing program was that we tried several small ideas in different places and didn’t iterate enough on concepts. Our ideas were driven by individual opinions and changes that we saw our competitors making, which is one of the worst ways to run a testing program. 

The tide turned for us when we began focusing on our own learners and what we knew about them—their perceptions of our product and consumer behavior that was supported by evidence.  

This research informed better hypotheses, which allowed us to launch better tests and start unraveling bigger problems—problems we couldn’t solve without changing fundamental parts of our business. That’s what led us to strategic testing. 

Once we realized this, we had to prioritize which areas of the business to test first. We already had a really strong brand and top of funnel traffic. So we settled on growth levers at the bottom of the funnel–experimenting with pricing, the checkout page, plan mix, and our trial model. 

Wins deep in the funnel would have an immediate impact on the business and deliver the growth we needed. The following are some of the strategic tests we ran at Codecademy. 

Resources to help you to do this:

Testing our pricing strategy and plan-mix  

Pricing plans.

In my opinion, price and plan-mix are the two most powerful growth levers in any subscription business. You can capture more money upfront, lower churn, and increase user motivation all by experimenting in these two areas. 

All users make a pricing decision when purchasing, making it one of the few truly global levers in your business.  

Experimenting here is no small undertaking, however. Price and plan-mix are complex areas to test because they impact several areas of your business and product, so any changes need to be made carefully. In addition, you need to coordinate with other teams that touch pricing, such as marketing, finance, customer support, and sales.

The first experiment we ran was around exaggerating the difference in price between the monthly and annual plans in high-GDP countries, which had a massive positive impact on the number of annual subscriptions. 

If you can find ways to incentivize users to enter an annual plan, you’re probably capturing more revenue upfront than the revenue generated from the average lifetime of a monthly user. While it’s a large undertaking, experimenting is likely to have a big payoff in these areas. 

Resources to help you to do this:

Testing our checkout process

When we started working with CXL Agency, one of the first tests we ran was on the checkout page. We started by reordering the plan cards, so a user’s eye would be drawn to the most attractive plans first, followed by the less-attractive plans. We also highlighted the savings on the checkout page. 

This test leaned on a psychological principle called the Rule of 100, which suggests that users perceive dollar amounts over 100 as being greater in value than percentages, despite both equating to the same amount. Because the savings on our annual plan was above $100, we tested it by showing dollar amounts. 

We saw a 28% lift in annual Pro plans, as well as a small bump up for overall page conversions.

It was good to see a classic psychology principle play out successfully in our experiment.

Before:

Pricing options.

After:

Resources to help you to do this:

Testing our free-trial models 

After seeing success in our pricing and purchase flow, we then moved on to test our free-trial model, which is the way that free users can experience the paid product before fully committing. This was one of the harder tests to execute and had a high level of risk and reward because we were tinkering with the company’s core revenue engine. 

The best trial models are the ones that align incentives to start a trial closest to where your users find value. But back in 2016, we had no trial model at all. Users entered the site and if they wanted, they could upgrade. Unsurprisingly, only a very small number of users opted for the paid Pro version.  

In 2017, we shifted toward a trial model where everybody who enrolled in our free plan was automatically enrolled in a trial of the Pro version. While this was better than where we were before and had an impact on paid conversions, it did come with a few downsides. 

For one, users experienced the whole product at once, so they weren’t able to differentiate between paid and free features. 

Additionally, because we were experimenting with the core revenue model for the business, we wanted to do it in a slow and measured way. We didn’t want to ship something that we saw succeed within the experiment window but would have a long-term detrimental impact on the business.

So we shipped our ideas in three stages: 

  1. The first was aimed at existing users who had used the product for 60 days or longer. We showed them an offer for the Pro trial, which they had to opt into. We saw a strong positive result, so we kept moving.
  2. In the second round, we tested whether a “credit card upfront” trial model would work better for new users than the control. This was perhaps our most important test because the new user experience is one of the core pillars and revenue engines for the business.  We shipped a variant, monitored it for two months, but didn’t see the results we were looking for. We went back to the drawing board to see what went wrong.
  3. We shipped the third version a month or so later with a slightly different paywall structure. Finally, we saw a variant beat the control. It was ultimately a huge success for the business and the product. 

Resources to help you to do this:

Testing our messaging, clarity, and propositions 

We then moved up another step in the funnel, focusing on our checkout page messaging and key decision points in the app.  

The most successful tests we’ve run are based on understanding our users’ perceptions, problems, objections, fears, and uncertainties. The way we use these insights, even in small tactical testing, ultimately ladders up to bigger strategic areas, such as helping our users understand our free versus paid product. 

We tested generic content versus more personalized content at checkout, based on up-funnel navigation patterns. Our aim was to increase the clarity and relevance of our product to specific users.

Such tests helped us understand which messaging really resonated with our users and what triggers them to sign up. This insight can help us in marketing—as well as product development—moving forward. 

Before:

Codecademy before.

After:

Codecademy data scientist.
Codecademy web developer.
Personalized content at checkout, based on up-funnel navigation patterns.

Resources to help you to do this:

Conclusion 

To move away from testing irrelevant ideas to more strategic experiments, you need to gain real insights and data about your users. Use experimentation to test the business’s most highly guarded areas, such as price, proposition, and even the product itself.

If you want to grow way beyond the local maximum, the only way to do that is through testing the strategic parts of your business. 

The post How Codecademy Saw 5X Growth with Strategic Testing appeared first on CXL.

Who’s Hiring in January 2019?

Here are our picks: Website Optimization Specialist – In Atlanta, SunTrust is looking for a specialist to be responsible for “developing and executing business strategies, processes and policies to enhance the sales and service experiences intrinsic to SunTrust’s digital spaces.” A/B Testing & Personalization Analyst – Join Barnes & Noble’s Optimization team in New York […]

The post Who’s Hiring in January 2019? appeared first on Brooks Bell.

Here are our picks:

Website Optimization Specialist – In Atlanta, SunTrust is looking for a specialist to be responsible for “developing and executing business strategies, processes and policies to enhance the sales and service experiences intrinsic to SunTrust’s digital spaces.”

A/B Testing & Personalization Analyst – Join Barnes & Noble’s Optimization team in New York to “improve bn.com’s content, design, and usability for customers and to create unique experiences based on customers’ preferences and behaviors.”

Director-Digital Product Analytics & Testing –  Join the Enterprise Digital and Analytics team at American Express in New York.  They are looking for a leader to “provide value to the online card shopping experiences within the Global Consumer and Commercial businesses through customer data and measurement, insights through analytics techniques and experimentation.”

Marketing Manager, International Conversion – Ancestry is looking for a candidate to join their Conversion Marketing team in San Francisco.  This person is “responsible for improving and optimizing the user experience at each step in the conversion funnel with the end goal of maximizing revenue from visitors in each of Ancestry’s key global markets.”

Marketing Manager, A/B Testing & Optimization – Join AuthO’s Growth Team in “driving improvement in key engagement metrics and customer experience throughout the customer lifecycle.”

Director of B2B Marketing, Demand Generation – Join Vimeo’s B2B marketing team in New York to “scale qualified lead acquisition, build and continuously optimize digital marketing, account-based marketing (ABM), email automation, social, and event-based marketing channels.”

Sr. Analyst, eCommerce Direct to Consumer Analytics – Newell Brands is looking for a senior analyst in Hoboken, New Jersey, to drive “sustainable growth online through the best-in-class use of data and analytics.”

Digital Marketing Leader – Website Optimization – Join GE Healthcare in Wauwatosa, Wisconsin to “develop a rigorous testing and experimentation framework, and conceive, scope and implement experimentation initiatives to improve the website user experience and drive conversion rate optimization.”

Manager, Marketing Planning, Test & Analysis – Express is looking for an individual to lead the testing and optimization program in Columbus, Ohio, “starting with A/B & multivariate testing taking us into experience optimization and eventually personalization.”

 

Looking for a job or to fill a position?  Give us a shout and we’ll help spread the word in our next careers blog post.

 

The post Who’s Hiring in January 2019? appeared first on Brooks Bell.

Free Guide: How to Strategize & Execute Profitable Personalization Campaigns

When I speak with our clients, it often strikes me how many of them feel overwhelmed by the very idea of personalization. Our imagination, often fueled by the marketing teams of various software companies, creates a perfect world where personalization enables every interaction to be completely custom for every individual. In this dreamland, artificial intelligence […]

The post Free Guide: How to Strategize & Execute Profitable Personalization Campaigns appeared first on Brooks Bell.

When I speak with our clients, it often strikes me how many of them feel overwhelmed by the very idea of personalization.

Our imagination, often fueled by the marketing teams of various software companies, creates a perfect world where personalization enables every interaction to be completely custom for every individual. In this dreamland, artificial intelligence and machine learning solve all our problems. All you have to do is buy a new piece of software, turn it on, and…BOOM: 1:1 personalization.

As a data scientist, I’ll let you in on a little secret: that software only provides the technological capability for personalization. Even further, the algorithms found within these tools simply assign a probability to each potential experience that maximizes the desired outcome, given the data they have access to. Suffice to say, they’re not as intelligent as you are led to believe.

If you caught our first post in this series, you already know that we define personalization a bit more broadly, as any differentiated experience that is delivered to a user based on known data about that user. This means personalization exists on a spectrum: it can be one-to-many, one-to-few, or one-to-one.

And while there are many tools that enable you to do personalization from a technical standpoint, they don’t solve for one of the main sources of anxiety around personalization: strategy

Most personalization campaigns fail because of a lack of a strategy that defines who, where and how to personalize. So I’ve put together a free downloadable guide to help you do just that. This seven-page guide is packed full of guidelines, templates and best practices to strategize and launch a successful personalization campaign, including:

  • Major considerations and things to keep in mind when developing your personalization strategy.
  • More than 30 data-driven questions about your customers to identify campaign opportunities.
  • A template for organizing and planning your personalization campaigns.
  • Guidelines for determining whether to deliver your campaigns via rule-based targeting or algorithmic targeting.

Free Download: Plan & Launch Profitable Personalization Campaigns.

The post Free Guide: How to Strategize & Execute Profitable Personalization Campaigns appeared first on Brooks Bell.

Thank You + Brooks Bell’s Best of 2018

It’s January 3, and if you’re like us, you’re already heads down at your desk and neck deep in emails. But we’d be remiss if we didn’t take a minute to reflect on the previous year. In November of 2018, we quietly celebrated 15 years of being in business. When Brooks Bell was founded, experimentation was in […]

The post Thank You + Brooks Bell’s Best of 2018 appeared first on Brooks Bell.

It’s January 3, and if you’re like us, you’re already heads down at your desk and neck deep in emails. But we’d be remiss if we didn’t take a minute to reflect on the previous year.

In November of 2018, we quietly celebrated 15 years of being in business. When Brooks Bell was founded, experimentation was in its infancy. But despite all the changes we’ve experienced since then, one thing remains true: it is the opportunity to connect with so many interesting people that are solving big problems for their business that makes our work worthwhile. Thanks for walking with us.

A look back at some of our big moments from 2018

Winning like Winona

In January, our Founder & CEO, Brooks Bell, was recognized as one of 25 women who rocked digital marketing in 2017. Later in the year, she was also announced as a Southeastern Finalist for EY’s Entrepreneur of the Year award. 

We also celebrated 2017’s record-breaking growth, were recognized as Optimizely’s North American Partner of the Year, and we garnered our local business journal’s Best Places to Work award.

Getting Lit with Illuminate

Fun fact: We originally built Illuminate to help us better manage and iterate upon our clients’ tests. Over time, we got so much great feedback, that we decided to make it available to everyone this year.

Now, with a successful beta launch under our belt and even more new features being added to the software, we’re excited to see where this new endeavor takes us in 2019.

F is for Friends, Fun and…Fear?

In October, things got a little spooky around the office and it had everything to do with Scott, our Director of Sales, who decided to channel his inner Ellen Degeneres for the day (much to our colleagues’ horror). Watch the video if you dare.

Making Bacon for our Clients

Back in 2014, we set a Big Hairy Audacious Goal to achieve $1 billion in projected revenue for our clients. By the end of 2017, we’d reached $500 million. And this past December, we hit $1 billion. (cue ::gong::)

But we’re not resting on our laurels. We’ve set some aggressive goals for 2019, with a focus on personalization, and we’re pumped to get to work.

Brooks Bell takes the Bay Area 

In September, we officially opened the doors to our San Fransisco office. This decision came after years of working with clients on the West Coast and our desire to work even more closely with them. And with the Bay Area’s rich history of innovation, we can’t think of a better place to help more companies push their boundaries through experimentation.

Still Clickin’ 

Last May, we hosted our annual Click Summit conference. We might be biased but this remains one of our favorite events as it’s filled with meaningful connections and seriously impactful takeaways. 2019 marks our 10th Click Summit, and we’ve got big plans. Request your invite today.

2018 on the blog

 


The post Thank You + Brooks Bell’s Best of 2018 appeared first on Brooks Bell.

New Features in Illuminate: Impact Analysis, Enhanced Filters, Updated Dashboard & More

Since we launched Illuminate back in May, our team has been working around the clock to develop even more features to help optimization teams better organize experiments, report performance and maximize impact. Today, we’re excited to share a few of these with you. What’s new in Illuminate? Show impact and determine priority Use our new Impact […]

The post New Features in Illuminate: Impact Analysis, Enhanced Filters, Updated Dashboard & More appeared first on Brooks Bell.

Since we launched Illuminate back in May, our team has been working around the clock to develop even more features to help optimization teams better organize experiments, report performance and maximize impact. Today, we’re excited to share a few of these with you.

What’s new in Illuminate?

Show impact and determine priority

Use our new Impact Analysis to show the overall impact of your tests by page type and identify where you should be focusing your testing efforts.

Sort and filter by what matters most

Filter your tests by 15 attributes including target audience, page type, start and end date, KPIs, revenue impact and more. Not seeing what you need? Add your own using our new custom tagging feature.

Keep sight of the bigger picture

Our new dashboard view enables you to view your program’s overall performance or view performance by a specific team or line of business.

+ a new tiled layout

If you love a good masonry layout (á la Pinterest), then you’re going to love our updated experiment view. Easily switch between a basic list of your experiments or a super slick-looking tiled layout.

Many of these features were developed in response to feedback from our beta users, bringing more of Brooks Bell’s advanced experimentation methodologies directly into the software.

“With Illuminate, you’re not just getting another test repository,” said Suzi Tripp, Senior Director of Innovative Solutions at Brooks Bell. “You’re getting 15 years of experimentation expertise and proven frameworks to help you do more, and do it better.”

Interested in learning more about illuminate? Learn more on our website or schedule a demo using the form below.

The post New Features in Illuminate: Impact Analysis, Enhanced Filters, Updated Dashboard & More appeared first on Brooks Bell.

What are your website visitors doing?

Chances are that you’re tracking your website visitors en masse. You’re probably tracking acquisition sites, tallying up conversions and working to optimize your pages for the best success. But with all of that quantitative research, do you know about each individual user’s journey, and where they are struggling on your site? If not, you should […]

The post What are your website visitors doing? appeared first on Brooks Bell.

Chances are that you’re tracking your website visitors en masse. You’re probably tracking acquisition sites, tallying up conversions and working to optimize your pages for the best success. But with all of that quantitative research, do you know about each individual user’s journey, and where they are struggling on your site? If not, you should check out one of our partners: SessionCam.

Jonathan Hildebrand, Brooks Bell’s Sr. Director of UX & Design, spoke at SessionCam’s user conference last week in Chicago. If you’re unfamiliar with SessionCam, the company began with a mission of building the best session replay solution on the market.  Over time the solution has grown into a fully-fledged behavioral analytics solution including heatmaps, conversion funnels, form analytics and more.

We’ve been blown away by the machine learning algorithms which identify signs of customer struggle and frustration on a website.  We sat down with Jonathan to ask him for a couple takeaways from the event.

As a UX expert, what do you appreciate most about SessionCam?

Where SessionCam really shines is in the qualitative data it provides, which can uncover major hurdles on your site in ways that quantitative data could never reveal. SessionCam’s recordings allow customers to watch a complete play-by-play of a visitor’s experience on the site, whether it’s through a mobile device or desktop.

What about specific to testing?

From a testing perspective, SessionCam can be great for post-test analysis since it allows you to watch videos from the live test experiences. The Customer Struggle Score is also a great way to understand where problems are occurring.

Any interesting case studies?

Definitely. One that comes to mind is a retailer that has a buy online, pick up in store (BOPUS) program. They were using SessionCam to uncover the source of order mistakes. When there was an error at pickup, they would go back and watch that customer’s online session to see if a problem occurred during the online order process and determine if there were any improvements they could make.

And you only need to check out their website to see the kind of value that SessionCam has added to many of the world’s leading brands.

If you’re interested in finding out more about SessionCam, give us a shout.

The post What are your website visitors doing? appeared first on Brooks Bell.