How To Build A Culture Of Experimentation

It’s one thing to run an A/B test correctly and get a meaningful uplift. It’s another thing entirely to transform your organization into one that cares and respects experimentation. This is the goal, though. You can only shake off so much additional revenue if you’re the only rogue CRO at your company. When everyone is […]

The post How To Build A Culture Of Experimentation appeared first on Blog.

It’s one thing to run an A/B test correctly and get a meaningful uplift. It’s another thing entirely to transform your organization into one that cares and respects experimentation.

This is the goal, though. You can only shake off so much additional revenue if you’re the only rogue CRO at your company. When everyone is involved in the game, that’s when you stride past the competition.

It’s not just about the tools you use, or even the skills, but also about the people involved. But organizational matters tend to be a bit complex, as anything that involves humans is. How do you build a culture of experimentation?

This article will outline 9 tips for doing so.

1. Get the Stakeholders Buy into CRO, and Establish Program Principles

First thing’s first—we need to get everybody on the same page.

It used to be more difficult to convince people of the value of conversion optimization. Now, it seems that it is more mainstream, and most people buy into the benefits.

We know from conducting State of the Industry Report that CRO is being more widely adopted, and those that are adopting it are increasingly establishing systems and guidelines for their program. All of this is good.

If you’re just getting started on your CRO journey, though, don’t fret. There are some simple and tactical ways you can start establishing a vision.

First, if you don’t have full buy-in from stakeholders, make sure you have at least one influential executive sponsor who is on your side. If you don’t have this, you won’t go far. (Programs tend to have a substantial ramp-up period before you see a good return.)

Second, write down up front, your program principles and guidelines. I like to create a “principles” document for any team I’m on (and a personal one as well), just so that we know what our operating principles are and we know how to make decisions when things are ambiguous.

Here’s an example of a principles document from my team at HubSpot (just a small section of it, but you would get the point):

Of course, we have tons of documentation from everything on how we run experiments to our goals, and more.

Andrew Anderson gave a great example of his CRO program principles in a CXL blog post:

  • All test ideas are fungible.
  • More tests does not equal more money.
  • It is always about efficiency.
  • Discovery is a part of efficiency.
  • Type 1 errors are the worst possible outcome.
  • Don’t target just for the sake of it.
  • The least efficient part of optimization is the people (with you also included).

Yours could look completely different, but just make sure you up-front script the critical plays and don’t leave any questions hanging in the air. This will help stakeholders understand what you are up to and will also help onboard new employees in your team when they get started.

2. Embrace the Power of “I Don’t Know”

With most marketing efforts, we expect a linear model. We expect that for X effort or money we put into something, we should receive Y as the output (where Y > X).

Experimentation is somewhat different. It may be more valuable to think of experimentation as building a portfolio of investments, as opposed to a machine with a predictable output (like how you’d view SEO or PPC).

According to almost every reputable source, many tests are going to fail. You’re not going to be right. Your idea is not going to outperform the control.

This is okay.

If, for every 5 tests that fail, you get 1 true winner, you’re probably already ahead. That’s because, on the 5 tests that failed to improve conversion rates, you are only “losing” money during the test period. You didn’t set them live for good, so you mitigated the risk of a suboptimal decision. (This alone is a great benefit!)

Outside of that, the one test that did win should add some compounding value over time. A 5% lift here and a 2% lift there add up; and eventually, you’ve got the rolling equivalent of a portfolio with compounding returns:


A side point to the whole “embrace I don’t know” thing is that you shouldn’t seek to test things to validate only what you think is right. The best possible case is that something wins that you didn’t think would win.

That’s how Andrew Anderson frequently frames conversion optimization, saying in this post that “the truth is, in optimization, the more often we prove our own perceptions wrong, the better the results we are getting.”

Ronny Kohavi, too, makes the point that a valuable experiment is when the “absolute value of delta between expected outcome and actual outcome is large.” In other words, if you thought it would win and it wins, you haven’t learned much.


3. Make It a Game

Humans like competition; competition and other elements of gamification can help increase engagement and true interest in experimentation.

How can you gamify your experimentation process? Some tools, such as GrowthHackers’ NorthStar, embed this competition right into the product with features like a leaderboard:


You can create leaderboards for ideas submitted, experiments run, or even the win rate of experiments. Though, as with any choice in metric, be careful of unintended incentives.

For example, if you create a leaderboard for the win rate, it might be possible that people are disincentivized from trying out crazy, creative ideas. It’s not a certainty, but keep an eye on your operational metrics and what behaviors they encourage.

4. Adopt the Vernacular

Sometimes, a culture can be shifted by subtle uses of language.

How does your company explain strategic decision making? How do you talk about ideas? How do you propose new tactics? What words do you use?

If you’re like many companies, you talk about what is “right” or “wrong,” what you have done in the past, or what you think will work. All of this, of course, is nourishing for the hungry, hungry HiPPO (who loves talking about expert opinion).

What if, instead, you talked in terms of upside, risk mitigation, experimentation, and cost versus opportunity instead?

The world sort of opens up for those interested in experimentation. Obviously, you still have to be grounded in reality. You can’t throw insane test ideas at the wall and hope that everyone jumps on board.

But if you can propose your ideas in the context of a “what if,” something you can test out with an A/B test rather quickly, you can probably get people on board.

We see here that 40% of our users are dropping off at this stage of the funnel. We’ve done a small amount of user research and have found that our web form is probably too long. It would take us very little time to code up X, Y, and Z variants, and we’d have a definitive answer in 4 weeks. The upside is big. The risk is low. Let’s run the experiment?

It’s much harder to argue against something like this.

Most of persuasion is framing. If the person you are trying to convince feels attacked or threatened (“you think a scrappy A/B test is better than my 25 years of experience?!”), you’re not going to get far.

If you pull people into the ideation process and propose ideas as experiments with lots of upside, it’s easier to get people involved in the process. Or maybe just start throwing the words “hypothesis,” “experiment,” “statistically significant,” “risk mitigation,” and “uncertainty reduction” into all of your conversations, and hope that people follow along.

It doesn’t need to be limited to experimentation, either. You can make it normal to talk about pulling the data, cohort analyses, user research, and others. These should be normal processes for decision making that replace gut feel and opinions.

5. Evangelize Your Wins

It’s important to stop and take the time to smell the roses. When you win, celebrate! And make sure that others know about it.

It’s through this process of evangelization that you both cement the impact and results you’re creating in others’ minds as well as recruit others to become interested in running their own experiments.

How do you evangelize your wins? Many ways:

  • Have a company Wiki? Write your experiments there!
  • Send a weekly email including a roundup of the experiments.
  • Schedule a weekly experiment readout that anyone can attend.
  • If possible, write external case studies on your blog. This isn’t always possible, but can be a great way to recruit interesting candidates to your program.

I’m sure there are many other interesting and creative ways to celebrate and evangelize wins as well. Make sure to comment in the end on how your company does it.

6. Define Your Experiment Workflow/Protocol

If you want everyone to get involved with experimentation, make sure that everyone understands the rules. How does someone set up a test? Do they need to work with a centralized specialist team or can they just run it themselves? Do they need to pull development resources? If so, from where?

These all are questions that can cause hesitation, especially for new employees; and this hesitation can really hinder the pace of experimentation throughput.

This is why it’s so beneficial to have someone, or a team, owning the experimentation process.

Even if you don’t have someone in charge of the program, though, you can still build out the documentation and protocol. At the very least, you can create an “experimentation checklist” or FAQ that can answer the most common questions.

In Switch, Chip and Dan Heath wrote:

“To spark movement in a new direction, you need to provide crystal-clear guidance. That’s why scripting is important – you’ve got to think about the specific behavior that you’d want to see in a tough moment, whether the tough moment takes place in a Brazilian railroad system or late at night in your own snack-packed pantry.”

“Clarity dissolves resistance,” they say.

Luckily, there are tons of examples of testing guidelines, frameworks, and rules out there to borrow from.

7. Invest in Ongoing Education and Growth Opportunities

This is anecdotal; but I’ve found that the best organizations, those that run very mature experimentation programs, tend to invest heavily in employee development.

That means granting generous education stipends for conferences, books, courses, and internal trainings.

Different companies can have different protocols as well. Airbnb, for example, sends everyone through data school when they start at the company. HubSpot gives you a generous education allowance.

There are tons of great CRO specific programs out there nowadays, specifically through CXL Institute. Some programs I think everyone should run through:

8. Embed Subtle Triggers in Your Organization

I’ve found one of the most powerful forces in an organization is inertia. It’s exponentially harder to get people to use a new system or program than it is to incorporate new elements into the current system.

So what systems can you use to inject triggers that inspire experimentation?

For one, if you use Slack, this is certainly easy. Most products integrate with Slack—Airtable, Trello, GrowthHackers Northstar, and others—so you can easily set up notifications to appear when someone creates a test idea or launches a test.

Just seeing these messages can nudge others to contribute often. It makes the program salient overall.

Whatever triggers you can embed in your current ecosystem—even better if they’re automated—can be used to help nudge people toward contributing more test ideas and experiment throughput.

9. Remove Roadblocks

According to the Fogg Behavioral model, there are 3 components that factor into someone taking an action:

  • Motivation
  • Ability
  • Prompts/Triggers


I think the ability, or the ease at which someone can accomplish something, is a lever that we tend to forget about.

Sure, you can wow stakeholders with potential uplifts and revenue projects. You can embed triggers in your organization through Slack notifications and weekly meetings so that people don’t forget about the program. But what about making it easier for everyone who wants to run a test?

That’s the approach seems to have taken, at least according to this paper they wrote on democratizing experimentation.

Some of their tips include:

  • Establish safeguards.
  • Make sure data is trustworthy.
  • Keep a knowledge base of test results.

To summarize, do everything you can to onboard new experimenters and mitigate their potential to mess up experiments. Of course, everyone has to go through the beginner phase of A/B testing, where they’re expected to mess things up more often than not. The trick, however, is to make things less intimidating while also making it less likely that the newbie may drastically mess up the site.

If you can do that, you’ll soon have an excited crowd anxiously waiting to run their own experiments.


An organization with a mature testing program knows that almost all of it is dependent on a nourishing experimentation culture. One cannot operate, at scale and truly efficiently, with only one or a handful of rogue experimenters.

The program needs to be propped up by influential executive stakeholders; and everyone in the company needs to buy into the basic process of making evidence-based decisions by using research and experiments.

This article outlines some ideas I’ve seen to be effective in establishing a culture of experimentation, though it’s clearly context dependent and not limited to the items on this list.

Got any cool ideas for implementing a culture of experimentation? Make sure you let me know!

The post How To Build A Culture Of Experimentation appeared first on Blog.

CMOs are Becoming CROs: How to Integrate Marketing and Sales to Actually Drive Revenue

Note: This is a guest article written by David Zheng, the Founder of GrowthWit and WiseMerchant and the Head of Growth at BuildFire.Any and all opinions expressed in the post are David’s. Marketing and sales teams have a reputation for rivalry. Although they work toward the same outcome, each has a different approach. As Chip Doyle once pointed out, marketing wants […]

The post CMOs are Becoming CROs: How to Integrate Marketing and Sales to Actually Drive Revenue appeared first on Blog.

Note: This is a guest article written by David Zheng, the Founder of GrowthWit and WiseMerchant and the Head of Growth at BuildFire.Any and all opinions expressed in the post are David’s.

Marketing and sales teams have a reputation for rivalry.

Although they work toward the same outcome, each has a different approach.

As Chip Doyle once pointed out, marketing wants to tell you what to buy, while sales want to hear why you’re buying it (so they can sell you more).

Marketing requires a one-way communication, while sales require a two-way conversation.

But technology and buyer habits are changing all of that. Marketing is no longer a one-way communication, and both teams are relying more heavily on the other to truly understand what the customer wants. Now every task is a Sales and marketing collaboration.

This also means that roles are changing. Chief Marketing Officers (CMOs) and Chief Revenue Officers (CROs) must find a way to play nice.

How the Relationship between CMO and CRO Is Changing

In the past, CROs were mostly responsible for driving profitability and sustainability. It was the job of the sales team to ensure financial success for the organization.

That typically meant putting people on phones to answer customer questions.

The CMO, on the other hand, was responsible for making sure that people knew about the organization—to gain awareness and find new potential markets for the sales team.

They both have the same ultimate goal, but each takes a different path to get there.

sales and marketing alignment activities flow chart
But the Internet changed all of that.

Where once the salesperson was the most trusted source of information about a given product or service, now shoppers have limitless access to information—product data, customer reviews, and so on.

One search gives them all the answers they need.

Customers also have a myriad of touchpoints with any given company. From social media to email outreach to an online contact form, they no longer have to call only one person to get what they need.

This has shifted the role of the CMO to the forefront.

In today’s digital market, it’s about finding ways to not only make people aware of the brand but also trust the brand’s message in the same way they earlier trusted the salesperson over the phone.

Of course, that doesn’t mean that the CRO is obsolete. Far from it, sales will always matter.

It simply means that the lines between the CRO/CMO are blurring together in a new way.

sales and marketing alignment for communication with customer

Following some of the Sales and marketing alignment best practices, both parties are now responsible for the financial well-being and reputation of the company. If one fails, the other fails too.

It’s more important than ever that these roles find ways to integrate so that both teams produce real, measurable results.

With that in mind, here are 5 best practices for sales and marketing to help them collaborate to drive revenue.

1. Sharing Sales and Marketing Data for Customer Research

Both marketing and sales use targeted buyer personas to inform their strategies.

According to the Data-Driven Marketing Survey by Teradata, 50% of marketers agree that data is the most underutilized asset in their organizations; but less than 10% use the data in a systematic way.

Salespeople have a leg up when it comes to data, as they’re often the first to develop buyer personas to understand their customers better.

But that data isn’t always accessible to the marketing department.

sales and marketing quality data report January 2017

Marketing teams also need these buyer personas to update its strategies.

The team may need to know whether the customer is a Millennial or a Gen X-er (social media or email?), their income level (affordable or luxury?), and any other behavioral drivers (mobile or desktop?) that might drive their purchasing decisions.

Who knows this data better than anyone else? Salespeople.

The sales team has insights into customer’s goals, mindset, and expectations, and potential obstacles to purchasing.

Marketing needs to have this data to create content and advertising that actually works.

Sales and Marketing Persona comic

To build an effective partnership, sales will need to share the following information with marketing:

  • Sales data:
    • Which products are selling well?
    • Which products are faltering?
  • Customer lifetime value:
    • How low or high are retention rates?
    • How long does the average customer stick around?
  • Internal performance metrics:
    • How fast is the turnaround for a product or service?
    • Are there any obvious bottlenecks?

In turn, marketing should share the following data with the sales team:

  • Traffic and engagement:
    • How many visitors are coming to the site? How many are engaging? Where are they coming from?
  • Email marketing: What are the open and click-through rates for each email campaign?
  • Clicks and conversions: What is the conversion rate of sales landing pages? What are the shopping cart abandonment rates?

With each party measuring these metrics, each can proactively adjust its strategies to achieve better results.

Lead flow for sales and marketing alignment

Marketing can see how its ad campaigns affect the lifetime value, or whether the promises are creating more demand than the team can keep up with (causing bottlenecks), for example.

Sales can see whether there is a significant gap in the sales process (too many people are leaving the website without buying!) or whether or not email is still the best outreach source for certain customer segments.

2. Using Sales CRM Data to Inform Marketing Strategies

Timing is critical in sales.

The sales team has a sense of its current month’s forecast (or even the next month’s) when it comes to the revenue. Part of the job of the CRO is to answer the “when” of the sales cycle.

When is the best time to promote a specific product or launch an outreach campaign? When should marketing initiatives be kicked off? When should sales expect to see results?

best time for sales team to contact customers

The marketing team is the “how” and “what.”

How should that product be promoted based on the sales cycle? Is it a seasonal product or available year-round? How will people be made aware of changes to the product? What is the desired outcome?

Sales should have a good idea of when the best time is to launch a new initiative, according to the purchasing data.

Marketing should know what that initiative should be and to whom it should be targeted, as well as the specifics of the time of the day and week (based on engagement metrics).

sales and marketing emails optimized for the best day of the week

Without both teams working in harmony, it’s possible to launch a revolutionary marketing campaign that doesn’t sell any products at a measurable level.

Here’s an example:

Say you have a 25% conversion rate for every step of the sales funnel. If your monthly sales target for the next quarter is $1 million and your average sales are around $10,000, you need around 100 conversions every month to achieve this goal.

But for some months, sales are slower than others.

Let’s assume that January and February are much slower sales months compared to June and July.

By using this information, the marketing team can determine what offer to include for customers during those months (discounts on orders over a certain price point, for example) in their campaigns.

But this means that the sales team needs a reliable way of identifying these trends, like a sales pipeline CRM, and give the marketing team access to this information.

sales pipeline

Sales should know where leads are coming from when the customers are more willing to buy, and what entices the customers the most so that the marketing team knows how to send out the right offer at the right time.

3. Adjusting Ad Campaigns by Using Sales Data

Advertising is one of the main drivers in sales, and one of the main tasks in marketing.

One of the challenges with advertising is that it’s easy for a company to spend more money compared to earn money.

It’s always a risk. You could drop millions on an ad campaign only to see a moderate sales increase. But this risk gap can be closed when sales and marketing work together to produce a certain outcome.

Take PPC advertising, for example.

For a marketer, a successful pay-per-click (PPC) advertising campaign might be the one that just drives engagement.

successful adwords campaign for driving more engagement

If someone clicks a Google PPC ad, goes to the home page, and then clicks through the website, that’s a success.

To that end, marketers may try to use specific keywords to improve website traffic or engagement.

But the sales team cares about one area—sales.

It doesn’t matter if website traffic improves but no qualified leads come from it. They might care if an ad had a high cost-per-click (CPC), and was essentially “ineffective” in producing a real, paying customer.

Sales is looking for revenue, not just metrics.

channel wise breakdown of ROI for marketing

So what does this mean for a partnership between sales and marketing? It means that both have to work together to create the most effective campaigns.

Marketers need to understand the Lead Scoring System (and subsequently, the sales CRM system) so that when they spend money on PPC ads, they know which targeted personas will be most likely to convert.

Both parties need to understand how the marketing funnel works and how it can be combined with the sales funnel to create something new.

new sales and marketing alignment funnelA top-of-the-funnel marketing “lead” (like a website visitor) may not ever turn into a customer, but that doesn’t mean it isn’t important for sales.

The marketing team needs to know how to measure successful campaigns based on sales data, not on just its own metrics.

4. Improving Brand Identity (and Sales) with Marketing

Not everything that impacts sales is measurable.

A study by Harvard Business Journal found that CEOs tend to favor sales over marketing because sales outcomes are often more “tangible.”

As a CEO puts it, “Why should I invest in more marketing when I can get better results by hiring more salespeople?”

Because of this mindset, many marketing teams are underfunded, and, as a result, underperforming.

suggested percentage of revenue that needs to be spent on marketing.

This is a problem because there are many immeasurable entities that can impact your bottom line.

Brand identity, for example, is not measurable by any metric, yet a brand’s reputation can be a key driver of that brand’s equity.

This is also known as the “halo effect,” or a situation when a customer buys from a brand based on its positive reputation, whether or not the product is truly inspirational.

In other words, the value of a brand can be measured by its marketing.

When Apple began marketing the iPod back in 2005, they put millions into advertising. You may remember the campaign.

marketing of apple ipod comic

Even though iPod (and iTunes) sales made up only 39% of Apple’s overall profits that year, by the end of their marketing campaign, they were hailed as a technology leader and revolutionaries.

As a result, its fiscal year sales in 2006 increased 38% and their profits rose by 384%.

It has since leveraged their reputation as tech innovators to create more and better products, making it one of the biggest companies in the world.

And it doesn’t even sell the iPod anymore.

sales of apple ipod year on year

This goes to show that when the marketing team is properly supported, they can produce results worthy of the sales department.

5. Improving Sales Outreach with Marketing Analytics

One of the biggest contention points between sales and marketing is measuring outcomes.

For marketers, a “good” outcome for an email outreach campaign is high click-through and open rates. However, sales don’t care about click-through rates. It cares about sales.

It might be better to measure your outreach campaign multidimensionally.

measuring content marketing valueOn the other hand, you won’t necessarily get sales if no one opens and clicks through the email.

This is where marketing and sales must come together to identify what a successful outreach campaign looks like.

The marketing team should introduce key analytic tools to the sales team.

While the marketing team can also forward crucial data or statistics, at some point, it inevitably will become an issue of “teaching a man to fish.”

Teaching helps as an economical resource

If the marketing team moves ahead based on important information, the sales team might accidentally ignore crucial statistics that can improve its sales strategy, just because they don’t fully understand it.

This can lead to miscommunication and a negative impact on sales.

If the sales team understands how to use the same tools that marketers use; however, it can create a seamless conversation between the two departments and reduce the odds of an essential piece of data being overlooked.

right marketing or sales tool for your job

Even beyond analytics, sales and marketing teams should a discuss other ways to use technology effectively.

For example, if the marketing team intends to produce content for potential customers on LinkedIn, then the sales team should guide it on best practices for targeted leads on that platform.

Marketing can also assist sales in some of its follow-up endeavors.

If the sales team becomes overwhelmed following up on cold email outreach, for example, the sales team can use a tool like Gmass to automate the process and eliminate the burden on the salesperson.

follow up email tools for sales.

This frees up the sales team to focus on metrics that matter rather than chasing down leads.

But if the sales team doesn’t understand how to use Gmail, they might not be automating their follow-up effectively and miss important sales opportunities in the process.

When marketing and sales work together with the same tools, they can maximize efficiency and move customers through the sales funnel as painlessly as possible.


Even though both the teams have notoriously been rivals in the past, it’s time for sales and marketing team to work together.

This process should be made easier with the addition of technologies that improve the marketing/sales relationships (automation tools like Gmass, or analytic tools like Google Analytics).

It’s important for the two teams to remember that when one succeeds, the other succeeds, even if they approach a problem from different angles.

When marketing is successful at getting traffic or open rates, for example, or improving brand reputation, sales will increase.

When sales are successful at closing leads and measuring their data, marketing will be more effective.

When the CMO and the CRO work together, everybody wins.

The post CMOs are Becoming CROs: How to Integrate Marketing and Sales to Actually Drive Revenue appeared first on Blog.