Part 2: Our Top Takeaways from Click Summit 2018

Last week, we shared the first of many takeaways from Click Summit 2018, our annual conference for professionals in digital experimentation and personalization. This week, we’re back with more insights from each impactful conversation, inspired by this year’s edition of Clickaways. 1. Manage the three P’s of scaling your testing program: people, process, prioritization. Many companies […]

The post Part 2: Our Top Takeaways from Click Summit 2018 appeared first on Brooks Bell.

Last week, we shared the first of many takeaways from Click Summit 2018, our annual conference for professionals in digital experimentation and personalization. This week, we’re back with more insights from each impactful conversation, inspired by this year’s edition of Clickaways.

1. Manage the three P’s of scaling your testing program: people, process, prioritization.

Many companies have found it more effective to establish a dedicated optimization team rather than having these duties dispersed across the organization. However, if that’s not possible for you, let your Center of Excellence take the lead on defining key processes, training and developing a maturity model to determine when each team is ready to start testing.

Develop a formal process for submitting, presenting, prioritizing and executing new testing ideas. Using various automation technologies can further simplify these steps.

Additionally, agree to one source of truth for your test results across multiple platforms. Companies that have various groups looking at different data sources struggle to establish the necessary credibility to scale their programs. This is one area where a knowledge platform that houses testing results, insights and ideas (like Brooks Bell’s Illuminate platform, or Optimizely’s Program Management) can help.

Finally, growing your experimentation program comes with the expectation of more tests, executed faster. When determining your velocity goals, be sure to consider quality over quantity. Always prioritize running a few, quality tests over many, low-impact tests.

2. Personalization and optimization teams should remain separate functions with connected but distinct goals.

Personalization is a worthwhile investment for any online industry, but it has to be adopted as a company-wide strategy in order to ensure you’re delivering a consistent customer experience.

To get the most out of your investment, establish a separate personalization team to run your program rather than looking to your existing experimentation team. Here are a few reasons for this: First, personalization is a longer-term strategy and “wins” occur at a much slower rate. Additionally, while there are similarities between A/B testing and personalization technologies, the questions you ask and the answers you get are very different.

Finally, running split tests is inherently easier and faster than implementing personalization. So long as your team is overseeing both functions, they’re likely to focus more on testing than personalization.

3. Focus on organizational outputs and customer insights, not just test outcomes.



Oftentimes, experimentation professionals find themselves nearest to the customer. Sure, you may not speak with them directly, but your work can have a direct effect on your customers’ experience and brand perception. That’s a lot of power, but also a lot of opportunity.

So here’s the challenge: Go beyond simple tests like button color or check out features and consider the bigger picture. Use testing to seek out insights that would be useful for other departments within your organization.

Here at Brooks Bell, we have our own framework for doing this (and we’d be happy to tell you about it). In lieu of our services, we’d encourage you to take a step back from test outcomes, spot trends and use these to develop testable customer theories.

Developing a customer theory requires you to conduct a deeper interpretation of your results–so don’t do it alone. Look to your working team to brainstorm customer theories and additional tests to validate or invalidate those. Bring in additional data sources like NPS, VOC or qualitative research to paint a more detailed picture of your customers.

Doing this can have huge implications for your customers, your experimentation program and your brand overall.

4. Build a program that strikes the perfect balance of innovation and ROI.

In order for creativity to flourish within your experimentation program, you have to establish clear goals. These are used as a framework within which your team can look for opportunities to innovate.

Develop a process for brainstorming test ideas that encourages participation and creative thinking, like using Post-It notes.



Finally, demonstrate a willingness to take calculated risks in order to make room for creativity in your optimization strategy. There is always something to be learned from negative or flat results.

Like the information in this post? Download this year’s Clickaways to access more tips, tricks and ideas from Click Summit 2018.

The post Part 2: Our Top Takeaways from Click Summit 2018 appeared first on Brooks Bell.

Part 1: Our Top Takeaways from Click Summit 2018

Another year, another epically productive Click Summit. In the weeks since Click Summit 2018, we’ve spent some time reflecting on the event and even our heads are still reeling from the depth and quality of each conversation. This event isn’t your run-of-the-mill marketing conference. We strive to create an intimate and super-productive experience in our […]

The post Part 1: Our Top Takeaways from Click Summit 2018 appeared first on Brooks Bell.

Another year, another epically productive Click Summit. In the weeks since Click Summit 2018, we’ve spent some time reflecting on the event and even our heads are still reeling from the depth and quality of each conversation.

This event isn’t your run-of-the-mill marketing conference. We strive to create an intimate and super-productive experience in our small group conversations. Of course, the true credit goes to our attendees and moderators for their candid participation. It takes a certain level of vulnerability to look to others for feedback and direction. Those types of conversations are where the true insights come to light.

Had to sit out Click Summit this year? You’re in luck. We’ve compiled the key takeaways from each of the 22 thought-provoking conversations into an easy-to-read, downloadable resource.

Here’s our summary of some of the insights you’ll find in this year’s Clickaways

1. Relationships are key to creating buy-in for experimentation. Get to the right meetings and make the right connections. Target influential leaders to gain traction and credibility for your program. Build working partnerships with other teams, taking time to understand their goals. Work with them to make testing and personalization part of the solution.



Finally, know that proving people wrong doesn’t create buy-in. Rather, invite other departments to participate in your program and frame your tests as an opportunity to learn together. Hold monthly or bi-weekly meetings with direct and indirect stakeholders to review test wins, brainstorm new tests and discuss any resulting customer insights.

2. Instill testing in your company culture through establishing a credible team and program. Trust is easily lost, so you really need to take steps to ensure your team is positioned as a source of truth for the business, rather than one that’s encroaching on other departments. Your team should not only be experts in optimization and behavioral economics, but also experts in your customers–know their behaviors online, what motivates them and what truly makes them tick.

Hold training sessions on best practices for testing, personalization and customer insights. Regularly communicate test results and any subsequent insights to the entire company. And when sharing results, consider your audience. It may be worth creating different reporting formats for different stakeholders

3. If you want to build an army of optimization evangelists, you’ve gotta get everyone on the same page first. So long as end-to-end optimization requires working across multiple teams, it’s important that you establish clear processes and governance. Develop a common language for testing terminology; abandon jargon in favor of words that are easy to understand and don’t have multiple contexts.

Set clear rules of engagement and expectations between all teams involved in optimization. This includes engineering, IT, analytics, marketing, creative and others. Make sure communication and reporting processes are defined and any associated technologies are being used consistently.Finally, take into account how success is measured for all these other stakeholders. Not all teams are incentivized with revenue targets or conversion goals. Connect your test strategy to their objectives to ensure a unified vision.

Like the information in this post? Stay tuned for part two next week. Until then, download this year’s Clickaways to access more tips, tricks and ideas from Click Summit 2018.

The post Part 1: Our Top Takeaways from Click Summit 2018 appeared first on Brooks Bell.

Testing Your App Listing in the Google Play Store

Recently, while attending a native app session at our annual conference, Click Summit, it was brought to my attention that not many people know about the ability to run A/B tests on their Google Play Store app listing.   Testing your store listing can be an untapped area for gaining key insights about your customers and […]

The post Testing Your App Listing in the Google Play Store appeared first on Brooks Bell.

Recently, while attending a native app session at our annual conference, Click Summit, it was brought to my attention that not many people know about the ability to run A/B tests on their Google Play Store app listing.  

Testing your store listing can be an untapped area for gaining key insights about your customers and increasing app installs. Additionally, the insights you gain by testing your Google Play Store listings could be transferable to your Apple App Store listing as well.

Within the Google Play Console, there’s a little known tool that will allow you to A/B test your app listing called “store listing experiments.” This can be found under the “store presence” menu item.  

There is no need for a technical resources or technical knowledge as the technology is built right into the Google Play Console. The Store Listing Experiments feature allows you to A/B Test six different attributes of the store listing: Hi-res icon, Feature Graphic, Screenshots, Promo Video, Short Description and Long Description. Tests can include all of these in combination or individually. You can run tests globally (graphics only) or localized (text and graphics). Note that you are limited to 3 variations in a test.

The analytics and reporting is all housed within the Google Play Console and unfortunately, cannot be exported. Three metrics area automatically tracked: Installs on active devices, installs by user, uninstalls by user. Results are measured at a 90% confidence interval.

For more details, check out Google’s step by step documentation.

When it comes to experimentation, Brooks Bell is happy to lend our expertise to help your optimization program expand its reach, capabilities, and impact. This can include testing store listings, to landing pages, to check out experiences and more. If you’re interested in learning more about Brooks Bell and how we can help optimize your web experiences, contact us today.

The post Testing Your App Listing in the Google Play Store appeared first on Brooks Bell.

Who’s Hiring in May?

Interested in optimization, personalization and testing?  There are a variety of positions to choose from, whether you have coding experience or love analyzing everything in this world! Here are our picks: QA Engineer/Senior QA Engineer – American Express is looking for a “QA Engineer to work on mobile applications that will be used by tens of millions […]

The post Who’s Hiring in May? appeared first on Brooks Bell.

Interested in optimization, personalization and testing?  There are a variety of positions to choose from, whether you have coding experience or love analyzing everything in this world!

Here are our picks:

QA Engineer/Senior QA Engineer – American Express is looking for a “QA Engineer to work on mobile applications that will be used by tens of millions of cardholders around the world” in New York.

Manager of Testing and Optimization – Join GoDaddy’s teams in Kirkland, Washington or Tempe, Arizona in “translating performance and customer data into insights and ultimately business strategy, working with business teams to find data-driven optimization opportunities and being a driving force in developing a playbook for test driven business improvement.”

Ultra Mobile is looking for an a number of amazing people to join their ecommerce team in Costa Mesa California!
Ecommerce Analyst – Contribute to “ecommerce analysis and optimization efforts by finding and sharing insights in web data.”
Senior Software Engineer, Ecommerce – Looking for someone with front-end and back-end development skills to “contribute to software design and delivery for current and future products.”
Software Engineer, Ecommerce – Working in a dynamic start-up environment in front-end and back-end development.
CRO Developer – Seeking a candidate to “work within Ultra Mobile’s testing tool to develop winning customer experiences and tests that matches customer experience goals and provides a best-practice online experience.”

Software Developer, Experimentation Platform – Hulu’s Experimentation Services and Platform team is seeking a “Software Developer who will contribute to building internal products that enable and promote test driven product development” in Seattle, Washington.

Analyst, User Experience – UX Designer – MasterCard is “seeking a UX designer to join their Experience Design team” in New York. “Working with fellow designers, you’ll design, improve, and develop products and experiences for desktop and mobile devices.”

Digital & eCommerce Insights Manager – Join Autodesk in San Francisco as part of the Digital and eCommerce Analytics team, being “responsible for reporting, analytics and insights for the Digital and eCommerce (DEC) organization at Autodesk to help connect the dots across the various digital customer touch points.”

Web Personalization Manager – Help Bed, Bath & Beyond “drive their personalization efforts in the areas of product recommendations, content and offer personalization and be the subject matter expert for personalization on all company sites” in Union, New Jersey.

Have a job opening in the optimization space?  Get in touch and we’ll post it next month!

The Brooks Bell Team

The post Who’s Hiring in May? appeared first on Brooks Bell.

How to Use Personalization to Enhance Your Existing Optimization Program

We know an A/B test can lead to powerful insights. However, the information gained from traditional A/B tests tends to be focused on what’s best for the majority of users – not every individual user. That’s where personalization comes in. Personalization enables you to leverage the specific wants and needs of each individual user on […]

The post How to Use Personalization to Enhance Your Existing Optimization Program appeared first on Brooks Bell.

We know an A/B test can lead to powerful insights. However, the information gained from traditional A/B tests tends to be focused on what’s best for the majority of users – not every individual user. That’s where personalization comes in.

Personalization enables you to leverage the specific wants and needs of each individual user on your site. This can lead to even more substantial results–higher conversion rates, deeper engagement with your site, and increased revenue.

Many of the traditional testing tools–Adobe Target, Optimizely and Maxymiser–have personalization capabilities available. There has also been an emergence of companies like Dynamic Yield and Evergage, which offer personalization technology as their core focus.

As technology in this space improves, personalization has become a major focus for many Brooks Bell clients. However, the question we’re often asked is not whether to implement personalization alongside existing optimization efforts – rather, its how to do this.

Luckily, there are many ways to do just that. For the purposes of this blog post, we’ve outlined two relatively simple strategies for implementing personalization alongside your existing optimization program.

Strategy 1: Rule-based targeting

Rule based targeting is a personalization technique that’s available on most A/B testing platforms. Instead of targeting all users, you select a specific segment of users to target an experience to: new or returning users; mobile or desktop users; or users in a specific location.

Because these different types of users are interacting with your site differently, you’ll likely see higher returns by personalizing your content to each group.

You can also apply rule-based targeting after running a traditional A/B test, by breaking down your results by those specific user segments. In doing so, you may find that a “winning” homepage experience performed very well among new users, but was flat for returning users.

Though pushing the winning variation live to all users would increase revenue, you might see a bigger increase if you were to push it live to new users only. This gives way to additional opportunities to test different strategies for returning visitors.

Strategy 2: Predictive personalization

Many testing platforms now offer predictive personalization, which works in real time to learn which experiences are ideal for certain types of users.

A predictive personalization “test” runs indefinitely – and adjusts as users’ preferences change over time, showing the optimal experience to each user.

Predictive targeting technology is exciting for many reasons. It accounts for the fact that a winner from a year ago might not be the best option for your users now.  

The technology also makes it easier to figure out the best option for short term website changes, like a holiday promotion–for which A/B testing is not a viable option due to time constraints.

Additionally, having the ability to step back and leave the analysis to the computer – instead of spending the time analyzing data yourself – is a huge benefit to experimentation professionals and the companies they work for.  

There are, of course, potential pitfalls to this form of personalization.

When you run a traditional A/B test with a clear winner across all users, it’s easy to make the decision to build the winning code into your site. However, with predictive personalization, you may have many different versions of a page for different segments of users, and continue relying on the testing tool to deliver the code, never building it into your site.

This can be risky for a few reasons: it can increase load time; and if, over time, other updates are made to your site, those updates could break the experience.

Additionally, you’ll also want to make sure you trust that the machine learning algorithms are actually making the best decisions for your users. To that end, many platforms offer a control experience which segments users randomly. You can then compare metrics from the control against the personalized segments to ensure the algorithm is working optimally.

Personalization offers the opportunity to gain new insights about your users and deliver the most valuable content for each individual. Incorporating personalization into your testing program is certainly worth the investment, with the potential for huge rewards.

At Brooks Bell, our Personalization Jumpstart program enables enterprise optimization teams to incorporate and scale personalization strategies into their existing optimization programs. To learn more about our services, contact us today.

The post How to Use Personalization to Enhance Your Existing Optimization Program appeared first on Brooks Bell.

Delivering a Winning Customer Experience Through 1-to-1 Personalization

As most online retailers today recognize, a personalized shopping experience is a must-have. Nearly nine out of 10 marketers (88%) say their customers expect individualized experiences. And according to Gartner, by the end of this year, organizations that have fully invested in online personalization will outsell those that haven’t by more than 30%. Yet even […]

The post Delivering a Winning Customer Experience Through 1-to-1 Personalization appeared first on Brooks Bell.

As most online retailers today recognize, a personalized shopping experience is a must-have. Nearly nine out of 10 marketers (88%) say their customers expect individualized experiences. And according to Gartner, by the end of this year, organizations that have fully invested in online personalization will outsell those that haven’t by more than 30%.

Yet even though personalization is becoming the norm, particularly in e-commerce, we’ve all had those obnoxious experiences where a brand that should “know” us completely misses the mark. Your favorite retailer may email you about a big sale in a category you’ve never shown interest in (e.g., baby clothes if you’re not a parent, or lawn care if you live in a high-rise apartment). In your work life, you’ve likely gone to a company’s website, only to be hit with a promotion for a report you’ve already downloaded.

In the past, shoppers might not have batted an eye at these lackluster experiences, because all of their experiences looked like that. But that’s not the case today. With a wide array of options, consumers today can (and do!) take their business to places where they feel recognized, appreciated and valued as an individual.

So retailers have to do better.

Moving Up the Personalization Maturity Curve

Applying personalization across all relevant channels – so shoppers are recognized and can pick up where they left off –  should be the goal. To do so, companies need to be able to: 1) track an individual’s behavior across different channels; 2) merge that information with pertinent customer data from other systems; 3) automatically interpret the data to determine affinities and intent; 4) house everything in a central place – creating a single, unified profile for each person; and 5) act on all of the data in real time.

But no company will be able to do all of this from the start. It’s important to crawl before you walk, and walk before you run.  

The most important element – and an essential part of all five steps – is, of course, data. You can’t personalize an experience for an individual if you don’t know her. Get started by leveraging a next-generation personalization platform to start bringing in deep, contextual, real-time, accurate behavioral data from one digital channel, like your website. You can use this data to start ensuring that your shopping experiences in that channel are truly individualized.

Then, you can begin incorporating more data sources like in-store purchase data. And, tie in other digital channels like email and mobile. And, when you have the strategy, process and technology in place, move ahead to personalizing your call-center and in-store interactions.

Machine Learning Powers 1-to-1 Experiences

At Click Summit 2018, I’m looking forward to discussing the nuances of a winning personalization strategy — particularly when it comes to using machine learning to provide a unique experience for every shopper. It’s a topic I am passionate about; in fact, I devoted my new book to exploring this topic in detail.

When we discuss machine learning in the context of personalization, it means using computers to process vast amounts of data, in milliseconds, to make the best decision about what to show each person. Machine learning puts the vision of “The One to One Future,” which renowned customer experience experts Don Peppers and Martha Rogers, Ph.D., predicted in 1993, truly (and finally!) within reach – accessible to businesses of all sizes and across industries.

This is not something reserved solely for advanced personalizers. It can be a part of your crawl/walk/run strategy.

E-commerce marketers should already recognize the value of algorithms for product recommendations. Today, many retailers recommend products based on what other shoppers viewed or purchased. This may be helpful, but it is not individualized. These same retailers are accumulating a ton of information about each of their shoppers all the time.

By observing what a shopper is viewing and how she is engaging with pages on your site, you can infer her favorite brands or categories, her preferred price points, the colors that tend to grab her attention the most, and more. Machine learning can take all of this information, interpret it, and use it to determine the most relevant product recommendations for her specifically — not anyone else.

But machine learning can be used for more than just product recommendations on PDPs. Search results, category pages, site navigation and more can all be powered with machine-learning algorithms.

And with the right platform, algorithms can be created, customized and managed by marketers and other business users – no need to cede control to a “black box,” and create a large, time-consuming IT project.

Click Summit

I look forward to discussing 1-to-1 personalization and machine learning more at Click Summit 2018 in the session, “Goals, Use Cases, Techniques: How Personalization Changes Things.” In this session, you’ll learn about the nuances of a winning personalization strategy, and how to put it into practice with a group exercise on a sample website.

The post Delivering a Winning Customer Experience Through 1-to-1 Personalization appeared first on Brooks Bell.

Who’s Hiring in April?

Here’s some of our picks: Personalization, Testing and Optimization Manager – Webster Bank in Waterbury, Connecticut is looking for someone to join as “a member of the Digital Marketing Team to lead the strategy, design and implementation of personalization and testing that will drive new account growth and deepening of existing relationships.” Full stack Web Development […]

The post Who’s Hiring in April? appeared first on Brooks Bell.

Careers in Testing image

Here’s some of our picks:

Personalization, Testing and Optimization Manager – Webster Bank in Waterbury, Connecticut is looking for someone to join as “a member of the Digital Marketing Team to lead the strategy, design and implementation of personalization and testing that will drive new account growth and deepening of existing relationships.”

Full stack Web Development Engineer – Join American Express in Phoenix, Arizona and “play a key role in the understanding of product owner strategy and collaborate with his/her peers, technology partners, and product owners to translate complex user stories into successful product releases.”

Web Personalization Manager – Located in Union, New Jersey, Bed Bath & Beyond is looking for “a talented and experienced manager who can contribute to the success of our web analytics initiatives.”

JavaScript Developer – Brooks Bell is looking for a JavaScript Developer to work closely with the execution and consulting teams in Raleigh, North Carolina.  You will be responsible for efficiently coding test variations on complex sites along with interacting directly with clients.

Sr. Website Optimization Manager / Analytics Engineer Testing & Optimization – LuckyVitamin is looking for a “senior manager will utilize their 10+ years of e-commerce and site analytics experience to craft testing and optimization plans that target the most impactful website KPIs, in Conshohocken, Pennsylvania.

Data Scientist – Advanced Auto Parts is looking for a candidate to analyze data sources and transaction to help improve business goals.  Responsibilities include finding interpreting information and building predictive models to help identify actionable insights and customer behavior.

Senior Manager, Digital Analytics and Optimization – Per Kohl’s, “Own the methodology, strategy, and execution for digital data collection, analysis and site testing. Lead team that provides data driven recommendations that solve for key pain points and experience optimization.”

Manager, Marketing & Web Analytics – New York & Company is looking for a candidate in New York to “understand customer behavior to influence business decisions and support business strategies.  This includes analyzing the health of the customer file, identifying customer segments to target with specific content or offers, understanding email performance, and uncovering data driven insights overall in an effort to better retain and reactivate buyers.”

Manager of Digital Testing & Optimization, Analytics – In Reynoldsburg, Ohio, L Brands is looking for a someone to “champion the advancement of their testing and optimization capabilities and be viewed as the optimization evangelist for their different brand partners.”

Have a job opening in the optimization space?  Get in touch and we’ll post it next month!

The Brooks Bell Team

The post Who’s Hiring in April? appeared first on Brooks Bell.