The Expert’s Guide to A/B Testing During the Holiday Season

In 2016, online spending topped in-store shopping for the first time ever. That trend continued in 2017, with Adobe Digital Insights reporting that 2017 holiday sales surpassed $91.7 billion, marking 11% YoY growth. Peak season offers peak opportunities for experimentation programs. Increased traffic and conversion rates open the door for higher velocity, shorter durations, and lower minimum […]

The post The Expert’s Guide to A/B Testing During the Holiday Season appeared first on Brooks Bell.

In 2016, online spending topped in-store shopping for the first time ever. That trend continued in 2017, with Adobe Digital Insights reporting that 2017 holiday sales surpassed $91.7 billion, marking 11% YoY growth.

Peak season offers peak opportunities for experimentation programs. Increased traffic and conversion rates open the door for higher velocity, shorter durations, and lower minimum detectable lifts without compromising statistical significance.

If you haven’t already created your experimentation strategy, the time is now. But here are some essential factors to consider while creating your holiday testing game plan.

Maximize Your Holiday Window
Thanksgiving Day kicks off the peak holiday season, which continues through December 23. If you know your holiday window and website traffic patterns and expectations, you’ve got what it takes to take full advantage of this opportunity.

It can get complicated, but here’s a simple way to start:

  1. Define your holiday window. Consult past data to determine when traffic and conversion increases start and stop.
  2. Layer in the changes your organization is forecasting over last year. For example, one of our clients is expecting a five percent increase in traffic over last year’s holiday season. That intel is reflected in our traffic assumptions.
  3. Start your roadmap with the most valuable pages so that early wins can positively impact the rest of the holiday season. Create a punch list of pages with this in mind.
  4. Use traffic assumptions, desired statistical significance, and minimum detectable lift to determine the sample size and duration of tests.
  5. Continue this process to fill the window of time. Use these dates to mobilize your team, communicating key dates of test strategy kickoff, when tests will move into development, when they will launch and end and when results will be shared.

Communication is Critical
Since the holiday season represents a large portion of annual revenue, stress and emotions run high. As a result, it’s important to create your communication plan in advance. Determine who your stakeholders are, the optimal frequency of updates and what information needs to be shared. This isn’t the time for surprises or big reveals, so plan to devote a chunk of time to telling the story of your program and communicating its value.

The Weather Outside May Not Be the Only Freeze You’re Experiencing

Some organizations implement a freeze on development code updates and changes during the holiday season to avoid the risk of broken digital experiences or performance disruptions.  Get acclimated with your company’s approach so you can have a plan for implementing winning test programs.

The ideal scenario is to push winners immediately into production. Based on years of experience with enterprise clients, Brooks Bell strongly advocates this approach so you can maximize the impact of that winning test.

If production updates aren’t on the table because of a code freeze, don’t immediately jump to the decision to push the winner to 100 percent through your testing tool. Though it sounds like the best way to manage through code freezes, it could cause delays and create an undesirable experience. Before you make the decision how to handle, get your organization’s development experts involved to help you evaluate the risks and rewards.

Holiday Shoppers are Different
Think about your own shopping behaviors during the holidays compared to the rest of the year.

When I’m shopping during the holidays, I find myself on a mission to knock out my shopping list. As the countdown clock ticks away in my brain (and often literally on websites), I have a very real and intense sense of urgency. For me, customer confidence indicators, obvious savings and a clear and easy path to checkout are the ticket.

During the rest of the year, shopping is more leisurely for me and allows time for more browsing and consideration. I may even visit a website a few times before making a purchase. I zoom in on product details. I read customer reviews. I have more time, and the only restrictions are my own.

I’m the same person but have a very different mindset. The same goes for your customers. Keep this in mind as you develop your holiday testing roadmap.

Here are four tips to help ensure your holiday experimentation wins continue to add business value:

  • Keep it simple. As illustrated in my example above, successful holiday strategies are frequently based on a streamlined path to purchase, removing any friction and creating a sense of urgency and scarcity.
  • Test your hypotheses again after the holiday season. Do these experiences still produce a conversion lift when the holiday rush isn’t in full effect? If not, it’s okay! It’s an important learning you can use to build your Holiday/Non-Holiday playbook to make each holiday season better than the last.
  • Know your “Out of Stock” strategy. Regardless of what changes you make to your Product Detail page, nothing zaps excitement out of a customer experience faster than something being Out of Stock. Understand how your site handles Out of Stock messages, such as using red copy or suggesting alternate options. If it’s less than optimal, do some early testing to determine the most effective messaging. If your site includes a lot of Out of Stock product, it’s even more important to make sure it‘s been optimized.
  • Document your findings. Carve out time to tell the story of your testing through the chaos of increased velocity. The data and insights will be helpful after the rush and can greatly influence your future program success. Be sure to look at new, returning and loyal segments, and evaluate the differences in their holiday and non-holiday shopping behaviors.

For more intel on how to make the most of the merry months ahead, download our white paper, “5 Testing Tips for the Holidays.”

Need help developing a game plan for holiday testing? Contact us today!

The post The Expert’s Guide to A/B Testing During the Holiday Season appeared first on Brooks Bell.

Changing Your A/B Testing Software? Read These Tips First.

With the number of testing and personalization tools available, it can be difficult to choose one to invest in. But once you’ve already selected a software, making the decision to transition to a new tool altogether can feel overwhelming. But this happens quite often. For many clients, cost is often the deciding factor in making […]

The post Changing Your A/B Testing Software? Read These Tips First. appeared first on Brooks Bell.

With the number of testing and personalization tools available, it can be difficult to choose one to invest in. But once you’ve already selected a software, making the decision to transition to a new tool altogether can feel overwhelming.

But this happens quite often. For many clients, cost is often the deciding factor in making the decision to switch testing tools–there are a few testing tools that offer similar capabilities at a lower price point. On the flip side, if you’ve increased your program budget and capabilities, it may be time for an upgrade.

And although all testing tools offer similar functions, each has unique features that are important to consider. Personalization, for example, has become a point of focus for many testing programs – perhaps you’re interested in transitioning to a tool such as Evergage or Dynamic Yield that puts personalization at the forefront. Or your testing program has enough velocity to run multiple experiments simultaneously, and you feel you’d make good use of Optimizely’s built-in mutually exclusive experiments feature. Maybe your company uses other Adobe products, like Adobe Experience Manager, so you feel Adobe Target is a good fit.

Regardless of which tool you select, once you select a new software–the next major obstacle is implementing it. Here are our tips for going about the process:

First, examine your testing roadmap.

Take inventory of the tests that will be running close to the date when you plan to stop using your previous tool. Make sure they will have reached significance and be ready to be turned off before you lose access. 

If your budget allows for it, we recommend giving your team a period of time where both tools are available. This will ensure your testing cadence isn’t affected while your team gets up to speed on using the new tool and allows you to transition more seamlessly – you’ll be able to let current tests run their course in the old tool while launching new ones in the new tool.

Then, test your testing software.

While you might be excited to dive in and start launching tests left and right, it’s important to take the time to ensure your new tool is implemented correctly.

Run a QA test that visually changes the page to check that the code is being delivered and the flicker looks reasonable. If there are a lot of flickers, you may need to move the testing tool tag higher up in the head of your HTML.

We also recommend running a live test without visual changes, just for the purpose of checking metrics. This enables your analyst to see that metrics are being tracked correctly within the testing tool, or if you’re using an outside analytics tool, that those metrics are being passed accurately to it. 

Once you’ve confirmed that visual changes are showing up as expected and metrics are tracking accurately, you’re ready to start using your new tool!

Switching testing software comes with its challenges. However, in the right circumstance, switching can offer substantial benefits to your testing program. Taking the time to pinpoint your reasons for switching, plan your testing roadmap carefully around the transition, and having patience as the new tool is implemented will ensure your tool transition goes smoothly.


Brooks Bell has over 15 years of experience working with enterprise brands to establish and scale their experimentation programs. We take a holistic approach to our technical diagnostics and analytics services, providing technology and data recommendations based on your business, your goals, your team, and your unique challenges.

What can Brooks Bell do for you?
✓   Clean, organize and centralize your customer data.
✓   Help you select the right a/b testing and personalization tools.
✓   Ensure your tools and systems integrate with one another.
✓   Train your developers and analysts.

Contact us to learn more.

The post Changing Your A/B Testing Software? Read These Tips First. appeared first on Brooks Bell.

Who’s Hiring in September?

Pumpkin spice is not the only thing in surplus this month, take a look at some job postings around experimentation and personalization. Here are our picks: Director, Digital Strategy – Universal Orlando is looking for a Director to “champion the consumer’s journey across channels to achieve business and campaign objectives and collaborates with leaders of […]

The post Who’s Hiring in September? appeared first on Brooks Bell.


Pumpkin spice is not the only thing in surplus this month, take a look at some job postings around experimentation and personalization.

Here are our picks:

Director, Digital Strategy – Universal Orlando is looking for a Director to “champion the consumer’s journey across channels to achieve business and campaign objectives and collaborates with leaders of non-digital channels to ideate and recommend campaign integration opportunities.”

Senior Web Experimentation Lead – The marketing experimentation team at esurance is looking for a leader to “embed an experimentation culture into the esurance DNA in San Francisco. This role will deliver increased cost savings, additional revenue and industry leading user experiences through the power of site testing technology and the scientific rigor of controlled experimentation.”

Senior User Experience & Small Business Project Manager –  Lenovo is seeking a candidate in Raleigh, NC to drive “UX projects to improve the online customer experience for Lenovo.com globally. The project manager will manage the identification, conception, definition, design, testing and implementation of UX projects with the goal of improving the customer experience, online engagement and purchase conversion.”

Manager of Digital Testing & Optimization, Analytics – Join the digital analytics and optimization team at L Brands in Reynoldsburg, Ohio and “lead digital testing and optimization efforts. This person will champion the advancement of testing and optimization capabilities and be viewed as the optimization evangelist for different brand partners.”

Digital Marketing Manager, Personalization – looking for an ambitious learner to lead a test & learn strategy through experimentation for our digital marketing channels. You will be the leader and subject matter expert of A/B testing with the goal of developing the strategy and approach on personalization.

E-Commerce & Digital Operations Manager – In New York, Zacharys Fine Wine is looking for a candidate to plan and execute “digital and website activities for retail including: content, merchandising, landing pages, site search, product recommendations, personalization, loyalty and other on-site conversion optimization tools.”

Sr. Integrated Marketing Manager – Web Analyst – Microsoft in Redmond, Washington is looking to fill a role to “work with the web lead to strategize, create, manage, execute and optimize web analytics. This includes building experimentation and personalization programs for Dynamics 365 and Power BI.”

Director, eCommerce – “Drive the strategy, development, implementation, and continued improvement of the eCommerce booking experience for Carnival Cruise Line” in Miami, Florida.  “Help lead the presentation across the eCommerce website and mobile app, supporting the integrated programs, promotions and initiatives across the organization.”

Sr Analyst A/B Testing & Site Optimization – Help “drive and support A/B and multivariate testing initiatives on the Homedepot.com site” in Atlanta, Georgia. “The Sr Analyst will be responsible for statistical design, analysis, and reporting aimed at the continued improvement of Homedepot.com onsite experience, with a focus on partnership for making data-driven decisions that drive improved conversion.”

User Experience (UX) Designer – Join the Brooks Bell’s UX team in Raleigh, North Carolina.  “The core function of this role is to research, concept, design, user test, and produce all files needed to execute A/B tests for our clients. This includes creating digital assets that are consistent with the development team’s standards and templates, as well as selecting images, designing layouts, and creating digital experiences that answer user issues outlined by our digital analytics and user research sessions.”

Trying to fill a position in testing and optimization? Send us your posting and we’ll include it on our next post!

The post Who’s Hiring in September? appeared first on Brooks Bell.

Don’t Make It Weird: 5 Tips for Balancing Privacy & Personalization

Imagine a simple scenario: Your coworkers are participating in a fun run for charity and want you to join. You’re up for it, but you know you need a decent pair of running shoes. The logical solution is to go online, search for information about running shoes and identify a few possible options. You could […]

The post Don’t Make It Weird: 5 Tips for Balancing Privacy & Personalization appeared first on Brooks Bell.

Imagine a simple scenario: Your coworkers are participating in a fun run for charity and want you to join. You’re up for it, but you know you need a decent pair of running shoes.

The logical solution is to go online, search for information about running shoes and identify a few possible options. You could order the shoes from an online retailer, but because proper fit is important for running shoes, you decide to visit a specialty retailer at the mall. A salesperson there is friendly and knowledgeable. The store has a pair of shoes you like, in your size. They’re a bit more expensive, but the fitting service added value and there’s no additional shipping cost, so you purchase the shoes on the spot. The next weekend, you run the race and the shoes feel great.

This illustrates a relatively traditional model of consumer decision-making. It begins with a spark that motivates a search for a product. It leads to a research phase, and a consideration set is developed. It then progresses to some type of product experience that narrows the consideration set. Ultimately, a purchase decision occurs and an evaluation of the final product is made.

But today’s online customers may notice a glaring omission from the process: It occurs a week after the run, when you visit a news website. There, in the right column of the page, is an ad for a pair of running shoes. The ad is tailored to your expressed preferences, but not personalized enough to know that a purchase has already occurred.

Seeing these ads, which follow us around the web, can be annoying, unnerving, and even potentially embarrassing. Because the targeting is so crude, it’s obvious that we’ve exchanged some degree of privacy for a marginal—in this case questionable—convenience. And, if this exchange has happened so frictionlessly with one online retailer, how often is it happening elsewhere?

If you can relate to this, you’re not alone. Research has found that consumers generally dislike targeted and personalized advertising. So if personalization makes customers uncomfortable, does this means brands should stop using tailored messages, offers and experiences?

The answer is, decisively, no.

Here’s why: the same body of literature that outlines a negative attitude towards personalization, also highlights the undeniable benefits of personalization. When an ad or message—such as an email subject line—is tailored, even superficially, there is almost always an increase in engagement with the subsequent content.

This contradiction is known as the “privacy paradox.” Consumers are willing to make a long-term trade of personal privacy in exchange for a short-term benefit or convenience, like more relevant advertising or a more specific shopping experience.

But while attitudes toward privacy may contradict behavior, they certainly shouldn’t be ignored. When an ad, message, or experience feels intrusive or creepy, it can diminish the effect personalization could have on your customer and their overall perception of your brand.

Luckily, there are many ways to deliver personalized experiences while also making your customers feel more at ease about their privacy. Here are our tips.

1. Be transparent

Numerous studies have found that the more transparently personalized content is presented, the more effective—and importantly, the more broadly effective—it is.

While making explicit references to data collection and sharing policies can increase privacy concerns, it can also diminish the effect the concern has on consumer behavior.

In an extreme example, Facebook somewhat-recently rolling out a new way for users to see their ad preferences, after the company’s advertisement platform and practices faced scrutiny following the 2016 Presidential Election.

But it doesn’t have to be that complicated. For instance, simply including ad security icons, for example, has been shown to increase the effectiveness of tailored ads even when the icon is unrecognized.

In addition, referencing privacy policies can diminish concerns over data sharing and personalization, even if consumers never read the policy. One study found that consumers interpret a privacy policy as a blank slate populated with all the usual safeguards. This means, of course, the burden is on your company to draft and enforce a responsible privacy policy whenever consumer data is being collected.

2. Be public about your data security efforts

Unsurprisingly, reassuring your customers of data security and describing the efforts you’re taking to protect their data can make them feel more at ease. But the effectiveness of this approach really depends on how much your customers trust your brand and your site. Building this relationship is difficult and can be easily destroyed.

But, if your brand has built a relationship of trust over time and is authentically dedicated to preserving this relationship, referencing the care you have taken to secure private information can not only be a boon to overall perceptions but increase the effectiveness of personalization.

3. Be personal to the right people, at the right time.

In marketing, timing is everything; and the same goes for personalization, it turns out. In e-commerce, personalization is most effective when your customer has established a consideration set and a final decision is about to be made. Additionally, as your customer engages more with a product category or brand, they begin to expect and look forward to a more targeted, relevant experience.  

4. Let newer customers opt-in to personalization

When it comes to moving customers toward a purchase, personalization is more effective in the “pull” direction than in the “push” direction. This means that you should implement personalization with more loyal customers and especially those who have requested more tailored experiences.

For new customers, we suggest waiting to provide personalization until a visitor has shown a specific interest in your company or product: they’ve viewed a few pages on a website, downloaded your app or signed up for your email newsletter. Once this happens, offer a dialog asking “Would you like a more personal shopping experience?”

While it’s true that many visitors may choose to continue on their own, others may not. This also gives you an early opportunity to show your brands’ interest in providing a relevant, convenient shopping experience, which may come into play later once they become loyal customers.

5. Let your customers run the show

Perhaps the most unsettling recommendation for balancing privacy and personalization is to give up some control over the degree of personalization consumers experience. Doing so evokes many of the tips we’ve already covered: it improves transparency, allows consumers to opt into personalization, and helps to build trust. Additionally, offering this service has been found to dramatically improve the effectiveness of personalization, even when some customers actually change settings beyond the default.

Personalization is a powerful tool. The effect personalized messages and experiences have on customers, however, is variable and possibly unpredictable. It’s important that companies balance concerns for privacy and general feelings of intrusion when delivering personalized experiences. Testing these approaches we’ve outlined above will help make your personalization efforts feel less creepy and ultimately, increase the effectiveness of the customer experience.


Transform your customer experience through personalization.

Brooks Bells’ Personalization Jumpstart Program uses a comprehensive, five-step process to help top brands incorporate personalization across their customer experience. Learn more today >> 

The post Don’t Make It Weird: 5 Tips for Balancing Privacy & Personalization appeared first on Brooks Bell.

You’ve got a winner – now what?

Winner winner chicken dinner! Discovering a winning variation is one of the most exciting moments for an optimization program. It’s the moment when all the work that went into creating a test finally pays off. But while you should always take time to celebrate your team’s accomplishment, hold off on busting out the champagne just […]

The post You’ve got a winner – now what? appeared first on Brooks Bell.

Winner winner chicken dinner! Discovering a winning variation is one of the most exciting moments for an optimization program. It’s the moment when all the work that went into creating a test finally pays off. But while you should always take time to celebrate your team’s accomplishment, hold off on busting out the champagne just yet. Your work has really only just begun.

Winning A/B tests can tell you a lot about your customers—what’s important to them, and why they respond the way they do. These results also enable you to quantitatively predict the impact it will have on your business’s bottom line (typically revenue), and project what that impact looks like over the next year.

Once you attribute a value to a winning experience, it’s critical that you also get the experience live on your site. This ensures you’re not leaving money on the table and also maximizes the impact of your testing program.

But to do this, you and your engineering team have to be on the same page. That is, you have to not only understand the way they work, but you also have to deliberately establish a process for implementing winners into your code base.

Most engineering teams operate using the Agile Method.
If you’re unfamiliar with Agile…well, first, what rock have you been living under? (Just kidding. But really?) Agile is a project management method that relies on incremental, iterative work sequences called sprints. For website developers and engineers, shorter sprints usually last 1-2 weeks and longer sprints last 3-4 weeks.

Most Agile engineering teams organize their projects by way of a prioritized backlog. This backlog is often managed by the product team, though other teams can request additions as needed. During each sprint, developers will work to add features and make other site updates based on what’s listed in the backlog.

During a sprint planning meeting, it’s important that you communicate the importance and impact of your winning experience. The higher the impact, the higher the priority, and the more likely it’ll be included in the upcoming sprint.

Of course, delays are common; particularly when your shared development resources are balancing many different priorities.

As an interim fix, you can use your testing tool to push the winner to production.
To do this safely, end the test campaign and duplicate the code into a new campaign, allocating 100% of traffic to the winner. We advise this method because pushing the winner through the original test campaign would risk displaying the losing experience to returning visitors who previously qualified for that experience.

Of course, there are risks to using a testing tool in this way—even if it’s only a short-term solution. While you might be able to cash-in quickly on your winning test, you could also face interference with future tests, maintenance issues and reduced page performance.

Beyond analyzing your results and getting your winner into production, there’s one final step following the identification of a winning test: capitalize on the win within your organization.

Communicating big wins for the business and customer insights drive momentum and support for experimentation within your company. Create powerful case studies; hone your storytelling technique to ensure you leave a memorable impression. Share your successes on Slack, by email, at town halls, or host a webinar…the opportunities are endless. Find the communication channel that catches the most attention in your organization, and run with it!

In our experience, cross-functional alignment is the biggest barrier and the largest contributor to the success of an optimization program. Have any additional ideas or examples of ways to create alignment around testing between engineering, product and optimization teams? Let us know in the comments!

Does your optimization process feel like less like a process and more like organized chaos? We’d love to help. Learn more about our services or contact us today.

The post You’ve got a winner – now what? appeared first on Brooks Bell.

Who’s Hiring in August?

Here are our picks: Director, Digital Marketing, Estee Lauder – North America – “Develop, execute and manage best in class national and coop digital marketing programs to promote brand awareness and drive retail sales” in New York.  “Manage digital budget, timelines and drive creative production for asset development.” VP, Digital Technology – Total Wine & More […]

The post Who’s Hiring in August? appeared first on Brooks Bell.

Here are our picks:

Director, Digital Marketing, Estee Lauder – North America – “Develop, execute and manage best in class national and coop digital marketing programs to promote brand awareness and drive retail sales” in New York.  “Manage digital budget, timelines and drive creative production for asset development.”

VP, Digital Technology – Total Wine & More is looking for a candidate in Raleigh, North Carolina to “build the right technology strategy to support the Omni-channel efforts across the company.  This individual should have deep experience with high-traffic, content-rich retail websites sites and applications focused on bringing digital solutions to all store and customer interactions.”

Associate Director, Testing (Marketing Analytics) – “Help the World’s largest omnichannel retailer, Walmart eCommerce, drive optimal efficiency and effectiveness of our Marketing investment through continuous testing of optimization scenarios” in San Bruno.

Personalization & Site Testing Analyst – Levi Strauss & Company is looking for a candidate in San Francisco to “play a key role in accelerating the growth of Levi’s and Dockers eCommerce businesses via site testing & personalization.”

UX/UI Developer & Designer – Direct energy is looking for a “passionate and dynamic Mobile UX/UI Design & Develop professional who understands the intricacies of cross-browser development and knows how to build simple interfaces by writing maintainable CSS.”

Digital Marketing Manager, Personalization – Charlotte Russe is going through an “exciting digital transformation to become a best-in-class fast-fashion retailer.” They are looking for an “ambitious learner to lead a test and learn strategy through experimentation for their digital marketing channels” in San Francisco.

Senior Digital Optimization & Testing Analyst – Dignity Health is searching for a candidate in San Francisco to “improve the user experience through Personalization, A/B & Multivariate testing and inform experimentation strategy across the organization.”

Testing & Personalization Specialist, Analytics & Data – In New York, IBM is seeking a “champion of user-centric thinking to help shape marketing through A/B testing and personalization.”

Manager, Site Analytics and Optimization – “Manage the optimization of Eddie Bauer’s digital properties and support a testing and analysis-based culture to drive customer experience” and business goals in Bellevue, Washington.

Vice President, Finance – Brooks Bell is looking for someone to join the leadership team in Raleigh, North Carolina.  They will be responsible for financial strategy, accounting and contract negotiations with clients.

Help us, help you! Trying to fill a position in testing and optimization? Send us your posting and we’ll include it on our next post!

The post Who’s Hiring in August? appeared first on Brooks Bell.

Adobe is Killing Ad Hoc Analysis & Everything is Going To Be Fine

Where were you when you heard the news? I was checking my analytics team’s Slack channel at work when my teammate shared this screenshot: At first, my thoughts went to Java. Then, it really hit me. Ad Hoc?! No. That’s not possible. But instead of letting this news ruin my week, I thought to channel […]

The post Adobe is Killing Ad Hoc Analysis & Everything is Going To Be Fine appeared first on Brooks Bell.

Where were you when you heard the news? I was checking my analytics team’s Slack channel at work when my teammate shared this screenshot:

At first, my thoughts went to Java. Then, it really hit me.

Ad Hoc?!

No.

That’s not possible.

But instead of letting this news ruin my week, I thought to channel my mom’s advice from my junior high school days: “Don’t get upset about things that are outside of your control.”

If I could influence Adobe, I would definitely try.  Actually, is anyone at Adobe reading this? Is there any chance that you could reverse the decision? No? Ok, that’s fine, too.

So, deep breaths. I’m going to jot down why this gave me feelings and try to determine whether my perceived issues are real problems at all. And, I thought, why not bring you all along for this personal therapy session of mine?

 

First, let’s unpack why this is such a big deal.

If you’re unfamiliar, Ad Hoc Analysis is a tool within Adobe Analytics. It’s used by analysts, like myself, to analyze and report on website performance— engagement, conversions, eCommerce, etc.

Adobe has another reporting and analysis tool within Analytics, Analysis Workspace. It operates in a similar fashion to Ad Hoc, but in a more visual way. The company is already encouraging Ad Hoc users to make the switch over to Workspace. 

Ad Hoc Analysis

Analysis Workspace

Rather than using the reporting dashboards available within various testing platforms, most analysts connect their test results to analytics tools like Adobe Analytics or Google Analytics.

Using these more sophisticated tools enables us to view A/B test results in conjunction with any one of the dimensions, segments, time periods, or metrics that exist within the analytics tools. 

So if Workspace exists, you’re probably wondering why we’re still stuck on Ad Hoc over here. 

First, Ad Hoc is flexible.  Once you’ve learned the capabilities, and assuming your site has the proper tagging in place, Ad Hoc enables you to answer any business question; business questions cause our little analyst gears to turn and assembling a report, custom segments, or calculated metrics transforms into a neat little puzzle to solve.

Also, analysts are creatures of habit.  When it comes to doing our jobs, analysts like to stick with what we know. Solving problems or answering questions is enough of a challenge, and we don’t want to spend extra time thinking about where to find segments or buttons in a tool. Those of us who live in Ad Hoc on a regular basis will need a little time to adjust. Bear with us.

The impact of this on my own day-to-day wasn’t lost on me. So, even as I mourned the loss of Ad Hoc, I also began to consider the challenges ahead.

Here were my concerns about switching from Ad Hoc to Workspace

Can I create all of my complex segments and calculated metrics in Workspace? Even though the two products look different, the functionality seems to be all there.  In general, Workspace is a prettier product; Ad Hoc just feels more real. And let’s face it: when things look a little too pretty, analysts become skeptical.

Doesn’t Workspace have limits on the number of rows you can export? Yes. Today, Workspace only enables you to view and export up to 400 rows at a time (though the default view is 50). So, while this isn’t something that we can work with today, Adobe does have plans to increase downloads for up to 50,000 rows from a freeform table. Cue: huge sigh of relief.


Should I use Data Warehouse instead? 
Adobe Analytics’ Data Warehouse tool is better for setting up a large and/or scheduled data-pulls. It’s not a good option for an exploratory tool.

Isn’t Workspace buggy and slow? When I asked my colleagues what they thought about Workspace, many of them used the word “clunky.” This impression exists because Workspace is a browser-based tool. It also automatically reloads your report every. single. time. you make a change. Compare this to AdHoc, where you can change as many elements as you want, but the report will only refresh when you hit that magical little “Replace Table” button.

Maybe this is in the list of upcoming upgrades, but I haven’t come across any mention of it yet.

How will I explore my data?
Short answer: Differently.

Long answer: Neither Workspace nor Data Warehouse are ideal for exploring new datasets. If you’re already completely up-to-speed with your dataset’s tagging, metrics props and eVars, you’re fine. However, when you get into new datasets, data exploration is critical to ensure that you are getting the most out of your data and analysis. This will be a bigger challenge for agencies and consultancies (like Brooks Bell), as data exploration is key to kicking off our work with new clients.

Workspace isn’t a bad option. It’s just different.

While there are definitely redundancies between Workspace and Ad Hoc, there are actually quite a few benefits to switching to Workspace.

First, Workspace is good for on-going test reporting.  Here at Brooks Bell, we can set up and share dashboards with both our colleagues and our clients, enabling everyone to actively monitor test results. This is particularly nice at the beginning of a test’s lifecycle and allows for transparency throughout the entire process.

It also has an undo option.  Many Ad Hoc power users can relate to the combination of defeat and hope they feel after accidentally closing the wrong tab, attempting that “close without saving” trick, while praying they didn’t change too much since last save.

Finally, any changes you make to segments will automatically update in Workspace. Meanwhile, in Ad Hoc, you have to remove your segment from the work area, and then add it back in from the full list of segments. Fewer steps = less time.

Finally, how to prepare for a world without Ad Hoc

1. Start your transition today.  I opened up Workspace for the first time in a while just last week.  I’m now using it to do most of what I would normally do in Ad Hoc. So long as I don’t update Java, I know I can always fall back on Ad Hoc for the large data-pulls and data exploration until those features are in place in Workspace.  For now, though, it’s all about building new “muscle memory” as I incorporate Workspace in my workflow.

2. Check out this process documentation for making the transition. I read through this a few days after hearing the news and wish I had read it sooner.

3. If you plan to continue to use Ad Hoc for the time being, don’t update Java. Ad Hoc will no longer work with future Java updates.

4. Give feedback!  Adobe is soliciting feedback all over the place right now! This shows that they care about their users and want Workspace to be a useful tool. Don’t hold back on feature requests—it never hurts to ask.

Ultimately, we all had a sense this day would come, especially as data and analytics technologies continue to develop. I feel much better now that I’ve dove into this change headfirst—I hope you do too!

The post Adobe is Killing Ad Hoc Analysis & Everything is Going To Be Fine appeared first on Brooks Bell.

Unlocking the True Power of Testing & Other Takeaways from Brooks Bell’s Interview With Ambition Data

Recently, our Founder and CEO, Brooks Bell, sat down with Allison Hartsoe, host of the Customer Equity Accelerator—a podcast produced by Ambition Data. Listen to the full podcast or read on for a few highlights from their conversation:  On what inspired her to build an experimentation consultancy… Originally, Brooks founded Brooks Bell Inc. in 2003 as […]

The post Unlocking the True Power of Testing & Other Takeaways from Brooks Bell’s Interview With Ambition Data appeared first on Brooks Bell.

Recently, our Founder and CEO, Brooks Bell, sat down with Allison Hartsoe, host of the Customer Equity Accelerator—a podcast produced by Ambition Data. Listen to the full podcast or read on for a few highlights from their conversation:

On what inspired her to build an experimentation consultancy…

Originally, Brooks founded Brooks Bell Inc. in 2003 as a website development agency. After working with a few local clients, a chance introduction led to her first major experimentation client, AOL.

Today, you might think of AOL as one of the [now-extinct] internet dinosaurs, but even back in the early 2000s, the media giant was facing its fair share of challenges. According to one story by Time Magazine, despite having 34 million members in 2002, AOL was battling slowing subscriber growth, falling ad revenue and exorbitant operational costs. 

So, the company turned to experimentation. “AOL had the right environment to build a testing culture,” said Brooks. “They had a closed technology environment, their own analytics platform, and their data was clean and connected.”

Back then, AOL relied on pop-ups to drive new subscriptions. Working with Brooks, the company issued a challenge: design a new subscription pop-up that would beat the control experience. And so, drawing from her background in design and psychology, she did—and then she did it again, and again, and again.

But that was just the start. As other large companies began to rely more on the digital space to drive their business, Brooks saw an opportunity to help them tap into the power of experimentation.

“We realized that no one was testing!” said Brooks. “No other large companies had the data, culture and processes in place to test. So we set out to help them build the data fidelity and really recreate what we saw at AOL in those early years.”

On the difference between optimization and experimentation…

It’s one of the more common questions we get: “Brooks Bell is an experimentation consultancy. What’s that? What’s the difference between experimentation and optimization?” As Brooks explains it, it all comes down to science.

By definition, experimentation is the application of the scientific method to determine something. And while optimization is one potential outcome of an experiment, true experimentation requires running tests without a prescriptive outcome or application.

To put it simply – you’re testing to learn. And as long as your results are statistically significant, there is always something to be learned from experiments—even those with flat or negative results.

On how to unlock the real power of experimentation…

Today, in the age of Amazon, a customer-centric experience is critical. But for some established companies, this requires a bigger paradigm shift in culture and processes.  

“Customer-centricity requires rethinking metrics, the type of data you collect, how teams are organized, how teams are incentivized, how you communicate and also your core values,” said Brooks.

The true power of experimentation lies in its ability to align your customer needs with your company’s strategic goals and your program’s agenda. Furthermore, you can use experimentation to learn new things about your customers in a scientific way.

“Having statistically-sound customer insights can totally change how you organize your store, how you train your team, and how you structure your website,” said Brooks. “This is where testing programs can really drive change.”

To that end, we recently celebrated the launch of Illuminate, our customer insights software for testing teams and executives. Illuminate not only provides a place to store, share and learn from your experiments, but also a means to develop impactful customer insights.

“We launched Illuminate to provide a repository of great test examples, to learn from each other, and to build a library of great test case studies,” said Brooks.  This is because outside of the testing program, any key learnings from an experiment can get lost within the data. Illuminate solves this by encouraging deeper thinking about customers, their needs, preferences, and behaviors. 

Learn more about Brooks Bell’s experimentation consulting services. 

The post Unlocking the True Power of Testing & Other Takeaways from Brooks Bell’s Interview With Ambition Data appeared first on Brooks Bell.