9 Things You Can Do Now to Ensure Your Experimentation Program Survives a Reorganization

It often begins with rumors and murmurs. Then, perhaps, a shift in executive leadership. At first, changes seem minor or isolated. But eventually, the inevitable becomes reality—a reorg is underway. Because testing programs typically work with and across several divisions in a business and don’t fulfill a traditional business function, they are particularly vulnerable during […]

The post 9 Things You Can Do Now to Ensure Your Experimentation Program Survives a Reorganization appeared first on Brooks Bell.

It often begins with rumors and murmurs. Then, perhaps, a shift in executive leadership. At first, changes seem minor or isolated. But eventually, the inevitable becomes realitya reorg is underway.

Because testing programs typically work with and across several divisions in a business and don’t fulfill a traditional business function, they are particularly vulnerable during the organizational upheaval.

However, to paraphrase Benjamin Franklin, with an ounce of prevention, it’s possible to avoid a pound of problems. Here are nine practices for surviving a reorg that you can start doing nowbefore you ever have to deal with one.

1. Calculate Impact

The first thing any new leadership will want to see is the value the testing program provides to the business. Of course, “value” is a multidimensional concept. It includes the contribution to business goals and priorities, insight into customer preferences and behavior, development of new innovations, and minimizing certain opportunity costs.

While it’s true all of these things will help communicate the importance and value of testing, nothing will be as compelling as a big annualized impact number. This number can be complex and time-consuming to calculate, but even if it isn’t an important metric for your program today, it’s worth having an analyst crunch, record and update just in case.

2. Write an Elevator Pitch

If you had five minutes or less with the CEO of your company, could you clearly communicate the mission, focus, and value of the testing program in a memorable way? If the answer to that question is “no,” or even “maybe” it’s worth spending some time crafting an elevator pitch for your testing program.

Open your pitch with a short anecdote about a problem that was solved with testing. Next, add a sentence or two about the essential mission of the program. Follow this with a sentence about the methods used. Close with a statement about the contribution the testing program has made to the business as a whole.

Once you’ve written this pitch, practice delivering it to your friends, family, team and other stakeholders. After a lot of practice—and incorporating the feedback you will inevitably receive—you’ll be ready to introduce testing to any new leader or team you may encounter.

3. Archive Results

Surviving a reorg isn’t all about making a case for testing. It’s also about maintaining a consistent pipeline after teams have been shuffled around.

The first step to protecting the testing pipeline is to create a complete, detailed, navigable archive of past test results. This is critical for developing new ideas, training new team members, orienting new teams and stakeholders, and simply making the case for what works and what hasn’t.

Recently, we launched Illuminate, our new software for enterprise-level testing teams. Illuminate offers an executive-friendly repository of your tests and any insights you’ve learned about your customers along the way. It’s direct integration with Optimizely, easy-to-use reporting tools, and custom case study generator significantly simplify the task of archiving and reporting your test results.

4. Outline the Process

Testing is complex and when teams get rearranged, processes that once flowed smoothly can become intractably clogged. To prevent this, document the process as it exists and identify areas of parallelization, possible redundancies, problematic bottlenecks and opportunities for redirection.

If you have access to a project manager, ask her to run a few what-if analyses to estimate potential problems if your processes were to be disrupted. Then, work together to develop possible solutions or workarounds to the most likely scenarios.

5. Clearly Define Roles and Responsibilities

Having a fast and nimble all hands on deck approach to managing the testing process is great until a critical person leaves the company or is assigned to a different team.

To avoid this, define the roles and responsibilities of each team member at each stage of the process. Doing this is critical for mapping the resource requirements of the testing process and quickly identifying gaps if the team is restructured.

It’s also important, however, to track ongoing responsibilities and duties in a more specific way. Having a project management platform or system that identifies what stage of the process each test is in, and which team member is responsible for that task is essential for avoiding disruptions.

6. Develop Training Programs

Looking on the bright side, a reorg could mean your testing team is greatly expanded. It might also mean fewer people are doing more, including jobs they have little experience with. In either case, having a developed and ready-to-execute training program is helpful.

Like all of these tips, the best time to develop a training plan is not the first day your new team member walks into the office. Instead, start training and cross-training your existing team right away. This gives you an opportunity to develop extensive content, deliver it, get a sense of what works, and make adjustments before a reorg renders training critical to the continuation—and not just the improvement—of the program.

7. Centralize Documentation

Having lots of documentation is useless if no one can find it. Moreover, it doesn’t help if it isn’t standardized in some way.
The archive of results, process documents, test plans, training materials, and everything else should be stored in a public or shareable archive, in a format that is easily accessible and navigable.

Using filename conventions, consistent directory structures, and standard documentation practices across the team may be mundane, but it’s just as important to the robustness of the testing program as tracking each person’s ongoing responsibilities.

8. Get essential access

One often overlooked consideration is whether the testing team has access to the essential technologies on which it relies.

Even if most development is done by an outside group, it’s important to have access to tools that enable you to upload and modify your code. Additionally, if reports are pulled by a sovereign analytics team, it’s equally important for someone from the testing program to have the access and ability to do so in a pinch.


Some solutions—like tag management systems—address this challenge. Training, cross-training, and collaboration is another helpful way to build the necessary competencies to get access to and make basic use of all your testing tools.

9. Keep it all up to date

Building the previous eight resources can take a lot of time and effort. Many teams will make any one of them a goal for the quarter, work hard to get it done, drop it in an archive, then forget it.

Months, maybe years, pass without giving the resource a second thought. Then, a sweeping reorg happens and the five-year-old process document is unearthed, dusted off, and found to be frustratingly obsolete. That’s why you must take care to not only produce these resources but maintain them as well.

A reorg can be a scary thing for a lot of reasons. However, by following these nine tips today, even the biggest organizational shakeup doesn’t have to disrupt the flow and productivity of the testing program.

The post 9 Things You Can Do Now to Ensure Your Experimentation Program Survives a Reorganization appeared first on Brooks Bell.

How to hire your testing unicorn (without using magic)

When I was running my own testing program, I was in desperate need of an associate to help me manage my small (but mighty!) team. My single associate and I were launching tests left and right and we were unable to do anything other than focus on the day-to-day of the program. A job description […]

The post How to hire your testing unicorn (without using magic) appeared first on Brooks Bell.

When I was running my own testing program, I was in desperate need of an associate to help me manage my small (but mighty!) team. My single associate and I were launching tests left and right and we were unable to do anything other than focus on the day-to-day of the program.

A job description had been posted and the company’s recruiters were doing everything they could to find the right hire.

I remember reaching out to an old friend of mine to see if she knew anyone who might fit the role. I told her that I was looking (simply) for a data-driven individual with stellar communication skills and the ability to manage several complicated web projects at one time.

“Oh,” she said. “So you’re looking for a unicorn.”

“No, Susan… I’m looking for a Testing Specialist.”

Now, I don’t want to be too dramatic here, but this unicorn revelation did rock my world a bit. (It also made me want a bowl of rainbow sorbet with sprinkles… but I digress.)

When I finally overcame this existential testing crisis, I realized that I believed, deep down, that testing unicorns did exist. But I also knew that due to magic (obviously), I might never find one.

There were three main things I was looking for in my unicorn:

  1. Strong analytics skills and the ability to develop advanced data-driven recommendations
  2. Amazing communication skills – for helping stakeholders understand and action off of that data
  3. Organized and efficient project management skills for planning and managing the execution of test strategies

First, I had to assess which skills I already had on my team.

I took a look at my own skills and the skills of the team I had in place. To be honest, I’m much better at talking about analytics than I am at sitting behind a desk and doing a deep dive into the numbers.

My personal strength is in the communication realm of testing and my associate was an awesome project manager. So, it became pretty clear to me that there was a need for a strong analyst on our team.

Then, I had to decide what was teachable.

This is where things get controversial. Because teachable skills can really depend on the skills of the trainee, the trainee’s willingness to learn, and the skills of the trainer.

I did a quick poll here at Brooks Bell to see which skills my colleagues believe is the toughest to teach.

As you can see, many people here believe that good communication skills are hard to coach. And during my search for a Testing Specialist, I felt the same way.

I was pretty confident that I would be able to help my next hire become a better analyst or project manager, but I wasn’t so sure I could teach someone to communicate well in a stakeholder-facing role.

Finally, I had to decide if I could tweak my program structure

Depending on my next hire’s strengths, there were a few scenarios that I had to consider in order to structure my program without a unicorn. Here are a few examples:

If I decided to hire a strong analyst with weak communication skills

In this scenario, I would consider making this Testing Specialist role a non-stakeholder facing role. Because this person would not be project managing or communicating directly with stakeholders, they would be solely dedicated to analytics and free up the rest of the team’s time to focus on project management and stakeholder communication.

If I decided to hire a strong project manager with weak analytics skills

Because I believed that analytics skills were teachable, this associate could focus on project management in the beginning and slowly take on analytics work when they were ready.

If I decided to hire a strong communicator with weak project management skills

In this scenario, I would start by putting this associate in a stakeholder-facing role focused on analytics. After some time, I would begin training him or her on project management.

The magical lesson I learned

When I first approached this seemingly impossible task of hiring my next Testing Specialist, I was discouraged by the reality that I wanted so many specific skills in one individual.

But the truth is, Experimentation and Optimization is still a very niche industry, so finding a single person with so many abilities is going to continue to be tough for a while. That’s why I recommend first looking at the structure of your team, and then deciding which skills you feel comfortable teaching.

And always remember this: Testing unicorns do exist, sometimes we just have to help them find their wings.

Are you a testing unicorn looking for your next big challenge? Check out our monthly “who’s hiring” post for open positions in testing and personalization at top companies.


About the Author:

Sam Baker has eight years of experience running experimentation and digital analytics programs for major e-commerce brands. As a consultant at Brooks Bell, she helps global brands build and grow their testing programs.

In addition to her role at Brooks Bell, Sam is also an accomplished career coach, providing guidance to ambitious women looking to land their dream careers. Originally from Indiana, Sam now lives in Raleigh, North Carolina with her husband and her dog.

The post How to hire your testing unicorn (without using magic) appeared first on Brooks Bell.