Years ago, I landed an interview for a sales position at a company. When the company’s CEO wanted to know what I’d like to do in the future I answered avidly: ”Sales and Marketing mathematics”. The CEO was, to say the least, overwhelmed. I thought I’d never get the job, but I did. And now, fifteen years later, I still love both sales and marketing mathematics.
Since I got that job, this whole thing about shopping on the Internet happened. And it turns out that finding and retaining customers in this environment has everything to do with combining sales, marketing and mathematics with user experience design.
One of the ways to do this is through something called A/B testing. As people in the know would love to tell you, A/B testing has to do with how Internet companies evolve their products. By giving billions of users different buttons to click. Or something.
But you’re not Facebook or Google, so why should you care?
Well, if you sell stuff, the answer involves more money, for you. So please allow me to enlighten you. I love sales, marketing and math, after all.
People exchanging money for cool stuff. Just like on the Internet. Photo by Blt Boy.
What is A/B testing and why does it matter?
As hinted at above, A/B testing is an approach where you test two versions of something against each other. In the context of the web, e-commerce and mobile apps, the purpose is to identify the better, winning variants of headline copy or perhaps the placement of a button. And since the internet is massive and full of everyone, you can do relatively easy experiments on an unwitting, real audience. Like a mad scientist!
Concisely put: simply expose A and B versions to thousands of people at random to get an understanding of what they prefer as the best option. And as you may have guessed, the terminology comes from the two options, A and B. The original version is referred to as A and B is the challenger.
With the right kinds of tools or plugins for your website, e-commerce system or app, you simply roll out the test, sit back, and wait for a winner to emerge. Your visitors will act as guinea pigs, and unlike test subjects in many other fields, we don’t have to think about unpleasant ethical questions. It’s a good time to be a mad scientist!
The goal of your A/B test might be to maximize user actions on your system, like time spent reading an article, money made with incremental sales or something else. The one who wins with a confidence interval, i.e. confidence level (preferably 95% or 99%), is the best way to go. But before you start, please allow yourself the time to set up a good test. Good testing does involve work. Continue reading, and I’ll do my best to help you avoid the most common pitfalls!
What is the formula for successful A/B testing?
There’s no way around it: First you need to understand what your business is and how your marketing supports it. Why do people buy your products or services? What is the value added? What is it you should be measuring? What is your rate of conversion? As in, how many of your visitors turn into paying customers?
You can ask the right questions about how to improve your business only by really understanding the current state of your your business and how it operates.
Getting there means questioning everything, within reason. Everything can be done better, but as stated by the Pareto principle, it’s common that 20% of your actions bring in 80% of the results. So make it a rule of thumb only to question things that really make an impact.
Once you have a preliminary question, you need to collect further evidence to support your hypothesis. Tools for understanding a website include analytics, preferably with heat maps, questionnaires, funnels and even live customer support chat. Find the underlying bottlenecks in the user experience that need to be crafted towards low friction with skill and precision.
Simply put: the minimum viable effort here is really to have Google Analytics installed on your website and letting it collect data for at least 3 or 4 weeks.
Once you have enough information, you can create a hypothesis on what might work better. This stage is important. Think of it as striving for an ultimate question you want answered to shed light on something specific. So make sure you base your question on information rather than guesswork. Or else…
Well, or else, you’ll find yourself stumbling towards endless tiresome references to the Hitchhiker’s Guide to the Galaxy novels. In which, of course, the ultimate Question and Answer can’t be known in the same universe. Tee hee, right?
Asking intelligent questions
Numbers can be scary.
If you feel this way, feel free to take a moment to blame your teachers. It’s probably fair game. But I promise you that there’s fame and glory ahead if we decide not to just stare in panic at the numbers you get from your site. Find what you’re looking for by drawing simple notes and sketches on printouts if remembering how to navigate Google Analytics or making good formulas in Excel feels like enemy territory.
When you have enough information from your raw data and your notes on it, try to define a test and a desired outcome. You need a clear theory on why the B sample in your shiny new A/B test should perform better. Just mucking around with something like changing colors on a web page is unlikely to increase conversions, unless of course something’s horribly wrong with your current state of affairs.
But if you’re out to test a call to action button that’s significantly different, conversions will probably change. Again, just make sure you understand what you’re after and accept that it’s ok to make faulty guesstimates. When making tests, please be subtle and refrain from making too many changes at once. If you make things too complex, you will sooner or later stumble down a rabbit hole where you’re unable to pinpoint the change that made all the difference.
As any mad scientist would, stick with running your experiment until the statistical significance is high enough. Any significance below 95% will eventually lead you towards guesswork and poor decision making.
Good A/B testing software for the web like Nelio or Optimizely will help you calculate all this. With good tools in place, you can focus on thoroughly analyzing the outcome and understanding why you got your results. What went wrong, what went right? What can you do better? What should be left out going forward? Use your analytics to get a broader understanding on why the results occurred and if you’re stuck, talk to someone.
People who like to think about numbers don’t necessarily have beards. We look like everyone and we’re everywhere, so ask around if you need help! Photo by Brad Montgomery.
Cheat sheet: A/B testing project steps
So far, we’ve introduced the concept of A/B testing and covered a few basic steps:
- Understand your business and goals
- Gather material
- Create hypothesis
- Create the right question to test
In the next part of this blog series, we’ll dive in a bit deeper and talk about the execution of your test:
- Make creatives
- Test and analyze
- Test more
Tomi Grönfors is an experienced entrepreneur and marketer based in Helsinki, Finland. He is the CEO and co-founder of Sniffie Software, and Managing Director and owner of the Groenfors Method OY. Be sure to check him out on Twitter for more information.
