If you’re just starting a business, or beginning to dive into the world of optimization and marketing, you probably have heard or seen the term A/B testing being thrown around. Hold on to this term and never forget it! Let me start off by saying A/B testing is vital in keeping your brand alive, breathing, and thriving. It is the sun and water to that seed you planted and as you begin to sprout it will help you grow tall and strong and attract all the bees, and the bees will continue to pollinate and keep you going. Ok, I digress and I apologize, this summertime quarantine might be having an effect on my words and I also currently have a bees nest in my house.
This will be the 3rd post in a series on Growth Marketing where I’m sharing with you what I’m learning from the CXL Institute minidegree in Growth Marketing. Today is the day where we will jump in the water together (cue summer references), and tread the surface on A/B testing. In today’s post, we will dive into the history, some pitfalls to avoid, when to use it A/B testing, and ultimately why you should be using A/B testing everyday.
So, what is A/B testing?
Put simply, A/B testing is comparing 2 versions of something to see which performs better. Now-a-days this is typically correlated to your website or app, but this technique, running randomized controlled experiments, actually dates back hundreds of years. It is truly a scientific process. There are hypotheses involved, mathematics, probability, statistics - the whole shabang. I will spare you the in depth science and math lessons and instead give you information to be aware of.
A little history please…
The World Wide Web was created in 1989 - the great year I was born. I remember dial up so clearly. When you actually had to be patient and wait while you listened to that beautiful, ear wrenching sound to get connected. Those were the days...
From 1995 - 2000, there wasn’t really A/B testing and cookies didn’t exist. There were a few, highly specialized people digging through files on the website to see behavioral changes, and using java script for redirecting. I can only imagine how lengthy of a process that must have been. Flash forward to 2006, Google Optimizer launched and changed the game! That’s when optimization slowly started to be injected into the world of marketing leaving traditional marketing and brand marketing behind. But 2010 was the most important year. VWO and Optimizely came to the play (web optimization and experimentation tools) with some errors included, but finally these tools made it accessible for anyone who wanted to learn the tool to use it. In 2016 the quality of the tools became way better, most issues were fixed, personalization came to the marketplace and this is the year that A/B testing went through puberty and fully developed. Ever since, A/B testing continues to rapidly evolve, and is now the essence of successful companies.
Today data is everything.
Data is your best friend, and if you don’t feel comfortable running experiments or analyzing the results, hire someone who does. In this day and age, data should be guiding you forward, not solely an idea you think would work. Projects and products have failed and will fail without proper data to give you that deeper insight that you need, and maybe you didn’t realize you even needed.
Expecting something will be successful without research or testing is like going scuba diving without being certified or knowing how to swim (again water reference because it’s summer). You know nothing about the terrain, expect to have an amazing experience, you don’t know how far down you can go, how your gear works, the effect it has, what the outcome will be...etc. You are blindly floating in the dark depths and I'm sorry to say, it is most likely not going to end well. By applying A/B testing, it helps you navigate through the unknown waters with ease, and assists you in making better trustworthy decisions, ultimately giving you bigger wins.
When to use A/B testing?
If you are a small company and just starting out, you should only use A/B testing once you have 1,000 or more conversions a day. A conversion does not need to be revenue based. It can be interactions with your site, or any KPI or goal you set. But regardless, you need enough data to make wise, trustworthy decisions. Once you have enough traffic to your site, you should begin implementing A/B testing ASAP for the following:
Deployments: When you deploy something new on your website (i.e. a new feature, product, anything new). You want to learn if your new deployment will have a negative impact on your KPI. If it has no negative impact, it’s a win! But if it goes down, do not unleash it to your site!
Research: You must always do your research before you optimize. By using a Conversion Signal Map you can see which elements are making an impact on your site. You can also use things like Fly-Ins (yes that box that flies in and annoys us all) to see if the motivation is specifically working for a specific group of people visiting your site. And believe it or not, these actually can work and do not annoy everyone!
Optimization: When you apply a change to the website on the client side. If it begins to work on a certain group, it can then become a lean, real deployment.
Common pitfalls - Don’t do it!
Collecting the right data is important and there are many false positives and false negatives that can occur throughout your experiment. Here’s some common pitfalls to be aware of:
Not having large enough data
Not using proper statistics
Not knowing when an experiment has failed or not
Not knowing what to call a winner
Not understanding what to experiment on
This is where having a team of people working on experiments and digging through the statistics is needed for you to optimize your website, your company, and your profits. There are models, calculators, and tools out there to help determine how many experiments to run, for how long, and how many interactions you need for each experiment to be successful. However, at the end of the day, it’s up to you (or your team) to analyze, and make sure the data you are getting is accurate.
Research must come first.
Research is SO important and you should not just run an experiment to run it. You always need to start with an analyzed reason. The 6V Conversion Canvas (discussed below) is a clear framework that will help you efficiently and effectively optimize. By using this model, you will gather clear insights on your customer's journey, understand the basic user behavior, and provide you input for setting a hypothesis:
Values: What company values are important and relevant? What delivers the most impact? What are your company’s short and long term goals? What’s valuable for your company?
Versus: Who are your competitors and what are they doing? What’s their full customer journey? Analyze everything - track changes to their site, any new offer, any change.
View: Know the view of your customer. Where do they start on your site? Where do they come from? Are there notable differences between segments or products? Look at traffic sources from your landing pages, customer journeys across key segments, products, time on a page, scroll depth, and heat maps.
Voice: Know the voice of your customer. Analytics tells you behavior but hearing your customer’s feedback gives you the WHY. Talk to your customer service, sales teams, interview customers, and pre-test your assumptions.
Verified: Are there already scientific research and insights available? Many studies have been done and are published - use them! Google Scholar is a great tool to find published scientific papers. Strong hypothesis are based on science, not solely internal data.
Validated: What insights are validated in experiments and your analysis? You can play with taking out content and seeing if the conversion rate stays the same, or do some elements have a negative impact? You can even look through your experiment tracker because yes, you should track every A/B test you do, and see if there is any insights you can pull from them.
In conclusion…
A/B testing is all about optimization efficiency and effectively. If you have over 1,000 conversions a day, start testing. There is always something to improve, nothing is perfect, and optimization is about constantly striving and adapting to bring in more revenue and success. The more you test, the more you know. The more you continue to grow and evolve with your customer, the stronger that relationship will be. There is always something to test and the smallest change, can bring you the biggest wins.
Let’s end this with a quote by Jeff Bezos to inspire you and let this all sink in:
