Course Description
In this quick tutorial, we go over the basics of A/B testing, as well as answer some in-depth questions such as why should businesses conduct A/B testing? Or how do you perform an A/B test?
What You'll Learn
- Introduction to A/B testing
- Scenarios where A/B testing is needed
Hi, and welcome to this quick introduction to A/B testing.
So, what is A/B testing? At a high level, A/B Testing is a statistical way of comparing two or more versions, such as Version A or Version B, to determine not only which version performs better but also to understand if a difference between the two versions is statistically significant.
Why do businesses conduct A/B tests? This is the way businesses are run these days and they have to take a data-driven approach. A common dilemma that companies face is that they think they understand the customer but, in reality, customers would behave much differently than you would think consciously or subconsciously. Users don’t often even know why they make the choices they make, they just do. But when running an experiment or an A/B test, you might find out otherwise and the results can often be very humbling and customers can behave much differently than you would think so it’s best to conduct tests rather than relying on intuition. Let’s visualize.
For example, in marketing, or a web design, you might be comparing two different landing pages with each other or two different newsletters let’s say you take the layout of the page you move the content body to the right now versus the left or maybe you change the call-to-action from green to blue or your newsletter subject line has the word “promotion” in Version A and the word “free” in Version B in order for A/B testing to work, you must call out your criteria for success before you begin your test.
What is your hypothesis or rather what do you think will happen by changing to Version B. Maybe you’re hoping to increase the conversion rate or newsletter signups or increase opens call out your criteria for success ahead of time. Also, you will want to make sure that you split your traffic into two it doesn’t have to be 50/50 but you will want to figure out what is the minimum number of people I need to run my A/B test on to achieve statistically significant results you can do this with multiple versions such as two buttons that are blue and two that are orange one blue and one orange button say RSVP and another blue and orange button say sign up this would be called a multivariate test or a full factorial test since you are comparing different factors.
So, what are some factors we can test on when conducting an A/B test? Changing the layout of the page and shifting where certain items are such as moving the content body to the right, the navigation to the left, or the call to action near the bottom you can change the call to action such as changing the color or the text or where the call-to-action is located on a landing page or email. You can compare two different images with each other to see if one has a higher conversion rate or a higher click-through rate.
What about on the back-end suppose the UX and the UI are the same but you update your machine learning algorithm to update the recommendations that are shown to people but what happens if something is broken or funky or the data is messy and the quality is off or there’s too much noise maybe there’s a sampling problem and you don’t randomize correctly it could be a one to two percent impact but you should make sure that your A/B test is being conducted properly first by setting up an A/A test.
Thanks for watching. Give us a like if you found this useful, or you can check out our other videos at online.datasciencedojo.com.
Blair Heckel - Blair holds a Bachelors degree in Marketing from Washington State University and has a background of leading data-driven marketing campaigns.
