Course Description
Multivariate testing is a technique for testing a hypothesis in which multiple variables are modified. In this tutorial, we will explain how a multivariate test differs from an A/B Test, how to create and conduct a multivariate test, and what questions you should be asking about your test.
The goal of multivariate testing is to determine which combination of variations performs the best out of all of the possible combinations.
What You'll Learn
- Introduction to multivariate testing
- Advantages and disadvantages to doing A/B vs multivariate testing
Hi, and welcome to this quick introduction to multivariate testing.
So, what is a multivariate test? We saw that with A/B testing we were comparing two distinct versions of an item, or variants. So, a red versus a green call-to-action. Multivariate testing is when you’re trying different possibilities. You have just an image, or you try a sign-up, or you add a video with an image, or you add a video with the sign-up. Or maybe you are testing four buttons, two that are blue and two that are orange. One blue and one orange button say RSVP and another blue and orange button say sign up.
So, what are some common advantages and disadvantages to doing a/b vs. multivariate testing? When running an a/b test it will be simple in design and small sample size may be okay. The limitations to doing an a/b test are that you’re only testing one alternative. When conducting a multivariate test, you can test many combinations at once. However, you will need a much larger sample size to run your experiment on.
For example, you have 200 users and you run an a/b test. So, you test one version on 100 people and the second version on another 100 people. But if you have a multivariate test, where you’re testing out different combinations at once, you will have a much more limited sample size to test each version on. With A/B testing, you are trying one alternative, but with multivariate testing, you can try many combinations at once. And for that, you’ll need a better understanding of interactions. Since there are so many variations the data can get a bit messy.
It can happen with drug testing. One is control, one is a test. One is a placebo and one is the actual drug. But what if the pharmaceutical company is in a rush and they are testing with cholesterol medication, hair loss medication, and blood pressure medication. What if some people only get two medications or some get all of the medications? You will need a better understanding of how these experiences interact with each other.
So, now you know the difference between an a/b test and a multivariate test. Before deciding what kind of experiment to run ask yourself these questions: What are the factors or levels you plan on changing? How big is your sample size? How long will you conduct your experiment? What is the business question you want to answer? What are your metrics and expected outcome? And who is in your experiment? Answering these questions is the first step in experimentation for your a/b or multivariate test.
Thanks for watching. Give us a like if you found this useful, or you can check out our other tutorials at tutorials.datasciencedojo.com.
Blair Heckel - Blair holds a Bachelors degree in Marketing from Washington State University and has a background of leading data-driven marketing campaigns.
