Course Description
In this tutorial, we will go over what you need for steps in online experimentation. When running a business, there are many different moving parts happening all at once. This is why it is helpful to carefully draw out your steps in experimentation.
What You'll Learn
- The overall experimentation process
- A/A Testing, A/B Testing, and Multivariate Testing
- How and when to use multiple testing methods
Hi, and welcome to a quick introduction to Steps in Experimentation.
When running a business, there are many different moving parts happening all at once. You might redesign your website, or add a new chatbot, or experiment with your recommendations you show customers. But before you jump ahead and launch new products or elements, you’re going to want to test to make sure that the experience is useful to users and that you are achieving the metrics you are hoping for with the new changes. This is why it is helpful to carefully draw out your steps in experimentation.
The first step in experimentation is planning. This is where you ask yourself your pressing business questions, such as: What is the element you are going to change and what is that change going to look like? How large is your sample size and how long will you run your experiment? In the planning phase, you will want to know what business problem you are trying to solve and what are the metrics and expected outcome? As mentioned previously, you are going to have to establish your criteria for success prior to running your test, and having a hypothesis helps to understand what you are hoping to learn from the experiment.
The second step in experimentation is coding and logging. This is where you set up the test and experiment that you want to run. For example, you change a call to action button from green to red. Or updating your algorithm to show your user different recommendations.
The third step in experimentation is performing an A/A test. As mentioned previously, you would want to run an A/A test to validate the setup of your experiment by comparing the identical experience on a different random set of users. For example, your original version, Version A or “control”, is a green “Buy Now” button and your alternate version, your treatment, is also a green “Buy Now” button.
It is good to set up an A/A test to show a random subset of users the same experiment and another random group of users the same experiment. You should have different users and different data, and if it is the same experience, but the metric is different, something is most likely wrong with the setup.
After running your A/A test and you find that the results of the experiment were not statistically different from one version to the other, you can then run your A/B or multivariate test. So once you’ve run your A/B or multivariate test, you can then look at the analytics and performance of the changes you’ve made. What are your metrics showing you and were they as you suspected when you established your hypothesis?
It’s always good to consider various factors that might have affected the performance of your tests, such as seasonality, the segments you ran your experiment on, or a newness effect. Did just seeing something new trigger excitement in your customers? And lastly, the final step in experimentation is to make a decision. Do you ship it or not? And this might require further discussion among colleagues or stakeholders. It might also lead to running a new experiment, where you will again begin the process of running through steps 1-6.
Thanks for watching. Give us a like if you found this useful, or you can check out our other videos at online.datasciencedojo.com.
Blair Heckel - Blair holds a Bachelors degree in Marketing from Washington State University and has a background of leading data-driven marketing campaigns.
