A/B Testing, Design and Analysis
When is making a change to a webpage, an algorithm, or products worthwhile? Should I increase or decrease prices? Is my advertising cost-effective? Rather than simply relying on the intuition of managers, businesses increasingly strive to accurately identify the impact of their actions. This is particularly the case in the digital economy where firms have access too large amounts of data.
However, properly identifying the causal effect of a business change requires knowledge of the appropriate methods. One such methods is to conduct field experiments, also referred to as A/B tests. Yet, implementing experiments is not always straightforward but instead requires a thorough understanding of both the underlying business problem as well as the challenges associated with the identification of causal effects.
This class teaches methods for measuring the true impact of business changes in the digital economy. Because of the prevalence of A/B testing in practices, we especially emphasize how A/B tests work and how to apply them. We will have hands-on discussions of how managers can measure impact in business situations and how to evaluate claims of impact made by others. While the focus will lie on A/B testing, we will also discuss other methods that can be used when experiments are not feasible. Methods are illustrated with examples from a variety of digital businesses including Microsoft, Amazon, eBay and Uber. We will cover examples in a variety of topical areas including advertising, pricing, product design and distribution channel decisions.
Guest speakers will discuss the practical importance and challenges of implementing A/B as well as what firms can do when implementing straight-forward A/B tests is not practical.
At the end of the course, students should be able to: