A/B Testing, Design and Analysis

95-819

Units: 6

Description

When is making a change to a webpage, an algorithm, or products worthwhile? Should I increase or decrease prices? Is my advertising cost-effective? Rather than simply relying on the intuition of managers, businesses increasingly strive to accurately identify the impact of their actions. This is particularly the case in the digital economy where firms have access too large amounts of data.

However, properly identifying the causal effect of a business change requires knowledge of the appropriate methods. One such methods is to conduct field experiments, also referred to as A/B tests. Yet, implementing experiments is not always straightforward but instead requires a thorough understanding of both the underlying business problem as well as the challenges associated with the identification of causal effects.

This class teaches methods for measuring the true impact of business changes in the digital economy. Because of the prevalence of A/B testing in practices, we especially emphasize how A/B tests work and how to apply them. We will have hands-on discussions of how managers can measure impact in business situations and how to evaluate claims of impact made by others. While the focus will lie on A/B testing, we will also discuss other methods that can be used when experiments are not feasible. Methods are illustrated with examples from a variety of digital businesses including Microsoft, Amazon, eBay and Uber. We will cover examples in a variety of topical areas including advertising, pricing, product design and distribution channel decisions.

Guest speakers will discuss the practical importance and challenges of implementing A/B as well as what firms can do when implementing straight-forward A/B tests is not practical.

Learning Outcomes

At the end of the course, students should be able to:

Explain why measuring impact by using the correct causal methods is important

Recognize whether claims are causal when they encounter them in the real world and to be appropriately sceptical of claims
Identify the conditions that allow one to learn a causal effect from data

Describe why experimentation solves the causal inference problem and identify when natural experiments can approximate actual experiments
Implement field experiments in a variety of empirical contexts such as advertising or pricing
Understand the challenges of why firms do not experiment more often and how to potentially overcome those

Prerequisites Description

N/A