Analysis strategies in online A/B experiments: Intention-to-treat, per-protocol, and other lessons from clinical trials

Epidemiologists analyse clinical trials to estimate the intention-to-treat and per-protocol effects. This post applies their strategies to online experiments.

January 14, 2022

Many is not enough: Counting simulations to bootstrap the right way

Going deeper into correct testing of different methods for bootstrap estimation of confidence intervals.

August 24, 2020

Bootstrapping the right way?

Video and summary of a talk I gave at YOW! Data on bootstrap estimation of confidence intervals.

October 6, 2019

Hackers beware: Bootstrap sampling may be harmful

Bootstrap sampling has been promoted as an easy way of modelling uncertainty to hackers without much statistical knowledge. But things aren’t that simple.

January 7, 2019

The most practical causal inference book I’ve read (is still a draft)

Causal Inference by Miguel Hernán and Jamie Robins is a must-read for anyone interested in the area.

December 24, 2018

Defining data science in 2018

Updating my definition of data science to match changes in the field. It is now broader than before, but its ultimate goal is still to support decisions.

July 22, 2018

Customer lifetime value and the proliferation of misinformation on the internet

There’s a lot of misleading content on the estimation of customer lifetime value. Here’s what I learned about doing it well.

January 8, 2017

If you don’t pay attention, data can drive you off a cliff

Seven common mistakes to avoid when working with data, such as ignoring uncertainty and confusing observed and unobserved quantities.

August 21, 2016

Is Data Scientist a useless job title?

It seems like anyone who touches data can call themselves a data scientist, which makes the title useless. The work they do can still be useful, though.

August 4, 2016

Making Bayesian A/B testing more accessible

A web tool I built to interpret A/B test results in a Bayesian way, including prior specification, visualisations, and decision rules.

June 19, 2016