Implementing A/B Testing

Before you can start doing A/B tests you need a system that can support them. That means either you find one off the shelf or you build it yourself.

Off-the-shelf A/B Testing

Most off-the-shelf A/B testing software is geared towards marketers as they were the first group online to adopt the technique en masse. The two pieces of software I'm most familiar with are Omniture Test & Target, which used to be part of Offermatica before Omnitured acquired them, and Google Website Optimizer.

Omniture Test & Target costs money and is designed for big corporate clients with equally big wallets. It's very nice software, but probably not what you're looking for if you're just getting started out.

Google Website Optimizer, however, is free and much more simple. It lets you do both A/B testing and multivariate testing, but is limited in that it only has a notion of "conversions."

You place a bit of code on every page that is part of an experiment and another bit of code on the page that counts as a "conversion." You can then track conversion rates across your treatments (or "variations" as GWO calls them).

Conversion Rate Experts has a google introductory article on Google Website Optimizer if you're interested.

Rolling Your Own

Rolling your own A/B testing system isn't that hard. Let's say we're running an email campaign called buy_our_book where we're trying to advertise our new book. Here's how the code might look (in PHP):

function send_mail($recipient, $campaign) {
	$subject = get_mail_subject($recipient, $campaign);
	$copy = get_mail_copy($recipient, $campaign);
	mail($recipient, $subject, $copy);
}

function get_mail_subject($recipient, $campaign) {
	if ($campaign == 'buy_our_book') {
		// Get treatment if it exists, else get random treatment
		$treatment = get_treatment_for('book_subject', $recipient);
		switch($treatment) {
			case 'control':
				return get_default_subject($recipient, $campaign);
			case 'discount_price':
				return "Get 50% off our latest book!";
			case 'direct_appeal'
				return "Buy our book, we beg you!"	
		}
	} else {
		get_default_subject($recipient, $campaign);
	}
}

get_treatment_for does the meat of the work. It should do the following:

1. Use an existing treatment if the recipient already has one.
2. Otherwise, assign that user a random treatment

Here is an example MySQL schema for A/B testing:

CREATE TABLE treatments (
	id INT UNSIGNED NOT NULL auto_increment,
	name VARCHAR(255),
	experiment_id INT UNSIGNED NOT NULL,
	PRIMARY KEY(id)
);

CREATE TABLE experiments (
	id INT UNSIGNED NOT NULL auto_increment,
	name VARCHAR(255),
	PRIMARY KEY(id)
);

CREATE TABLE users_treatments (
	user_id INT UNSIGNED NOT NULL,
	experiment_id INT UNSIGNED NOT NULL,
	treatment_id INT UNSIGNED NOT NULL,
	PRIMARY KEY(user_id, experiment_id)
);

This schema assumes each user can be uniquely identified by an integer, but if the requirements of your website are different you can change it. For example, if users aren't required to sign up and you track their activities using cookies you'd store the cookie ID in the database.

Use Weights

It's also advisable that you add weights to your treatments so that, e.g., you can select one treatment 90% of the time and another 10% of the time. All this logic can be encapsulated in the get_treatment_for function. I even wrote an article about how to get weighted random elements in PHP.

Why weights? If you're dealing with revenue running a 50/50 split between the control and a total unknown puts your bottom line at risk.

Even if you're not monetizing your traffic you probably don't want to put your traffic itself at risk. Weighting your treatments takes care of this.

What's Next

Hopefully you understand enough to go out and implement a basic A/B testing system yourself. In my next two articles I'm going to cover instrumentation and analysis. That is, what should you be measuring (and how) and how do you know which treatment was successful?