Scientific Product Development

Growing up every kid learned about the scientific method, about hypotheses, testing, measurement, and analysis. Data-driven development is about taking these scientific principles and applying them (at least in part) to all aspects of a business — especially product development.

It's about subjecting your decisions to empirical reality and letting the data guide your intuition. Mike Speiser wrote about this last September and it's a good phrase, so I'm going to steal it.

Let's take a look!

Scientific Product Development

In that spirit, let's break down the process of scientific product development. A picture is worth a thousand words, so here goes:

Let's break it down.

Hypothesize

The first step in scientific product development is to come up with a testable hypothesis. This can be a hypothesis about anything: your userbase, your market, whatever.

Measure

Many scientific breakthroughs happen because of improvements in instrumentation. Think microscope, telescope, x-ray crystallography, etc. The nuts and bolts of instrumentation, data collection, and measurement are just as important in building a data-driven business as they are in science.

What data do you need to collect? How do you collect it? How do you store it? And finally, how do you extract it in a way your analysts can make sense of it?

This is also the step for the metrics-obsessed.

For most startups measurement and instrumentation involves little more than SQL and your database of choice, but the big boys use technologies like Hadoop, Cassandra, and BigTable to solve various problems in this domain. Google Analytics fits here, too.

Test

I've been talking about this step the most. Once you are recording data and have the ability to extract intelligence from it you can subject your products to A/B tests, multivariate tests, and other experimental techniques.

You can either build your own testing infrastructure or use off-the-shelf stuff like Google Website Optimizer.

Analyze

The final step in the process is analysis. This means taking the data you've collected and generating insight.

Analysis involves looking at all the interlocking variables, isolating the ones that matter most, and presenting them in a way that is easily understood. Statistics is helpful here, as is a background in Information Design.

Your analysis is what gives you the data to contradict or support your hypothesis. It also tells you what you're missing as you iterate your data-driven process. Are you not collecting data you need? Are there additional hypotheses that you need to test? Good analysis yields a few good answers and several more questions.

Directed versus Undirected Analysis

When I was learning the scientific method as a kid it was always presented as a rigid process. You form a hypothesis then you go about methodically testing that hypothesis, refining it until it's a "theory."

This model is too rigid. Flashes of insight often come when you're freely exploring the data. There isn't always a statement to test — sometimes you need to look at the data to figure out what's worth testing.

I'm Not Alone

Other people, like Mike Speiser and Eric Ries, are talking explicitly about building products in this way.

On the strategic and measurement side there are people like my friend Andrew Chen and Dave McClure.

If you have any recommendations of other scientific-minded bloggers I'm all ears!