Data-Driven Development

by Jesse Farmer on Saturday, November 15, 2008

There are lots of smart people out there talking about metrics and tech startups, but the one thing they all have in common is an empirical mind-set.

Another common thread is that a lot of these practices come from other, older industries that have had time to mature. It's high-time we apply this to the startup world.

What is Data-Driven Development?

Data-Driven Development is centered around the belief that business decisions — whether technical, artistic, or financial — are best made based on what is actually happening rather than your personal model of the world.

Of course, everyone agrees with that. The problem is that everyone always believes their version of the world is the correct one. It's not, at least not all the time. How do you know when you're correct and when you're not?

Applying the principles of Data-Driven Development helps you understand what actually works and what doesn't.

Principles of Data-Driven Development

Three key principles are as follows:

Everyone Is Biased
Decisions should be made through the lens of empiricism rather than the lens of intuition.
Universal Instrumentation
Without visibility you can't tell when you're succeeding — or failing.
No Sacred Cows
The most dangerous beliefs are the ones held universally. Test (and measure) everything.

Why Data-Driven Development?

Business decisions center around risk/reward calculations. Data-Driven Development gives you clearer pictures of both. The best decisions an early-stage startup can make are the high-yield, low-risk ones, i.e., the low-hanging fruit. Without real data you won't know what tree you're picking from, let alone what fruit.

Also, it's worth noting that Data-Driven Development isn't about any single practice, at least not in the same way that Agile is associated with things like XP and Scrum. It is pragmatic and understands its own limitations. Data-Driven Development won't help you know what to value, nor does it aim to make perfect decisions — only calculated ones.

Practical Examples and the Future of 20bits

As you can see the byline of this blog has changed to "Driven by Data." I'm going to refocus this blog to be about Data-Driven Development. These are some topics you can expect to read about:

  • Case studies to illustrate Data-Driven Development
  • Tutorials on probability, statistics, and data analysis
  • A/B and multivariate testing
  • Metrics, their use (and misuse)
  • Operational problems like data warehousing and extraction
  • ...and so much more!™

The rule I've written for myself is that every post has to involve at least one of code, data, or insight. Hopefully this one counts towards the latter. And in the spirit of Data-Driven Development, please drop me a line and tell me directly what you want to hear about.

Cheers, and happy coding!