8 Tips for Crafting Metrics That Matter

Metrics are the marketer's microscope. They show him what his customers are actually doing, as opposed to what they say they are doing or intend to do. With proper metrics he can make decisions faster and more accurately.

You can decide to measure anything, but what metrics matter and what ones are just for show? Here are some rules I hope will guide you toward creating meaningful metrics that help, rather than hinder, the decision-making process.

Be Actionable

If I had to give a one-sentence answer to the question "What metrics should I implement for my product?" it would be "Whatever metrics are actionable." This means the line from a question to a metric and the line from a metric to an action should be as short as possible.

Most of the tips below are meant to focus attention on this issue. What can you do to make sure your metrics are actionable?

Be Understandable and Trustworthy

Do you understand what your metric measures? Does everyone in your organization also understand and do they trust it?

Trust is the important part. Everyone has to trust the metric if you're going to use it to make decisions, otherwise you'll be getting constant pushback. This will slow the decision-making process and cause a lot of ill-tempered arguments.

Measure Results

Does your metric measure customer behavior or a correlate of customer behavior? In the past approximations and correlations were necessary because measuring behavior directly was hard, but on the web there's no excuse — you have access to every single thing a person does on your site, down to where their mouse is hovering and for how long.

For example, if you want to know how good Twitter is for your business don't measure the number of positive tweets about your company. Instead, measure how many customers it drives to your product and how much money those customers put in your hands.

Understand the Downside

What would you do if your metric were 50% off the mark today? Would you be able to articulate why this is a problem for your business? What it costs you? Would you know where to start looking for possible causes?

As an example, I've worked with a startup that used "number of MySpace friends" as a go-to metric in every marketing meeting. Is that really material to the business?

What would happen if tomorrow we had half as many MySpace friends? Would we lose $100? $10,000?

This number says nothing about the value of MySpace as a marketing channel, which in the most charitable case is what it is trying to approximate, and the downside is completely ambiguous. Is the number of MySpace friends tied to anything meaningful?

Like the Twitter example above, if I thought MySpace were an important marketing channel for my product I'd be measuring things like the number of qualified leads from MySpace and the value they generate for the business.

Understand the Upside

Conversely, ask yourself, "What value does improving the metric bring to the company?" Some metrics are blindingly obvious in this regard, e.g., top-line revenue numbers and some "efficiency" metrics like effective CPM and revenue per user.

Some metrics are less obvious. What about page views? Can you imagine a scenario where your pageviews double but the net effect was bad for your business?

Metrics like these are dangerous because they lull you into a false sense of security. Everything on your analytics dashboard is going up and to the right, but the fundamentals of the business might still be suffering.

Don't Be Ambiguous

Does your metric measure one thing and one thing only? Or is it really an aggregation of several variables, each of which can rise or fall independently?

A good example of an ambiguous metric is the notion of a "daily active user," or the total number of people who interacted with your product today. This number is ambiguous because it is really the sum of two metrics: the number of new users and the number of returning users.

Ambiguous metrics are bad because they obscure the underlying variables that truly reflect customer behavior and delay the decision-making process as you are forced to determine which of these variables is actually affecting the aggregate metric. Furthermore, it's possible that one variable accounts for all the growth in the metric, e.g., you have millions of new users per day but no returning users.

This latter scenario has been the death of many Facebook apps.

Segment by Purpose

Whenever I'm building a web product I divide usage into key segments. Generally these are acquisition, retention, engagement, and monetization. That is, how do customers find my product? Are those customers coming back? Are they doing the things I want or need them to do? And how much money are those customers making me?

See Dave McClure's Startup Marketing for Pirates presentation, which focuses on this idea.

Doing this lets you tackle each segment independently and allows for a sharper product focus. Early on you might want to focus on acquisition, or maybe you want to focus on monetization from day one. Eventually you'll starting caring about longer-term metrics like retention and you won't be distracted or overwhelmed by unrelated metrics from different segments.

You might select one or two from each category to act as top-line variables in a dashboard that you look at every day, e.g., new users (by channel), returning users (by channel), activity, and revenue.

Appropriate Granularity

Sometimes you need a bird eye's view and sometimes you need a tunneling electron microscope. Know when you need which.

As a general rule of thumb I focus on the microscopic when I am designing specific optimizations but focus on the macroscopic when I'm determining whether the decisions we're making are working. Another way is to finish this sentence, "I know my product is healthy because..."

You'd never finish that sentence with "because the click-through-rate on my login page is 20%." You'd say something like "because 80% of my customers return every week" or "because our revenue is growing by 5% month-over-month."

Conclusions

These are just tips, not hard-and-fast rules. They are meant to focus the discussion of what metrics matter and why because the choice of metrics has a long-lasting impact if your team is committed to building a data-driven culture.

If you have any tips on deciding what metrics matter and why, leave them in the comments! This list wasn't meant to be exhaustive and I know (or hope) people will have some strong opinions!