How worried should we be that the feature we thought improved our product actually does nothing, or worse, hurts our bottom line? How can we ever really know that we’re making the correct decision? And is it better to run tests more quickly or more accurately?

The answers to these questions depend on the cost of a bad decision. If mistakes are cheap then it’s better to make 1,000 decisions and get only 60% of them right than to make 100 decisions and get 100% of them right.

One way to achieve this balance in the context of A/B testing is to tune the confidence level.

Tuning the Confidence Level

Intuitively, the confidence level of an A/B test tells you how certain you can be of the result of the A/B test. For example, a confidence level of 95% means that there’s a 5% chance that a statistically significant result is actually random variation, i.e., there is a 5% chance of a false positive.

Of course, we’re free to choose some other confidence level besides 95%. We could choose 80%, 90%, or 99.999%. A higher confidence level requires more data before reaching statistical significance, but we will be more certain of the result.

If you’re not comfortable with the nuts and bolts of statistical analysis, confidence levels, and A/B testing I recommend reading my article about statistical analysis and A/B testing, which explains exactly how one “chooses” a confidence level.

In short, the confidence level acts as a dial between speed and certainty, and we’re free to choose where to set that dial depending on the priorities of our business or product.

Speed vs. Certainty

So where on the speed-certainty spectrum should you, as a product manager or startup entrepreneur, sit?

Mike Cassidy has a great presentation where he argues that speed is the primary business startegy for startups.

Why is speed great for startups? Because mistakes are cheap and calculated risks are rewarded. Most product decisions can be undone, and important early tests can be redone at a higher confidence level when the product has more traction.

But mistakes aren’t always cheap. Here are some factors that increase the cost of a mistake.

Volume

Volume is leverage. If you have millions of customers, like Google or Amazon, a 1% improvement to the bottom line is a huge win. Conversely, a 1% mistake is a huge hit.

Fortunately this problem helps mitigate itself. Increased volume affords you the luxury of running A/B tests at a higher confidence level in the same amount of time.

Reversibility

Most product decisions in a consumer technology startup can be undone, for a price. For example, it’s easy to undo a bad decision for a web-based product, slightly harder to undo a decision for desktop software, and very difficult (and costly) to undo a decision for a physical product.

The less reversible a decision is the more certain you should be before you make it. In the context of A/B testing a product feature this means a higher confidence level, even if it takes longer to run the test.

Real Money

Imagine you’re an ad network. You’re constantly A/B testing formatting, positioning, offers, etc. to see which performs best. Making a mistake in this regard costs your publishers money.

Like volume, money creates leverage. But it is more complicated than that: publishers don’t just want increased revenues, they want reliable cash flow. That is, when money is involved, not only do you have to perform better but you have to perform more consistently because of phenomena like the peak-end rule.

In this case a “three steps forward one step back” strategy might actually be worse than going step-by-step in the right direction, even if the former averages out to better performance.

Conclusions

Maintaining momentum in a startup isn’t about making only correct decisions — it’s about making enough correct decisions. This presents a continuum from speed to certainty. At one extreme you run the business with a magic eight-ball; at the other you agonize over every detail until you’re 100% certain that you’ve made the correct choice.

This thought process extends naturally to A/B testing where the idea of “certainty” and “cost” can be quantified. To recap:

• A/B testing is a great tactical tool for testing specific hypotheses about your customers.
• However, there is a tradeoff between speed and certainty, controlled by the confidence level of the A/B test.
• The cost of doing A/B tests quickly is that you will make more wrong decisions, but that is ok if mistakes are cheap.
• For example, it’s better to make 1,000 decisions and get only 60% of them right than to make 100 decisions and get 100% of them right, all else being equal.

A Spreadsheet Model

Below is a little spreadsheet model that illustrates all my points above.

The two independent variables are the gain from a good decision and the cost of a bad decision. The spreadsheet assumes a fixed time period, so a higher confidence level means more certainty but fewer tests. The ideal confidence level is highlighted as you change the parameters of the model.

You can download the A/B testing confidence model here.

For the statistically inclined this model assumes that traffic increases linearly over time, that the sample statistic is normally distributed, and that a one-tailed t-test is the appropriate statistical test.

Credits

This article was inspired by a conversation with Sean Ellis and edited by my wonderful girlfriend, who is probably going to yell at me for linking to her Twitter account.

You can decide to measure anything, but what metrics matter and what ones are just for show? Here are some rules I hope will guide you toward creating meaningful metrics that help, rather than hinder, the decision-making process.

Be Actionable

If I had to give a one-sentence answer to the question “What metrics should I implement for my product?” it would be “Whatever metrics are actionable.” This means the line from a question to a metric and the line from a metric to an action should be as short as possible.

Most of the tips below are meant to focus attention on this issue. What can you do to make sure your metrics are actionable?

Be Understandable and Trustworthy

Do you understand what your metric measures? Does everyone in your organization also understand and do they trust it?

Trust is the important part. Everyone has to trust the metric if you’re going to use it to make decisions, otherwise you’ll be getting constant pushback. This will slow the decision-making process and cause a lot of ill-tempered arguments.

Measure Results

Does your metric measure customer behavior or a correlate of customer behavior? In the past approximations and correlations were necessary because measuring behavior directly was hard, but on the web there’s no excuse — you have access to every single thing a person does on your site, down to where their mouse is hovering and for how long.

For example, if you want to know how good Twitter is for your business don’t measure the number of positive tweets about your company. Instead, measure how many customers it drives to your product and how much money those customers put in your hands.

Understand the Downside

What would you do if your metric were 50% off the mark today? Would you be able to articulate why this is a problem for your business? What it costs you? Would you know where to start looking for possible causes?

As an example, I’ve worked with a startup that used “number of MySpace friends” as a go-to metric in every marketing meeting. Is that really material to the business?

What would happen if tomorrow we had half as many MySpace friends? Would we lose $100? $10,000?

This number says nothing about the value of MySpace as a marketing channel, which in the most charitable case is what it is trying to approximate, and the downside is completely ambiguous. Is the number of MySpace friends tied to anything meaningful?

Like the Twitter example above, if I thought MySpace were an important marketing channel for my product I’d be measuring things like the number of qualified leads from MySpace and the value they generate for the business.

Understand the Upside

Conversely, ask yourself, “What value does improving the metric bring to the company?” Some metrics are blindingly obvious in this regard, e.g., top-line revenue numbers and some “efficiency” metrics like effective CPM and revenue per user.

Some metrics are less obvious. What about page views? Can you imagine a scenario where your pageviews double but the net effect was bad for your business?

Metrics like these are dangerous because they lull you into a false sense of security. Everything on your analytics dashboard is going up and to the right, but the fundamentals of the business might still be suffering.

Don’t Be Ambiguous

Does your metric measure one thing and one thing only? Or is it really an aggregation of several variables, each of which can rise or fall independently?

A good example of an ambiguous metric is the notion of a “daily active user,” or the total number of people who interacted with your product today. This number is ambiguous because it is really the sum of two metrics: the number of new users and the number of returning users.

Ambiguous metrics are bad because they obscure the underlying variables that truly reflect customer behavior and delay the decision-making process as you are forced to determine which of these variables is actually affecting the aggregate metric. Furthermore, it’s possible that one variable accounts for all the growth in the metric, e.g., you have millions of new users per day but no returning users.

This latter scenario has been the death of many Facebook apps.

Segment by Purpose

Whenever I’m building a web product I divide usage into key segments. Generally these are acquisition, retention, engagement, and monetization. That is, how do customers find my product? Are those customers coming back? Are they doing the things I want or need them to do? And how much money are those customers making me?

See Dave McClure’s Startup Marketing for Pirates presentation, which focuses on this idea.

Doing this lets you tackle each segment independently and allows for a sharper product focus. Early on you might want to focus on acquisition, or maybe you want to focus on monetization from day one. Eventually you’ll starting caring about longer-term metrics like retention and you won’t be distracted or overwhelmed by unrelated metrics from different segments.

You might select one or two from each category to act as top-line variables in a dashboard that you look at every day, e.g., new users (by channel), returning users (by channel), activity, and revenue.

Appropriate Granularity

Sometimes you need a bird eye’s view and sometimes you need a tunneling electron microscope. Know when you need which.

As a general rule of thumb I focus on the microscopic when I am designing specific optimizations but focus on the macroscopic when I’m determining whether the decisions we’re making are working. Another way is to finish this sentence, “I know my product is healthy because…”

You’d never finish that sentence with “because the click-through-rate on my login page is 20%.” You’d say something like “because 80% of my customers return every week” or “because our revenue is growing by 5% month-over-month.”

Conclusions

These are just tips, not hard-and-fast rules. They are meant to focus the discussion of what metrics matter and why because the choice of metrics has a long-lasting impact if your team is committed to building a data-driven culture.

If you have any tips on deciding what metrics matter and why, leave them in the comments! This list wasn’t meant to be exhaustive and I know (or hope) people will have some strong opinions!

Metcalfe’s Law, which states that the value of a network grows in proportion to the square of the number of users of that network, express this idea formally. The value of a social network rests in its ability to foster communication, in its connections.

If you’re building a social network, whether it’s a destination website or an application that exists on another social network, density must figure into key strategic decisions. Let’s see how.

Islands

One way to think about social networks is as a network of networks. On Facebook, for example, if I graph the connections between all my friends I see distinct groups: my high school friends, my college friends, my current circle of friends, and my professional network.

Each group of people is more or less isolated from each other. Density exists within each of these islands, but not between them. I’m fairly certain this is a topological property of any social network.

Case Study: Facebook

Facebook started at Harvard and was initially college-only. Their growth strategy was explicit from day one: move from school to school as demand warranted. I’ve been told that Sean Parker wouldn’t consider opening up access to a new school until at least several dozen students from that school requested an account.

Each school was an island. Once Facebook saturated a specific set of colleges it moved onto the next round. Eventually there was enough anticipatory buzz that they could launch at large, state schools without risk of fading away.

They still pursue this strategy today. After establishing a critical density among colleges they opened up access to high schools and then to everyone with an email address. From there they started moving country-to-country.

The countries where Facebook is having the most difficulty gaining traction are the ones with already-established social networks, like Germany with StudiVZ. In fact, if you look at this old map of the most popular social network in each country, you get an idea of how isolated this country-by-country growth really is.

Other Networks

Facebook isn’t the only example of a social network who grew this way. hi5 has a similar story, starting with smaller markets overseas and spreading from country-to-country. Or Craigslist, by starting small in San Francisco and eventually becoming a presence in most major US cities.

The MySpace team had a background in direct marketing, which is all about targeting specific offers at the people who are most likely to respond. They started with the club scene in LA and grew from there.

The key to all these strategies was density.

If you’re launching a new social service, even if your end goal is to have everyone and their mother using it, it’s important to understand the impact density has on the growth

Multiple Dimensions of Density

So far the only kind of density we’ve talked about is network density, i.e., multiple people connected through their shared use of a service. You could call this “product density.”

Sometimes product density isn’t enough. Take IM, for example, or any network that requires synchronous communication. Not only do two people have to be using the same product but they have to be using it at the same time. What good is your friend being on IM if you’re never awake at the same time?

Xfire, an IM client for gamers, is an example of a product that innovated in this space by tackling a segment of customers who were already interacting synchronously.

Mobile social networks take this to an even greater extreme. To connect with people on Loopt or Google Latitude not only do we have to be using the same product at the same time, but we have to be in the same place!

This isn’t to say building these networks is impossible. Rather, they come with an extra handicap in the form of reduced density. Overcoming that problem has to be a key part of the product strategy.

Conclusion and Counterexamples?

Most product strategy discussions, in my experience, are focused on acquisition or other topline metrics that go “up and to the right.” Instead, if you’re building social software, I believe density is a necessary condition for long-term success and needs to be a part of the strategy discussion from day one.

First, understand the density requirements for your product. Do customers need to sign up for the same service? Do they need to be using it at the same time? Do they need to be in the same place? Is there anything you can do to lower the density requirement?

Second, build a “depth first” strategy. Are there any naturally dense customer segments that might fit your product? Do you have the ability to target specific demographics or segments for acquisition? Which ones respond positively and is it possible to build density there?

Once you’ve achieved sufficient density on one island hop to the next and repeat.

And if anyone out there can think of any counterexamples — social networks or services that got big “all at once” — leave a comment and let me know! I honestly can’t think of any.

I had two jobs. The first was to help manage the dorm computer labs. If a new virus broke out and computers were banned from the network because they were spamming everybody on campus, I had to go in and clean it up. I still remember when the Blaster worm infected most of the Windows machines on campus.

The second job was to help build a web-based dorm management system called Chopin. This system was at the center of the daily operation of the dorms. Students could use it to submit work requests and report problems with the dorms. Staff could use it to send out mass mailings, sell students printing credits for the dorm printers, and any number of other routine tasks.

Enter Verna

Verna was a front-desk clerk at one of the dorms. She was responsible for helping students if a problem came up and used Chopin to get her job done.

She had also lived in Hyde Park, the neighborhood surrounding the University, for most of her life and just wanted to do her job without having to deal with whiny students or crappy software. Chopin was supposed to help her do that.

One day while fixing the the front-desk computer I asked Verna, “What do you think of Chopin?” She immediately supplied a laundry list of complaints. It didn’t work well, it was confusing, she never knew where to go or what to do, and so forth.

It would have been easy to blame her. Maybe she just needed better training or maybe she wasn’t trying hard enough. But that was all ego: I was just upset because she told me the product I helped build was pretty awful.

Her critique was absolutely fair. I had built solutions I liked for problems I wasn’t even sure existed. In fact, before then, I had never honestly talked to any of the people who actually used Chopin about the product itself. In retrospect that seems completely insane.

And watching her use Chopin I saw so much waste. She would click ten times where two would have sufficed. Some features would never get used, others used for things they weren’t intended for. Once I was there, talking with her and watching how she used Chopin, I saw that the whole process was messed up, from top to bottom. Most of her energy was spent working around problems I had caused!

Go and See

If my job hadn’t required that I work on Chopin and get out of the office I never would have even realized there was a problem.

That experience taught me that whenever I didn’t understand a customer’s frustration or thought that maybe they were feeling this way or that way I should just go ask them before building solutions that might be worse than the problem. When in doubt, go and see for yourself. Actually, scratch that: always go see for yourself.

Too often I’d pass off mere belief as knowledge, or generalize from a specific set of circumstances to a fundamentally different set of circumstances. Maybe this solution worked over there, but why should it work over here? The only people who can validate your product are your customers — everyone else, including yourself, can wait their turn.

Verna taught me that.

This problem might seem artificial. Why not put up an ad on your product, or send a notification to every single person who might be interested? There are several reasons the number of people you can notify might be constrained.

1. There is a technical constraint, e.g., Facebook limits the number of application-to-user notifications at the API level.
2. There is a financial constraint, e.g., you’re sending notifications over SMS and every message costs you money.
3. There is a strategic constraint, e.g., sending notifications too frequently causes fatigue and reduces the effectiveness of future notifications.

So, the situation is not too far fetched. Let’s investigate the issue.

For the rest of the article the “application” is going to be a Facebook application and it can only send 100 application-to-user notifications per week. Which 100 users should we notify?

The Basic Considerations

In the linear cascade model when a user in a social network adopts a new behavior there is a probability that each neighbor in the network will adopt it.

Under this model we probably wouldn’t want to notify two people who are friends, and especially not a cluster of friends or a clique. The new feature wouldn’t spread very far beyond this group.

Likewise, we wouldn’t want to notify people who are very far apart on the social network because a user is more likely to adopt a behavior if more than one of his friends has also adopted it. So there is a balancing act between notifying users who are close together, to achieve density, and notifying users who are far apart, to achieve breadth.

Heuristics and Centrality Measures

The easiest solution is to pick 100 random users to notify, but this is also the most naive since it takes into account neither the structure of the network nor likelihood that a person will influence their neighbors.

A better¹ solution to this problem is to develop a heuristic that ranks every user in the network according to some metric. If we can only send 100 notifications then they are sent to the first 100 people on this ranked list.

The idea here is to use centrality measures to come up with heuristics. In graph theory “centrality” is a measure of how important an individual node is.

The simplest measure is called “degree centrality” and is equal to the number of neighbors of a node. On a social network this is the number of friends of a given user. So, if you wanted to send out 100 notifications using this heuristic we’d send notifications to the 100 users with the most friends. This heuristic involves convincing celebrities to use the new feature.

There are other, more complex heuristics. The Wikipedia article linked above has a list of other centrality measures, and I wrote an article about calculating eigenvalue centrality, which is similar to PageRank. Each of these admits a heuristic which can tell us which users to notify.

Of course, which strategy works best is hard to know beforehand, as it varies with respect to both time and the underlying notification. A/B testing this is difficult because the effects are intentionally dependent. If anyone has a good solution to this that doesn’t involve collecting massive amounts of data about user behavior I’d be interested in hearing it!

It should be noted that each of these heuristics only takes into account the underlying structure of the graph and not the probability of “infection.” By including the latter we can come up with a nearly exact model of the optimal subset of users to notify.

A Global Solution

Brendan Meeder at CMU pointed me to a great paper that discuss this very topic, Maximizing the Spread of Influence through a Social Network by Kempe, et al.

Rather than take a localized view of the problem by ranking each node individually, we create a statistical model of how the new feature propagates through the network.

First, we start with a finite seed set, A. In our case A is a set of 100 users. Say we convert each of these 100 users.

In our model if a user u is converted then for each neighbor v there is some probability

that v will also be converted.

After the process has run its course some set of users has adopted the new feature. Because adoption is probabilistic the size of this final configuration is a random variable. Using the notation from Kempe, et al., for a given seed set A the size of the final set of adopters is a random variable denoted by

Our goal is to pick the set A which maximizes the expected value of this random variable.

Formally, we want to find the subset A such that

is maximized, where

is called the influence function.

The Algorithm and The Results

It turns out that calculating the influence function exactly is NP-hard, but there is a greedy algorithm which approximates the value under certain (unrestrictive) conditions.

If you want more details read the paper linked above or the related Influential Nodes in a Diffusion Model for Social Networks by Kempe, et al.

Using Monte Carlo methods Kempe, et al. simulated the diffusion process using this algorithm versus several of the heuristics I described above. The results are fairly striking: their algorithm performs at least 18% beter than the best-performing heuristic (degree centrality) and 48% better than if the seed set were randomly selected. I’ve embedded a graph of their results below.

The “target set” is the initial seed set of users to notify, and the “active set” is the final set of users who actually adopted the new feature or product. The more users who adopt the feature the better the strategy.

Feasibility

Kempe’s algorithm is more feasible than many of the heuristics discussed above, although the best performing heuristic — degree centrality — is also the easiest to calculate. He also doesn’t include eigenvalue centrality in his analysis, which I’d be interested in comparing.

The biggest downside to his algorithm is that it requires both full knowledge of the underlying graph and an accounting of all the user-to-user transmission probabilities. Modeling these probabilities would require a lot of data about users over an extended period of time.

Whether the additional 18% is worth the extra computation and data collection depends on a lot on specific circumstances, but personally I’m going to try to implement it in my projects and see how the performance compares first-hand.

1. “Better” according to what? As we’ll see, randomly selecting seed users performs worse than all the other heuristics.

If Facebook wants to be the internet’s cable company what are they going to have to do to turn themselves into a $40Bn company?

Are Virtual Goods the Key?

Not all social networks are struggling to find a great business model. Tencent, a Chinese social networking company, pulled in over $1Bn in revenue last year, primarily through its use of virtual goods.³

But Facebook doesn’t need to look overseas to see that virtual goods could work for them. Most of the top Facebook games use virtual currency to make money, powered by leadgen-based ad networks like Offerpal and Gambit. There are reports that some of these apps are pulling in eight figures per year.⁴

And of course there’s Facebook’s own gifting service, which has recently moved to a virtual currency system, pricing gifts in “points” that can be bought with real money.⁵

All of this is to say that it appears that virtual goods are a natural business model for social networks and Facebook has enough data to see that. Why isn’t Facebook pursuing this strategy more aggressively? Why do they seem dead-set on building advertising technologies like Social Ads and Beacon?

The US Advertising Crutch

In the world of advertising not all countries are equal. US traffic is generally valued the highest, followed by other English-speaking countries, the G20 , and finally the rest of the world.

Until recently Facebook was concentrated in the English-speaking world. It’s the second largest social network in the US, after MySpace, and the largest in both Canada and the UK.

Unlike other social networks which don’t have a significant presence in the English-speaking world, Facebook can support itself through advertising. This is a crutch that prevents Facebook making bold decisions with their business model. I believe Facebook sees themselves as the next Google, one piece of technology away from changing the world of advertising.

The Demographic Crunch

Not all social networks have Facebook’s demographics, of course. hi5, the world’s third largest social network after MySpace and Facebook, has an extensive presence throughout Latin America and other countries which advertisers and publishers typically ignore. The same can be said of the advertising market in China, but recall that Tencent pulled in $1Bn last year through virtual goods.

It’s little wonder, then, that hi5 is aggressively pursuing a virtual goods strategy.⁶ Their demographics makes this strategy much more appealing. Facebook has the money and the audience to waste pursuing a pure-advertising strategy for social networks.

What once seemed like a demographic disadvantage might turn out to be a demographic advantage for hi5. Will they beat Facebook to the business model punch?

And a year from now will we be reading articles about Facebook’s virtual goods strategy compares to hi5’s, as opposed to articles about how Facebook’s new homepage compares to Twitter?

You’re crazy. You know that, right?

Obviously hi5 has an uphill battle. Facebook is growing on the order of 500,000 new users per day and shows no signs of slowing. But the same was said of MySpace and Friendster when Facebook launched. I think we still have a few more twists in the story of social networking on the web, and this is just one possible twist among many.

1. Microsoft acquires equity stake in Facebook, expands ad partnership (cnet)
2. Update On Facebook’s Dubai Fundraising Trip (Business Insider)
3. The world’s most lucrative social network? China’s Tencent beats $1 billion revenue mark. (VentureBeat)
4. Developer Analytics: Facebook game Mob Wars making $22,000 a day (VentureBear)
5. Gift Shop Credits Have Arrived (Facebook)
6. Hi5’s virtual entertainment plans could hit a virtual jackpot (VentureBeat)

In this article I’m going to outline two models that describe how new behaviors, ideas, and messages propagate through social networks.

The Threshold Model

The first model is called the Threshold Model.¹ It says that people adopt a new behavior bceause a sufficiently large proportion of their friends have adopted that behavior. Early adopters have a very low threshold, say 5% or 10%, while late adopters would have a much higher threshold. Every person, however, has their own individual threshold.

For example, my girlfriend’s stated reason for signing up for Twitter was that “all my friends were using it.” And during the 2008 US Presidential election, some Obama supporters would adopt Hussein as their middle name.² When I saw that lots of my friends were doing it I was certainly tempted to do the same.

The underlying psychological principle is one of “missing out” or “when in Rome.” The key variable here is the initial distribution of thresholds across a social network, which describes in totality the final extent of the behavior.

It’s worth noting that this model says nothing about how people initially adopt behavior. That is, it says nothing about innovators, only about the spread of innovation through a social network.

The Cascade Model

The second model is called the Cascade or Word-of-Mouth Model³, and is the method of “viral growth” that most social application developers are familiar with. It says that every person has a chance of adopting a new behavior whenever one of their neighbors adopts it.

This model describes phenomena like product recommendations or user-to-user notifications on Facebook. The probability that a person adopts the new behavior is the conversion rate for the notification.⁴

This probability is both a function of the sender and the recipient, so more influential people are more likely to convince you to adopt a behavior (or purchase a product, or install an application).

Practical Implications

Both of these models describe facets of real-world interaction on social networks. My take is that the cascade model is more accurate at the beginning of a social network’s life, where behavior is spreading through sparse areas, connected by influencers. Later on, after a critical density has settled in, people start adopting the behavior because everyone else is adopting it and there’s a social cost to not doing the same.

We see this pattern in services like Facebook and MySpace, both of which got their start by harvesting emails and spreading through word-of-mouth (and spam) across a social network.⁵ Eventually each network reached a point where a sufficient number of people were familiar with the product and new users adopted it not because their friends recommended it (the cascade model), but because there was a social expectation that they do (the threshold model).

Also, with respect to analytics and viral growth, the threshold model is more difficult to track. In the cascade model we record who sent what to whom and which messages they responded to. It’s clear who gets credit for a user’s conversion. In the threshold model you have to track passive exposures, and there’s no clear causal relationship.

If ten of my friends are doing something and I decide to start doing the same thing, who gets credit? Most analytics packages will show this behavior as a direct visit, with no connection to other users’ behavior, even though there is a viral process underlying it.

In short, the threshold model requires a certain level of behavioral density, while the cascade model doesn’t. However, we see both models expressed in how people actually adopt new behaviors in social contexts.

Formalisms

In the threshold model every person u has a threshold

and each of their neighbors v is weighted according to

If

then the person u adopts the behavior.

The set of thresholds, weights, and initial adopters completely determines the extent of the behavior in the social network.

In the cascade model, for every person u and neighbor v there is a random variable

which describes the likelihood of u adopting the behavior if v has adopted it.

Takeaways

I’ll try to boil all this down into a few, practical takeaways.

1. The Threshold and Cascade Models describe two mechanisms of behavior adoption in social networks.
2. The Threshold Model says that people do something if enough of their friends are doing it.
3. The Cascade Model says that people have a chance of doing something if one of their friends is doing it.
4. Both models correspond to different real-life adoption patterns.
5. The typical “viral loop” involves the cascade model, but most successful social networks rely on the mechanics of the threshold model in the long run, i.e., density is important for long-term success.
6. The cascade model is a good tool for analyzing acquisition scenarios, but the threshold model is probably more helpful for understanding retention and engagement — it at least implies that density is a key factor in social network growth, a metric that’s not often discussed publicly.

Agree? Disagree? Leave a comment, send me an email, or follow me on Twitter!

1. See Threshold Models of Collective Behavior (1978) by the famous sociologist Mark Granovetter.
2. See Obama Supporters Adopting Middle Name “Hussein” As Their Own
3. See Talk of the Network: A Complex Systems Look at the Underlying Process of Word-of-Mouth (2001) by Goldenburg, Libari, and Muller.
4. More accurately, we’d model the “probability” as a random variable whose mean was the conversion rate.
5. See Stealing MySpace: The Battle to Control the Most Popular Website in America for details about the MySpace team’s background in direct marketing. The ConnectU vs. Facebook court documents, which you can find via Google, paint a similar story for Facebook’s early years.

“Going viral” has become a sort of holy grail and most people would say they’d rather have a viral application than not, but it has a distinct downside: uncontrolled growth and ever-increasing operational costs. By being almost-but-not-quite viral you can dramatically reduce your cost of acquisition without setting your servers on fire.

The Paid Strategy

The key variables for a paid strategy are cost of acquisition and average revenue per user, or ARPU. If your ARPU is greater than your cost of acquisition then you can buy as many users as your budget allows, reinvesting the new revenue into acquiring new users. Generally this is done by advertising your product through something like AdWords or Facebook’s Social Ads.

The best thing about a paid strategy is its relative simplicity. If your ARPU is $2.00 then you can run as many ads you want so long as the cost of acquisition is less than $2.00 and still be profitable.

Eventually, though, you will run out of ad inventory. There are only so many publishers who are willing to accept $0.10 per click. At this point your only options are to decrease your cost of acquisition¹ or increase your ARPU.

The Viral Strategy

The viral acquisition strategy requires you get current users to invite other users to the application. The two key variables for the viral strategy are the average number of invites each new users sends and the rate at which those invites convert into new users.

The ratio of converting invites to new users is your viral coefficient, k. If this is greater than one you will see self-sustaining, viral growth.² If the coefficient is less than one each user will bring in a fixed number of new users, but the application’s growth is still linear.

People who have decided on a viral acquisition strategy focus on this number obsessively. It’s the first big hurdle to clear and if you haven’t had experience engineering a viral application it can take months to build something viral.

But being viral isn’t an either-or proposition. Increasing your viral coefficient from 0.5 to 0.8 has other advantages, especially if you integrate it with a paid strategy. Let’s see how.

A Hybrid Strategy

Say you’re building a game on Facebook backed by a virtual currency. Users give you money or fill out offers to get coins, so you have a positive ARPU. This means you’re free to pursue a paid acquisition strategy.

On the other hand, you’re on Facebook, which has many viral hooks. Many of the technical hurdles are much smaller there, so it becomes more a question of design and optimization rather than implementation. At the very least you encourage players to invite their friends.

The first version of your application has a viral coefficient of k=0.5, that is, every new user who joins brings in 0.5 new users. Equivalently, for every two users you acquire you get one free.

That’s interesting, especially if you’re also paying for users. If you’re paying $1.50 per user then you paid $3.00 to get two users, but acquired a third user for free! This means that you effectively paid $1.00 per user: $3.00 paid / 3 users.

This process is actually geometric, however. If you purchased 4 users with a viral coefficient of 0.5, you’d first get 2 new users for free. These 2 new users would then bring in 1 additional user, for a total of 3 new users, reducing your cost of acquisition even further. This is an infinite geometric series, which I’ll outline below.

Having a non-zero viral coefficient reduces your effective cost of acquisition.

This means that you get more users for every dollar you spend on ads. Or, if you’ve run out of inventory, this means you can now spend more per user and retain the same net revenue level.

Let’s figure out how to calculate your effective cost per acquisition.

Effective Cost of Acquisition

If you just want the formula, here it is:

Where C is your cost of acquisition, k is your viral coefficient, and C’ is your effective cost of acquisition.

If k=0, i.e., you have no viral acquisition, then C’ = C.

If k = 1 and your application is viral then C’ = 0 and your application grows without spending any additional money. But rather than being an either-or proposition — you’re either viral or you’re not — there’s a sliding scale. The more viral you are the cheaper it is to acquire users.

The Benefits of Being Almost Viral

From the above formula, if you have a viral coefficient of 0.90 then you have reduced your cost of acquisition by 90%. This is a great situation to be in. You might ask, “Why wouldn’t I want go the last 0.10 and make my application viral?”

The one benefit of being viral is huge growth, which looks sexy on a graph and can tip an investor to your side if you’re looking for outside money, but the growth is unpredictable. Not only do you have little control over what demographics come to dominate your application, but sometimes it grows so quickly that you run into operational problems (servers on fire, etc.).

By being almost viral you can grow very cheaply, control your rate of growth and demographics, and get enough traffic to conduct meaningful experiments. Need to grow more slowly? Just decrease your daily ad spend. Need statistically significant results more quickly? Increase your daily ad spend.

Put another way, with a viral coefficient of 0.9 you’ve dealt with your acquisition risk. Rather than going fully viral and dealing with the operational difficulties, it might be worth your time to deal with other market risks: retention, engagement, and monetization.

So, stop sweating about “being viral.” Sometimes it’s better to be almost viral.

Deriving the Formula for Effective Cost of Acquisition

You can skip this and go right to the comments if you’re not interested in the math.

We have an application with a viral coefficient of k = 0.5. Every new user who joins brings in 0.5 new users. Another way of thinking of it is that for every new user who joins there is a 50% probability that he will bring in another user.

But this potential user also has a 50% chance of bringing in a new user, so the expected number of users is now 1 + 0.5 + 0.5*0.5. This continues ad infinitum.

Formally, if we acquire one user and have a viral coefficient of k then the number of users we expect to join is N(k), given by the formula

The initial 1 is our original user, whom we paid for, and each k represents the expected number of users from each step in the viral process.

This is a geometric series, so we know that

Therefore, our effective cost of acquisition is

1. If you increase the conversion rates for your ads then you can pay more per click to get more impressions without hurting your bottom line — you pay the same to acquire a user, but get more users through the door.
2. See my article Three Myths of Viral Growth for more information about viral growth.

It doesn’t matter how large you are, it doesn’t matter what your goals are, and it doesn’t matter what your product is. I think if you’re building a social application then you’re trying to build a new social network. As we’ll see, this has both strategic and technical implications.

What is a Social Network?

First, if I’m going to convince you that something is a social network we should understand what a social network is. If you ask a person to name a few social networks, they will probably list services like Facebook, MySpace, and Twitter. And if an investor tells you they’re “not investing in social networks,” they mean it in this concrete, social-network-as-a-product sense.

Others, like Brad Fitzpatrick and Mark Zuckerberg, use the term social graph² to distinguish between the underlying social relations between people and the services, called social networks, that are built on top of them.

But if there’s one thing I learned from my mathematics education it’s this: we’re free to define things however we want so long as they’re consistent. Therefore we ought to choose the definition that helps us get our job done.

So, here is my first, and most abstract definition:

A social network is a collection of people bound together through a specific set of social relations.

By “social relation” I mean a connection between people that permits the exchange of information. This prevents artificial relations like “Alex and James are connected if they have the same hair color.”

When I say “social network” I always mean the actual collection of people. Facebook is a social network. There are actual people engaged with the site, creating relationships, sharing information, and doing all the things they’d do in “real life.” Or, put another way: a family is a social network, a family tree is not.³

If you don’t like the above definition I can give you a functional one which I believe is equivalent.

A collection of people is a social network if and only if it is possible for something to spread virally through that collection.

In Web 2.0 speak, a “social network” is a collection of people over which you can “go viral”. I believe that virality and social networks are fundamentally linked, and that both the above definitions are equivalent.

Social Applications are Social Networks

Accepting the above definitions, even if for the sake of argument, I don’t think it’s too hard to see why social applications are social networks. Let’s take Slide’s Top Friends as an example. Is Top Friends a social network in its own right?

I think it’s easier to see that Top Friends meets the first definition. It is certainly a collection of people: the set of Facebook users who have installed the application. Are those people bound by specific social relations? Yes, and those relations are distinct from the ones represented in Facebook. For example, Alex adding James as a top friend is a social signal distinct from Facebook.

What about the second definition? Top Friends doesn’t have an external API so it’s impossible to build apps or plugins for Top Friends.⁴ So, what “goes viral” over Top Friends? New features and patterns of usage do.⁵

I’d also argue that the converse is true: social networks are all social applications. YouTube spread through MySpace, Facebook spread through email, email spread through the real-life “social graph”, and PayPal spread through eBay.⁶ All social networks are social applications built off of pre-existing social networks.

Strategic Implications

If Top Friends is a social network in its own right then there are strategic implications for Facebook. Prima facie, Top Friends is competing with facebook for users’ attention on its own platform. Before Facebook launched the Platform it was the Eye of Providence, collecting, collating, and analyzing every bit of activity that occurred on its network.

After the Platform launched these third parties were able to infect portions of Facebook’s network. In some cases, e.g., the Causes application, the relationship was symbiotic. In others, e.g., Top Friends, the relationship was antagonistic, with Facebook actually shutting down Top Friends at one point.⁷

What does Facebook gain by having Top Friends on its Platform? Nothing substantial, as far as I can tell. What does it lose? Control and insight over the activities of its userbase.⁸

In effect, Top Friends is a social network bootstrapped off of Facebook, with its own set of communication channels over which Facebook has no authority or insight. This tension is present everywhere in the Platform because application developers’ interests are not wholly aligned with Facebook’s and will probably never be.

Technical Implications

I’m going to save the technical implications for another article, but it boils down to this: social networks in the sense that I defined above are fairly well understood. I believe the techniques used on the web today to grow “viral” applications are under the research from fields like social network analysis and epidemiology.

Since I believe social applications and social networks are synonymous, we can better understand how these applications grow by understanding how social networks grow.

In the meantime, I recommend reading The Statistical Evaluation of Social Network Dynamics by Tom A. B. Snijders from the University of Groningen if you’re interested in the technical aspects of social networks and social applications.

And please, leave a comment if you have any thoughts about the above!

1. Google Open Social + Friends vs. Facebook Platform
2. See, e.g., Thoughts on the Social Graph
3. Ceci n’est pas un Social Network
4. For all I know Slide has an internal Top Friends API that lets them build new services that ride on Top Friends’ success, but that’s only speculation.
5. This is the essence of engagement loops. Eric Ries talks about going “beyond viral.” There is no “beyond viral.” Rather, on social networks viral processes govern the whole stack: acquisition, retention, engagement, and monetization.
6. Slide is to Facebook as Paypal was to eBay. Anyone buy it?
7. See this TechCrunch article.
8. There’s a broader argument that ceding control in this way is the right strategic move, but Facebook is not there yet — the limit of that argument is something like OpenSocial.

For music startups iTunes figures into most strategic decisions. If you’re streaming music for free to consumers you’re going to be an iTunes affiliate³. If you’re selling music to consumers you’re going to competing directly with iTunes — consumers have no reason to get their music from both you and iTunes if you both have it.

It’s understandable if your heart skips a beat when you catch rumor that Apple will be building a similar product. How can you maneuver in this environment?

Finding Room to Breath

The iTunes Store is a lot like Wal-Mart: ubiquitous⁴, highly integrated, and bland. People shop there because it’s easier, not because it’s sexier, even though in other areas Apple is very good at selling precisely that hip, sexy lifestyle.

But Wal-Mart’s strategy isn’t the only strategy out there. Companies like Whole Foods can still thrive in their niche even though people can get cheaper food at Wal-Mart. Where is the Whole Foods of digital music? Does such a thing even make sense?

Building a Community

Corner record stores are about more than just the transaction. They attract a certain crowd and embrace a certain culture. High Fidelity is a good example of this on film.

iTunes misses out on the cultural and communal aspects of music altogether. It’s very sterile. It’s also a terrible means of discovering new music, a role which traditional record stores can fulfill.

As an example, say you’re really into electronica. What good are the reviews on the iTunes music store to you? They’re left by idiots who don’t know Aphex Twin from Paul van Dyk. You go there when you know what you want and leave the second you have it⁵.

And knowing iTunes, they might not even have music from your favorite bands if they’re obscure enough.

Instead, imagine a hub of engaged electronica fans with a custom, electronica-centric store⁶. The community itself spurs demand for its own store due to its reputation for quality electronica-related recommendations.

Improved discovery, better quality merchandise, exclusive deals with bands, and a community of like-minded people are just a few reasons why people might prefer to shop at a genre-specific music store rather than iTunes if they’re forced to choose.

Plus an independent store is more freely able to experiment with payment models, distribution methods, and marketing campaigns. This might interest bands who see iTunes as a love-it-or-leave-it environment controlled from top to bottom by Apple.

Will This Work?

I don’t know if this will work, but I think it’s a reasonable enough hypothesis to test. This is just one possible strategy for building a music product and probably has several flaws I haven’t thought through. Leave a comment and let me know your thoughts.

May a thousand music stores bloom!

Update

Adam from Heroku pointed me towards Beatport, which has been pursuing this exact strategy for the last few years.

After a little digging I found a few others, too. Insound.com for indie music and Mondomix for world music. I also know of one stealth startup pursuing this strategy for another genre. Do you know of any others? How well does this strategy perform?

In the limit you can imagine a “Ning for iTunes Stores,” where the costs of implementing the store are shared but the community-building aspects are left to the company.

1. iTunes Store Top Music Retailer in the US
2. iTunes Sells 6 Billion Songs, And Other Fun Stats From The Philnote
3. Both imeem and Last.FM are, for example
4. Who did Apple pass as the top music retailer? Wal-Mart
5. This is a problem with iTunes in general. It’s the last step in your marketing campaign, not the first. See my article The $0.99 (App) Store.
6. Of course, you can break down genres into subgenres, and so forth. Maybe the sweet spot is having an ambient store, a trance store, a house store, a D&B store, etc.