It depends on your funnel consistency. If you are selling a self service product with transparent pricing, fully realized ARPU can be a solid number. It will change as your customer base changes, and there still could be outliers, but it's decent and easy to reason about (and project with).
If, however, you are kind of self service, but do an Enterprise sales deal ... you have two funnels with two vastly different customer profiles. Not only is ARPU bad in this scenario, most numbers are.
The trick is narrowing the scope of a given metric enough that you can use it to make good decisions, and but not so much that it ignores important parts of a business.
I am vaguely reminded of a recent discussion* about negotiating with Steve Jobs where kind of the reverse point was made: The author was advised to make the number fit the scenario Steve claimed he wanted. The author did so, creatively, without lying.
A lot of people do not understand the substance behind the numbers and this leads to garbage in, garbage out. That's what this article is about: Understanding what's behind the numbers and not being fooled by them. The previous piece was also about understanding the substance and knowing how to work the numbers to make other people happy with the proposed deal.
A good read on similar topics: How to Lie with Statistics.
Absolutely – one of my favorite examples about the importance of visualization, and why summary statistics often don't do a great job of summarizing a distribution.
So what indicator statistics are good, then? "Take a look at the entire picture, not just the average" is great, but what am i going to put on my dashboard? What am i going to make my goal for the year?
On the technical side, when trying to boil down scads of metric datapoints into a single number, the statistics i usually end up using are the median and the 95th centile (or some higher centile). The median gives some sort of rough idea of where the middle of the main mass distribution is, ignoring the extremes, and the high centile gives an indication of where the top edge of the main mass is, ignoring freak outliers.
Would those be any use for revenue per customer?
Would we be better off with the median and some number that tries to capture the shape of the power law? Something that says "for every dollar you go up in revenue, the number of users drops by X%", or something like that?
One of the things to take away is that it's often counter-productive to take one summary statistic like "mean" or "median" and try to just optimize that. The correct strategy is likely to be a bit more nuanced and depend on your specific situation.
For example, maybe you have a few different pricing tiers, and instead of optimizing for ARPU, you optimize for the number of people in a given tier that you can up-sell into a higher pricing tier. Now you're optimizing for something meaningful. Note that even if you did a good job in accomplishing this goal, your ARPU might actually go down, if you simultaneously saw a lot of growth in your lower pricing tiers from new signups.
There are a lot of ways of looking at this data. But if it's usage based I would suggest ranking things by revenue to see what types of customers are the most valuable.
10% comes from the top 0.1% who spend at least X$,
10% comes from 0.1%-0.9% who spend at least Y$
etc.
Or if you have some category's like 5$/month accounts = X% of revenue 50$/month accounts = Y% etc.
The value is to know what types of users you should focus on just be careful as you need a lot of data to make some of this stuff meaningful.
I'm not sure I follow this, or buy into the suggestion of the post.
First, I don't see how it's true that data with a relatively large amount of variance will tend to be power law distributed. Defining what a "large amount" of variance is is tough (it depends on your intuition and choice of variance metric) but there are lots of distributions with considerable variance that are, for example, normally distributed (many more than are power law distributed, as far as I can tell).
Second, if you find that this is misleading your projections, why not just use a different kind of average? For example, if you just want to know, "How much is the next customer likely to spend?", you might use the mode. Or, if you want a more robust average (i.e., less likely to be seriously thrown off by outliers), why not use the median? You can even complement these with confidence intervals if you want to get a sense of their precision.
Like twic already said, you need some indicator to understand what's going on with your business. I think that in many cases, this will be the mean. But if you want something more robust or more practical, perhaps the median or mode might suit you better.
I should also add... you can also use median absolute deviations, standard deviations, or interquartile ranges to identify and remove outliers who you think don't reflect your business's true status. But it all depends on what you want your model to do!
To address your first point – that's fair, I didn't really dive into the statistics in detail (since it wasn't really the point of the post). If variance of a distribution is finite then the central limit theorem applies, and given a sufficient number of trials, the distribution will begin to approach a normal distribution. However some data sets have infinite variance and may (under some conditions) begin to approach a power law distribution; or, they have finite but extremely large variance, in which case the number of trials it will take to begin to look like a normal distribution is very, very large.
For your second point – there are legitimate uses for other summary statistics (mode, median), but they can still be very misleading. For example, you mention using the mode of the distribution as a predictor of what the next customer will pay – this definitely wouldn't work for a company with a metered pricing plan, for example. Distributions are often not well characterized by singular summary statistics, Anscombe's Quartet (mentioned elsewhere in this comments section) is a good example of this: http://en.wikipedia.org/wiki/Anscombe's_quartet
Re: power law distributions. I'm not sure what you mean about finite versus infinite variance. Are you referring to whether you are analyzing a bounded versus unbounded scale? Even if a scale is unbounded, that still wouldn't make a dataset any more likely to power law distributed. A power law distribution would be, however, somewhat more likely to be observed in cases where there is only an upper- or lower-bound (though again, not always... it really depends).
Re: appropriate averages. Your case (metered billing) is an interesting one. I don't see why the mode is necessarily wrong - the next customer is most likely to spend the amount that is currently your most popular amount). However, in order to calculate a mode, you likely would want to bin your amounts into ranges so that $10.11 isn't treated as distinct from $10.12, etc.
You're definitely right about one summary statistic not being sufficient. I would advocate for visualization and summary statistics, with some estimate of your confidence in the estimate displayed visually.
Well, you can think that averages are meaningless, but they do mean something. If your average sale is $10 and you are spending on average $9 to acquire them, you have a 10% margin on average.
Segmenting is great and can actually move the averages wildly, but that doesn't mean the averages are pointless. It means that averages are just one view of the data.
If I want a really high level view of the data, an average is great. If I want a super low level view of the data, looking at each customer or small groups of customers is great too. It depends on what you want to know and what problem you are trying to solve.
While I agree with the spirit of this, the conclusion is not necessarily correct. The mean can be highly informative, but should never be used alone.
Assume you know only the mean revenue and the maximum revenue (but forgot to measure variance). You could make an extreme scenario with the maximum possible variance to generate a "worst case" distribution. In this scenario, all customers either provide zero revenue or the maximum. This distribution has the maximum possible variance for a given mean and maximum.
Will you be profitable this year? Your chances will be better than the worst case scenario described above! If higher moments are known (variance, skew, etc.), more accurate bounds can be found.
In conclusion, the mean can be very useful, especially if higher moments are known.
It's trivially easy to say "metric x is meaningless" if you assume the person interpreting it doesn't actually understand the context of that metric. You might as well say "metrics that aren't based on a solid understanding of your business model and user behavior are meaningless." Of course they are; this is not useful information.
If you run analytics on bad or incomplete information or information that doesn't realistically relate to something important to your bottom line, or if you try to interpret useful data without knowing what you should actually be looking at in the context of the service you're providing, it'll obviously be meaningless.
14 comments
[ 2.0 ms ] story [ 36.3 ms ] threadIf, however, you are kind of self service, but do an Enterprise sales deal ... you have two funnels with two vastly different customer profiles. Not only is ARPU bad in this scenario, most numbers are.
The trick is narrowing the scope of a given metric enough that you can use it to make good decisions, and but not so much that it ignores important parts of a business.
A lot of people do not understand the substance behind the numbers and this leads to garbage in, garbage out. That's what this article is about: Understanding what's behind the numbers and not being fooled by them. The previous piece was also about understanding the substance and knowing how to work the numbers to make other people happy with the proposed deal.
A good read on similar topics: How to Lie with Statistics.
* https://news.ycombinator.com/item?id=7451018
On the technical side, when trying to boil down scads of metric datapoints into a single number, the statistics i usually end up using are the median and the 95th centile (or some higher centile). The median gives some sort of rough idea of where the middle of the main mass distribution is, ignoring the extremes, and the high centile gives an indication of where the top edge of the main mass is, ignoring freak outliers.
Would those be any use for revenue per customer?
Would we be better off with the median and some number that tries to capture the shape of the power law? Something that says "for every dollar you go up in revenue, the number of users drops by X%", or something like that?
For example, maybe you have a few different pricing tiers, and instead of optimizing for ARPU, you optimize for the number of people in a given tier that you can up-sell into a higher pricing tier. Now you're optimizing for something meaningful. Note that even if you did a good job in accomplishing this goal, your ARPU might actually go down, if you simultaneously saw a lot of growth in your lower pricing tiers from new signups.
The value is to know what types of users you should focus on just be careful as you need a lot of data to make some of this stuff meaningful.
First, I don't see how it's true that data with a relatively large amount of variance will tend to be power law distributed. Defining what a "large amount" of variance is is tough (it depends on your intuition and choice of variance metric) but there are lots of distributions with considerable variance that are, for example, normally distributed (many more than are power law distributed, as far as I can tell).
Second, if you find that this is misleading your projections, why not just use a different kind of average? For example, if you just want to know, "How much is the next customer likely to spend?", you might use the mode. Or, if you want a more robust average (i.e., less likely to be seriously thrown off by outliers), why not use the median? You can even complement these with confidence intervals if you want to get a sense of their precision.
Like twic already said, you need some indicator to understand what's going on with your business. I think that in many cases, this will be the mean. But if you want something more robust or more practical, perhaps the median or mode might suit you better.
For your second point – there are legitimate uses for other summary statistics (mode, median), but they can still be very misleading. For example, you mention using the mode of the distribution as a predictor of what the next customer will pay – this definitely wouldn't work for a company with a metered pricing plan, for example. Distributions are often not well characterized by singular summary statistics, Anscombe's Quartet (mentioned elsewhere in this comments section) is a good example of this: http://en.wikipedia.org/wiki/Anscombe's_quartet
Re: appropriate averages. Your case (metered billing) is an interesting one. I don't see why the mode is necessarily wrong - the next customer is most likely to spend the amount that is currently your most popular amount). However, in order to calculate a mode, you likely would want to bin your amounts into ranges so that $10.11 isn't treated as distinct from $10.12, etc.
You're definitely right about one summary statistic not being sufficient. I would advocate for visualization and summary statistics, with some estimate of your confidence in the estimate displayed visually.
Segmenting is great and can actually move the averages wildly, but that doesn't mean the averages are pointless. It means that averages are just one view of the data.
If I want a really high level view of the data, an average is great. If I want a super low level view of the data, looking at each customer or small groups of customers is great too. It depends on what you want to know and what problem you are trying to solve.
Use the right tool for the job.
Assume you know only the mean revenue and the maximum revenue (but forgot to measure variance). You could make an extreme scenario with the maximum possible variance to generate a "worst case" distribution. In this scenario, all customers either provide zero revenue or the maximum. This distribution has the maximum possible variance for a given mean and maximum.
Will you be profitable this year? Your chances will be better than the worst case scenario described above! If higher moments are known (variance, skew, etc.), more accurate bounds can be found.
In conclusion, the mean can be very useful, especially if higher moments are known.
If you run analytics on bad or incomplete information or information that doesn't realistically relate to something important to your bottom line, or if you try to interpret useful data without knowing what you should actually be looking at in the context of the service you're providing, it'll obviously be meaningless.