I thought I was pretty statistically literate, but learned a couple of new things here in the discussion, and this was not a tool that I would have thought of using before (now, in certain situations, I might think of it).
The trimmed mean and Winsorized mean are both super useful in monitoring and metrics systems. In most cases you don't actually want the median but you also don't want the extreme outliers to throw everything off with a mean.
Both methods give you better metrics for comparing periodically, like day over day or week over week.
For instance, all commercial A/B testing services (as far as I can tell) compute the 99%-Winsorized stats for their reports. This is partially due to several common cases that can mess up an average computation, especially between two halves: How many times has the developer testing account clicked on “Buy”? Obviously, those should not been the dataset in the first place, but when dealing with corporate clients, the commercial thing to do is to fix the problem and not confuse the client with the details, unless they dig in the documentation.
I've seen published estimates of statistical efficiency for a few test distributions for the trimmed mean, but not the Winsorized mean. Can anyone suggest references on this?
Any of: low amount of data points; wanting a continuous value if your points are integers; slightly different behavior in high dimensional vector spaces.
For a one-sided, unbounded distribution when you still want to observe changes without being susceptible to outliers.
If you're monitoring response timings on a server, for example, the median might be very close to 0, and it won't shift unless a majority of the distribution slows down. If you take a winsorised mean, you can trim useless long response times that mess with the mean, but still see if e.g. 1/3 of your responses are suddenly slower than normal.
When processing astrophotos, multiple exposures are “stacked” together. There is a certain amount of noise in each frame - due to electronic noise or simply the random number of photons that strike a pixel in any one exposure - that you want to average out on a per-pixel basis.
However some frames may contain unwanted outliers, for example if a satellite briefly passes overhead it will appear as a very bright streak in only one frame.
By winsorizing, outlying pixel values can be eliminated while still maintaining the same number of samples per pixel as the rest of the stacked image.
A lot of financial data is of dubious quality. Lot's of spurious data points, mostly in the extremes, just aren't real. Median is useful for many things, but sometimes you just have to get rid of the bad data to get a valid mean.
I'm not sure I would feel comfortable using the Winsorized mean -- it doesn't have any particular statistical properties, and it lacks any intuition appeal because it's not clear what the value represents.
I can understand a line of logic that would give rise to something like the Winsorized mean -- after you look at your data, you see some obvious outliers. It feels dirty to just drop those values (which would lead to the truncated mean) because the information from an implausible value is more likely to be near the extreme than it is to be near the center mass.
What to do with those extreme values?
Here's something I now want to experiment with -- bootstrapping the extreme values. Take note of the original empirical distribution. Then, create a new distribution by removing the top and bottom X% of the observations and replacing them with values drawn i.i.d. from the original empirical distribution. This could lead to some values being replaced with the outliers that we originally wanted to drop. After we do this, record the mean. Then create new sample distributions until we have a distribution of new means. What I am curious about is how the shape if this distribution of means will be impacted depending on that X% value selected at the beginning.
What are some well-known distributions that appear to have outliers? A log-normal distribution maybe?
My surface level take is they are similar to M-estimators. Whereas M-estimators are more mathematically rigorous, Windsorized metrics might be easier to compute manually.
It does feel like it's a very early 20th century technique. Nowadays we have so many tools which would be less feasible for calculators (the people) and more feasible for software.
Is taking logs (or arcsinh or whatever) really all that good an idea if (a) you don't have a good physical model justifying it or (b) your data spans several orders of magnitude?
It makes nonlinear relationships linear. Makes the model less sensitive, too. For instance if the data spans several OoM, adding or removing one datapoint in one of those orders can generate a lot of skew before the log-linearization.
It's easy to cast the log back to the original distribution by taking the exponent afterwards.
As far as I understand directly transforming your data can lead to problems. In any case, its what link functions do better in generalized linear models[1].
The reason why the arsinh transformation is useful (and this is not mentioned in the link you posted) is that it is the optimal variance-stabilizing transformation [1] under the assumption that your data is contaminated by a mixture of additive and multiplicative noise (the same way that the log transformation is the optimal variance-stabilizing transformation when your data is contaminated only by multiplicative noise).
Read the Wikipedia article for a more formal explanation.
I think it makes a lot of sense. The best replacement for an outlier is the closest thing to the outlier that isn't an outlier. Resampling doesn't make a lot of sense because your new point is completely disconnected from your old one.
I don't really like the Winsorized mean but not for the reason you list. I think the main issue is that you are assuming exactly the top and bottom 10% are outliers instead of looking at the actual data distribution to see what the outliers are then using a similar replacement technique on only the outliers.
The tradeoff is that while there are many estimators of the expected value that are more efficient (have less variance) than the sample mean for various distributions, they typically introduce bias.
> What are some well-known distributions that appear to have outliers? A log-normal distribution maybe?
All of them. Which is why outside of a handful of contexts, the consensus in statistical modeling nowadays seems to be not to worry about outliers unless the values are completely unreasonable or there are a suspicious amount of them.
As to your bootstrapping idea, why not bootstrap the entire distribution, why only the tails? If you only bootstrap the tails but allow draws from the entire empirical distribution then you are changing the underlying distribution.
Doesn't it depend on what is the likely reason for the outliers?
- A world with a different distribution than the one you are trying to fit
- A measurement environment subject to bad contacts or noise spikes or experimental mistakes
- A reporting system with occasional typos
- etc
Seems to me what to do with the outliers should be informed by some understanding of the environment. And in some case, noted aside while waiting to see if there is more data "out there" in the outliers' vicinity.
In some cases, a replacement for the outlier might be "nearby" while in other cases we know nothing about where the replacement should be.
I used to design PID controllers where some of the analog readings would be actually bad. And grossly so.
Like 1002, 998, 1004, 48723, 2104, 1003, 997...
Estimating the deviation from the mean of the last n readings and ignoring ones too far out works well. Also calculate the percentage of bad readings and have a way of displaying it.
Right! In a case like this, some outliers are just useless for calculating the mean - they come from something else - BUT they might still be useful in a SMART style of trying to detect degradation of the system. A second measure.
My take about these things is you really need to understand what you are filtering for and what you are filtering against. If you don't you'll have a bad time.
One of the fascinating things about statistics is that in some ways it's more an art than a science, and the question of "when would I choose to use this over a normal mean or median" is a great example of that.
I use it mainly as a data-exploration method. For example, if the Winsorized mean gives a value that differs a lot from the conventional mean, then I might examine the outliers in a bit more detail with tools like a boxplot or a histogram.
The source of the data matters a lot in what methods make sense. For example, hand-entered numbers might involve transposed digits, or missing signs, or decimal points in the wrong place. Numbers deriving from some electronic measurements might have problems with numbers "pegging out" at some limit. In other cases, those numbers might "wrap around". Data that have been examined at an earlier stage might have numbers changed to something that is obviously wrong, like a temperature of -999.999 or something. The list goes on.
My point is that exploring outliers is often quite productive, and comparing means to Winsorized means can be a very quick way to see if outliers are an issue. This is not so much an issue for interactive work, for which plotting data is usually an early step, but it can come in handy during a preliminary stage of processing large datasets non-interactively. It can also be handy as part of a quality-control pipeline in a data stream.
when you talk about Winsorizing, you're talking about applying some subtle sophistication to statistically sparse data... so then you mention histograms, which also apply subtle sophistication, so you best be talking about histograms with unequal bar widths. Equal bar widths is a bar chart. It doesn't get to be called a histogram (pace wikipedia) unless every bar reflects the same proportion of the population.
I'm a confused. Are you saying that a chart isn't a histogram unless each bar represents the same number of items? I'm not sure I've even seen a chart like that before, I'd love to see an example if you've got one handy.
It's what I was taught in statistics class at MIT. But you're right, I rarely see them, people just call bar charts "histograms". What it does is smooth out outliers. There is always (i mean, within reason) a good sample at the mean, but as you go out from the mean you get more and more variance (it's sort of the definition of variance)
A gaussian bell curve is a "population density function" and this definition of histogram reflects keeping the population density meaningful in terms of height. I will try to look for some. But the wikipedia article on histograms does reference them, that's what they are talking about wrt "varying width" bar charts. It's just that there is a rule to varying the width, "x times y is equal for all the rectangles" (oh yeah, you even get to "eyeball it" to make them. Unless you have a massive number of sample points, if you can choose a width parameter that makes your curve look normal, your curve probably is normal, or at least, you can't say that it's not, no matter what the bar chart looks like.
to put it another way, if you follow this rule for a perfect normal distribution, you'll still get a plot of a perfect normal distribution. But if you have a sparse sample that's a jaggy bell curve, it will move the curve toward a smoother curve, accurate in the sense of "this curve is statistically equivalent to your data, so if it looks "normal", probably your data is normal, and you won't imagine a mode where there isn't one.
The parent is more confused than you are, I think. I have almost never seen an histogram with bins of unequal sizes. The claims were dubious so I checked Wikipedia, which mentions this:
> The bins (intervals) must be adjacent and are often (but not required to be) of equal size.
And this:
> A histogram is used for continuous data, where the bins represent ranges of data, while a bar chart is a plot of categorical variables.
To be fair, there probably are variations in nomenclature across fields. But saying that a histogram with bins of equal width is a bar chart is wrong, at least in the general sense.
If every bar reflects the same proportion of the population then that would mean every bar of the histogram would have exactly the same height, because the y axis on a histogram is density or proportion.
I don't get how you can be so confident about a concept you only vaguely remember from a class you took long ago, and that even after reading the relevant article on wikipedia you conclude that therefore wikipedia must be wrong rather than thinking "hmm, wonder whether I'm misremembering things here..."
There's some interesting discussion in this thread about truncated vs winsorized means. For my own part, this is the first time I've come across either of these terms.
I tend to benefit the most from seeing the entire distribution visually, and that helps me decide if I'm looking for a median, a "normal" mean, a "mean minus some weird outliers", or something different entirely.
Does anybody happen to know of a good visual guide for how different measures of central tendency apply to various distributions? Anything that emphasizes pathological cases is helpful.
55 comments
[ 3.6 ms ] story [ 106 ms ] threadBoth methods give you better metrics for comparing periodically, like day over day or week over week.
Here, we get:
as you see "Windsorized mean" is exaggerated on the positive direction.[1] this is when I remove top and bottom 2 values and calculate the mean of all the remaining values.
If you're monitoring response timings on a server, for example, the median might be very close to 0, and it won't shift unless a majority of the distribution slows down. If you take a winsorised mean, you can trim useless long response times that mess with the mean, but still see if e.g. 1/3 of your responses are suddenly slower than normal.
The windsorized mean reduces the weight of outliers, so it can measure "how many outliers are there?" instead of "how extreme are the outliers?"
However some frames may contain unwanted outliers, for example if a satellite briefly passes overhead it will appear as a very bright streak in only one frame.
By winsorizing, outlying pixel values can be eliminated while still maintaining the same number of samples per pixel as the rest of the stacked image.
I can understand a line of logic that would give rise to something like the Winsorized mean -- after you look at your data, you see some obvious outliers. It feels dirty to just drop those values (which would lead to the truncated mean) because the information from an implausible value is more likely to be near the extreme than it is to be near the center mass.
What to do with those extreme values?
Here's something I now want to experiment with -- bootstrapping the extreme values. Take note of the original empirical distribution. Then, create a new distribution by removing the top and bottom X% of the observations and replacing them with values drawn i.i.d. from the original empirical distribution. This could lead to some values being replaced with the outliers that we originally wanted to drop. After we do this, record the mean. Then create new sample distributions until we have a distribution of new means. What I am curious about is how the shape if this distribution of means will be impacted depending on that X% value selected at the beginning.
What are some well-known distributions that appear to have outliers? A log-normal distribution maybe?
It does feel like it's a very early 20th century technique. Nowadays we have so many tools which would be less feasible for calculators (the people) and more feasible for software.
https://en.m.wikipedia.org/wiki/M-estimator
Recently, we started using the arcsinh instead of the log as well because the function has nice properties[1]
1. https://worthwhile.typepad.com/worthwhile_canadian_initi/201...
It makes nonlinear relationships linear. Makes the model less sensitive, too. For instance if the data spans several OoM, adding or removing one datapoint in one of those orders can generate a lot of skew before the log-linearization.
It's easy to cast the log back to the original distribution by taking the exponent afterwards.
[1] https://en.m.wikipedia.org/wiki/Generalized_linear_model
Read the Wikipedia article for a more formal explanation.
[1] https://en.m.wikipedia.org/wiki/Variance-stabilizing_transfo...
I don't really like the Winsorized mean but not for the reason you list. I think the main issue is that you are assuming exactly the top and bottom 10% are outliers instead of looking at the actual data distribution to see what the outliers are then using a similar replacement technique on only the outliers.
https://projecteuclid.org/journals/annals-of-statistics/volu...
How does it do on normal and other common distributions?
All of them. Which is why outside of a handful of contexts, the consensus in statistical modeling nowadays seems to be not to worry about outliers unless the values are completely unreasonable or there are a suspicious amount of them.
As to your bootstrapping idea, why not bootstrap the entire distribution, why only the tails? If you only bootstrap the tails but allow draws from the entire empirical distribution then you are changing the underlying distribution.
- A world with a different distribution than the one you are trying to fit
- A measurement environment subject to bad contacts or noise spikes or experimental mistakes
- A reporting system with occasional typos
- etc
Seems to me what to do with the outliers should be informed by some understanding of the environment. And in some case, noted aside while waiting to see if there is more data "out there" in the outliers' vicinity.
In some cases, a replacement for the outlier might be "nearby" while in other cases we know nothing about where the replacement should be.
Like 1002, 998, 1004, 48723, 2104, 1003, 997...
Estimating the deviation from the mean of the last n readings and ignoring ones too far out works well. Also calculate the percentage of bad readings and have a way of displaying it.
Rust's `cargo bench` "winzorizes" the benchmark samples before computing summary statistics (incl. the mean).
https://github.com/rust-lang/rust/blob/master/library/test/s...
The source of the data matters a lot in what methods make sense. For example, hand-entered numbers might involve transposed digits, or missing signs, or decimal points in the wrong place. Numbers deriving from some electronic measurements might have problems with numbers "pegging out" at some limit. In other cases, those numbers might "wrap around". Data that have been examined at an earlier stage might have numbers changed to something that is obviously wrong, like a temperature of -999.999 or something. The list goes on.
My point is that exploring outliers is often quite productive, and comparing means to Winsorized means can be a very quick way to see if outliers are an issue. This is not so much an issue for interactive work, for which plotting data is usually an early step, but it can come in handy during a preliminary stage of processing large datasets non-interactively. It can also be handy as part of a quality-control pipeline in a data stream.
A gaussian bell curve is a "population density function" and this definition of histogram reflects keeping the population density meaningful in terms of height. I will try to look for some. But the wikipedia article on histograms does reference them, that's what they are talking about wrt "varying width" bar charts. It's just that there is a rule to varying the width, "x times y is equal for all the rectangles" (oh yeah, you even get to "eyeball it" to make them. Unless you have a massive number of sample points, if you can choose a width parameter that makes your curve look normal, your curve probably is normal, or at least, you can't say that it's not, no matter what the bar chart looks like.
to put it another way, if you follow this rule for a perfect normal distribution, you'll still get a plot of a perfect normal distribution. But if you have a sparse sample that's a jaggy bell curve, it will move the curve toward a smoother curve, accurate in the sense of "this curve is statistically equivalent to your data, so if it looks "normal", probably your data is normal, and you won't imagine a mode where there isn't one.
> The bins (intervals) must be adjacent and are often (but not required to be) of equal size.
And this:
> A histogram is used for continuous data, where the bins represent ranges of data, while a bar chart is a plot of categorical variables.
To be fair, there probably are variations in nomenclature across fields. But saying that a histogram with bins of equal width is a bar chart is wrong, at least in the general sense.
I don't get how you can be so confident about a concept you only vaguely remember from a class you took long ago, and that even after reading the relevant article on wikipedia you conclude that therefore wikipedia must be wrong rather than thinking "hmm, wonder whether I'm misremembering things here..."
I tend to benefit the most from seeing the entire distribution visually, and that helps me decide if I'm looking for a median, a "normal" mean, a "mean minus some weird outliers", or something different entirely.
Does anybody happen to know of a good visual guide for how different measures of central tendency apply to various distributions? Anything that emphasizes pathological cases is helpful.