Averages are the mind-killer. Don't show me the average, show me the distribution. Any time that you present an average over a data set that isn't approximately a normal distribution, you're committing statistical malpractice.
> Don't show me the average, show me the distribution.
Put another way, don't give me a measure of center (mean) without a corresponding measure of spread (variance/standard deviation).
Providing a measure of center for something that is approximately normal doesn't obviate the need for a measure of spread--you could have two distributions with the same mean but wildly different spread of results due to variance.
Figuring out which of two things is bigger is hard. The article lightly touches on the two main reasons:
- Sometimes the definition is complicated. E.g., I'd accept a computer 10x slower at most tasks if a particular small subset were sped up even 2x. No symmetric mean (geometric, arithmetic, harmonic, or otherwise) will appropriately capture that.
- Nobody agrees on the definition. Even for very regular workloads, parsing protobufs is very different from old-school ML is very different from reversing the endianness of an unaligned data fragment. Venturing into even more diverse workloads muddies the waters further.
The article basically gives up and says that the geometric mean is the least bad option because it has some nice properties (ones I don't think anyone cares about for this purpose...) and because it's what everyone expects.
That latter argument is at least based in something sound, but I think the real point here is that attempting to come up with a single definitive "speed" is foolhardy in the first place. The only point is to compare two architectures, but without a particular context in mind no such linear ordering exists, and that's before we get to the difficulties in defining it appropriately even when given a context.
Isn't this why benchmarking is used? You select some common payload and then use that as a realworld simulation of how your architecture performs with it.
Of course this comes with its own can of worms, like overfitting and such, but I could imagine a benchmarking solution that gives you a more granular look at which specific tasks an architecture performs well.
For the specific purpose of what the article presents, I feel something simpler like 10%/50%/90% times can be better (two systems that have times as 0.1/1/10 or 0.9/1/1.1 will have "the same average" but where one might be adequate the other one might not)
Metrics can be useful when they are thoughtfully chosen for the context in which they are used. The more abstract or synthetic a benchmark is, the less context there is to inform such a choice. Said another way, metrics characterizing synthetic benchmarks inherit/expose one of the key problems of synthetic benchmarks - a lack of context.
In practical settings, the way to characterize performance will be more clear from context, often giving one (-ish) clear metric for each dimension you care about. For example, if an endpoint has a hard timeout of 100 ms, then it's pretty interesting to look at the percentage of real world requests that are faster than 100 ms. If the same solution is also used in a setting where throughput is more important than latency, then an additional metric is probably needed for that use case. Multiple metrics are needed for multiple use cases to capture trade-offs being made between those use cases.
This kind of debate isn't that unusual, is it? To figure out what calculation to use, one has to made tradeoffs about desirable properties such as resilience to outliers, being bias-free, and/or having the best predictive performance in least-squared error. Bringing some grizzled statisticians into the conversation will probably clear a lot up. They are familiar with the pros and cons of various measures of central tendency.
In the era of multivariate models, machine learning, and AI, some of the classic wisdom from good-old linear modeling gets overlooked.
Ironically, AI research canonically uses arithmetic mean for averaging basically everything. Whatever suite of experiments you are running, somebody will sooner or later ask you to average them.
I disagree with the article's claim that the geometric mean lacks physical meaning.
Say you have two benchmarks with different units, frames/second and instructions/second. You can't take the arithmetic mean of these unless you divide by some baseline first (or a conversion factor).
But the geometric mean has well-defined units of (frames * instructions)^.5/second. And the reason you can divide by another geometric mean is because these units are always the same.
Having coherent units isn't exactly the same as "physical meaning", but it's a prerequisite at the least.
Geometric average is just estimating the mean of logarithmic units. Units of measurement are arbitrary, and justification for using logarithmic units for statistical purposes can be easily made with distributional considerations. E.g. durations are bounded to be positive, so there will be at least some skew in the distribution (although this can be negligible in practice).
Yep, I always end up using a bunch of graphs because picking one number feels off every time. Honestly, stats can get slippery real fast if you don't watch it.
Given only speedups any formula to combine them is meaningless, because the real question involves a concrete workload: would the performance of what I'm going to do with my computer improve enough to justify the cost of improving it?
At the very least, weights for each benchmark task are needed; complications of the cost model (beyond a binary old system/new system comparison) are also likely.
Measuring performance improvement by speedup of throughput is also often naive, since there are other dimensions (like power and latency) and complex requirements.
Physical interpretation of geometric mean… you have a collection of multiplicative speedups, and some of them apply to any given workload. I think the geometric mean falls out of the resulting statistical distribution in a natural way but I’m sleep-deprived at the moment and can’t write out the math. Like, in this distribution geomean is the median speedup.
Think of the case with two values, 2.00 and 2.42. Decompose it into three speedups… 2x, 1.1x, and 1.1x. The speedup 2x happens with probability 1. Each of the 1.1x speedups happen with probability 0.5. The geomean is 1 2x speedup and 1 1.1x speedup, giving 2.2x.
There are many such decompositions, that one is not unique. Exercise for the reader to show which conditions give you the geometric mean and explain why that is reasonable… I’m terribly sleep-deprived at the moment and this is where I stop.
There is of course no correct choice of mean here, just a bunch of different choices with different interpretations justifying them.
An “average” (whether a mean/ median etc) is a very lossy compression algorithm.
You’re attempting to describe a whole series of numbers with just one (or two) numbers.
Trying to come up with a good general purpose way to reduce/compress/aggregate data via a lossy algorithm is intractable.
While that all might sound obvious, it can be very hard to internalise this.
(And that’s before getting into the motivated reasoning that biased actors [aka normal people] will use to preference one lossy algorithm over another)
arith-mean = E[x] , the first moment of x
geo-mean = exp(E[log x]) , so log geo-mean = first moment of log x
They are both equivalent in amount of information preserved, but arithmetic preserves additive structure whereas geo preserves multiplicative structure
I didn't want to imply that there is a loss of information. Yes, it's one level up in the hyperoperation chain, I suppose, but what I meant is that it's not a typical way of doing statistics, especially the higher moments.
Sometimes, you can't find the average because it's undefined: it can happen with a Cauchy and a few other statistical distributions: the wikipedia page has a nice plot of how the first 2 moments don't converge https://en.wikipedia.org/wiki/Cauchy_distribution#History
When in doubt, don't use the mean: prefer more robust estimates, as even with degenerate statistical distributions, there are still some "good numbers to report" like the mode or the median.
And if you don't know statistics, just use a plot!
Indeed, the best averaging method depends on the underlying probability distribution from which data is drawn. Arithmetic is best for normal whereas geometric is better suited for lognormal distribution, and as the above comment suggests average is meaningless for most power law distributions where exponent is less than 2.
However, When all else fails, define your own Von Neumann entropy. Figure out how often you compile GCC, FFT, or video compression, then compute probabilities (ratios) and multiply by logarithm of speedups for each use case. Sum them up and report it as machine/architecture entropy and you'll win every argument about it.
I agree with your point, but it is funny to think about true consumer workloads: I... mostly JIT and run Javascript, layout algorithms, and whatever compositing cannot be offloaded to the GPU.
Averages are the mind-killer. Don't show me the average, show me the distribution. Any time that you present an average over a data set that isn't approximately a normal distribution, you're committing statistical malpractice.
> Don't show me the average, show me the distribution.
Put another way, don't give me a measure of center (mean) without a corresponding measure of spread (variance/standard deviation).
Providing a measure of center for something that is approximately normal doesn't obviate the need for a measure of spread--you could have two distributions with the same mean but wildly different spread of results due to variance.
I think the GP wants to see the actual distribution graph.
Figuring out which of two things is bigger is hard. The article lightly touches on the two main reasons:
- Sometimes the definition is complicated. E.g., I'd accept a computer 10x slower at most tasks if a particular small subset were sped up even 2x. No symmetric mean (geometric, arithmetic, harmonic, or otherwise) will appropriately capture that.
- Nobody agrees on the definition. Even for very regular workloads, parsing protobufs is very different from old-school ML is very different from reversing the endianness of an unaligned data fragment. Venturing into even more diverse workloads muddies the waters further.
The article basically gives up and says that the geometric mean is the least bad option because it has some nice properties (ones I don't think anyone cares about for this purpose...) and because it's what everyone expects.
That latter argument is at least based in something sound, but I think the real point here is that attempting to come up with a single definitive "speed" is foolhardy in the first place. The only point is to compare two architectures, but without a particular context in mind no such linear ordering exists, and that's before we get to the difficulties in defining it appropriately even when given a context.
Isn't this why benchmarking is used? You select some common payload and then use that as a realworld simulation of how your architecture performs with it.
Of course this comes with its own can of worms, like overfitting and such, but I could imagine a benchmarking solution that gives you a more granular look at which specific tasks an architecture performs well.
Or even better just measure your actual workload.
Yeah
For the specific purpose of what the article presents, I feel something simpler like 10%/50%/90% times can be better (two systems that have times as 0.1/1/10 or 0.9/1/1.1 will have "the same average" but where one might be adequate the other one might not)
Metrics can be useful when they are thoughtfully chosen for the context in which they are used. The more abstract or synthetic a benchmark is, the less context there is to inform such a choice. Said another way, metrics characterizing synthetic benchmarks inherit/expose one of the key problems of synthetic benchmarks - a lack of context.
In practical settings, the way to characterize performance will be more clear from context, often giving one (-ish) clear metric for each dimension you care about. For example, if an endpoint has a hard timeout of 100 ms, then it's pretty interesting to look at the percentage of real world requests that are faster than 100 ms. If the same solution is also used in a setting where throughput is more important than latency, then an additional metric is probably needed for that use case. Multiple metrics are needed for multiple use cases to capture trade-offs being made between those use cases.
This kind of debate isn't that unusual, is it? To figure out what calculation to use, one has to made tradeoffs about desirable properties such as resilience to outliers, being bias-free, and/or having the best predictive performance in least-squared error. Bringing some grizzled statisticians into the conversation will probably clear a lot up. They are familiar with the pros and cons of various measures of central tendency.
In the era of multivariate models, machine learning, and AI, some of the classic wisdom from good-old linear modeling gets overlooked.
Ironically, AI research canonically uses arithmetic mean for averaging basically everything. Whatever suite of experiments you are running, somebody will sooner or later ask you to average them.
I disagree with the article's claim that the geometric mean lacks physical meaning.
Say you have two benchmarks with different units, frames/second and instructions/second. You can't take the arithmetic mean of these unless you divide by some baseline first (or a conversion factor).
But the geometric mean has well-defined units of (frames * instructions)^.5/second. And the reason you can divide by another geometric mean is because these units are always the same.
Having coherent units isn't exactly the same as "physical meaning", but it's a prerequisite at the least.
Geometric average is just estimating the mean of logarithmic units. Units of measurement are arbitrary, and justification for using logarithmic units for statistical purposes can be easily made with distributional considerations. E.g. durations are bounded to be positive, so there will be at least some skew in the distribution (although this can be negligible in practice).
Yep, I always end up using a bunch of graphs because picking one number feels off every time. Honestly, stats can get slippery real fast if you don't watch it.
Given only speedups any formula to combine them is meaningless, because the real question involves a concrete workload: would the performance of what I'm going to do with my computer improve enough to justify the cost of improving it?
At the very least, weights for each benchmark task are needed; complications of the cost model (beyond a binary old system/new system comparison) are also likely.
Measuring performance improvement by speedup of throughput is also often naive, since there are other dimensions (like power and latency) and complex requirements.
Physical interpretation of geometric mean… you have a collection of multiplicative speedups, and some of them apply to any given workload. I think the geometric mean falls out of the resulting statistical distribution in a natural way but I’m sleep-deprived at the moment and can’t write out the math. Like, in this distribution geomean is the median speedup.
Think of the case with two values, 2.00 and 2.42. Decompose it into three speedups… 2x, 1.1x, and 1.1x. The speedup 2x happens with probability 1. Each of the 1.1x speedups happen with probability 0.5. The geomean is 1 2x speedup and 1 1.1x speedup, giving 2.2x.
There are many such decompositions, that one is not unique. Exercise for the reader to show which conditions give you the geometric mean and explain why that is reasonable… I’m terribly sleep-deprived at the moment and this is where I stop.
There is of course no correct choice of mean here, just a bunch of different choices with different interpretations justifying them.
Your favorite stock doubled in 6 months. What was it growing on average in a day?
An “average” (whether a mean/ median etc) is a very lossy compression algorithm.
You’re attempting to describe a whole series of numbers with just one (or two) numbers.
Trying to come up with a good general purpose way to reduce/compress/aggregate data via a lossy algorithm is intractable.
While that all might sound obvious, it can be very hard to internalise this.
(And that’s before getting into the motivated reasoning that biased actors [aka normal people] will use to preference one lossy algorithm over another)
You can use an increasing number of statistical moments.
https://en.wikipedia.org/wiki/Moment_(mathematics)
The arithmetic mean is one of them, which would be an argument in favor of it.
I don't think it is an argument in favour of it.
arith-mean = E[x] , the first moment of x geo-mean = exp(E[log x]) , so log geo-mean = first moment of log x
They are both equivalent in amount of information preserved, but arithmetic preserves additive structure whereas geo preserves multiplicative structure
I didn't want to imply that there is a loss of information. Yes, it's one level up in the hyperoperation chain, I suppose, but what I meant is that it's not a typical way of doing statistics, especially the higher moments.
arithmetic mean of percentages = terrorism
I'm using median because it is more robust against outliers.
Sometimes, you can't find the average because it's undefined: it can happen with a Cauchy and a few other statistical distributions: the wikipedia page has a nice plot of how the first 2 moments don't converge https://en.wikipedia.org/wiki/Cauchy_distribution#History
When in doubt, don't use the mean: prefer more robust estimates, as even with degenerate statistical distributions, there are still some "good numbers to report" like the mode or the median.
And if you don't know statistics, just use a plot!
Indeed, the best averaging method depends on the underlying probability distribution from which data is drawn. Arithmetic is best for normal whereas geometric is better suited for lognormal distribution, and as the above comment suggests average is meaningless for most power law distributions where exponent is less than 2.
However, When all else fails, define your own Von Neumann entropy. Figure out how often you compile GCC, FFT, or video compression, then compute probabilities (ratios) and multiply by logarithm of speedups for each use case. Sum them up and report it as machine/architecture entropy and you'll win every argument about it.
I agree with your point, but it is funny to think about true consumer workloads: I... mostly JIT and run Javascript, layout algorithms, and whatever compositing cannot be offloaded to the GPU.
it seems pretty similar to voting problems in political elections. No matter the algorithm you choose, someone will complain.