Benford's law does not apply to some of those things! In particular not to expenses.
Benford's law only applies to things that experience exponential growth, things that obey a power law distribution, and things that are totals of random processes.
How could expenses not have a power law component to their distribution? Sometimes you're buying an expensive thing, and sometimes a cheap thing, and you don't buy something between $999 and $200 dollars nine times as often as you buy something between $199 and $100.
Yes, you will certainly get artifacts at the top of what you can expense. You will probably also get distortions at the bottom where you'll have some threshold of "not worth expensing." There will almost certainly be distortion around the boundaries, but as long as you have a big enough range you should be able to see a Benford's law effect in the middle.
welcometo: http://www.netetrader.com
The website wholesale for many kinds of fashion shoes, like the nike,jordan,prada,, also including the jeans,shirts,bags,hat and the decorations. All the products are free shipping, and the the price is competitive, and also can accept the paypal payment.,after the payment, can ship within short time.
free shippingcompetitive priceany size availableaccept the paypal
===== http://www.netetrader.com =====
Benford's law would not have stopped the groupthink, willful ignorance and corruption which actually drove the Greek disaster. It's not like Greece's problem with tax collection was a well-kept secret, it's just that nobody wanted to face the facts.
I seem to remember at the time there was widespread reported skepticism about a number of countries' data. It was just accepted that the central European elite wanted to expand to sell their wares to these dodgy countries and prevent conflict. It shows how dependent the lender is to the lendee and how both sides don't face up to the reality.
Also, Greece’s original accession to the EU was motivated in part by geopolitical considerations; Greece applied for membership shortly after a democratic government replaced seven years of harsh military rule.
The issue with using Benford's Law with analysis is that as soon as people realize that you're using it, data can easily be manipulated to make it compatible.
The previous article by Tim Harford mentioned that Madoff's financial data was Benford compatible.
So I doubt that using it would have actually prevented the Greek or any other disaster.
The article says the opposite, though it's hard to put a lot of credibility on something starting with "according to statisticians"...
According to statisticians, it is almost impossible to manipulate data in a way that a certain outcome is guaranteed and Benford’s Law is met at the same time. Hence, tax authorities in several countries are using Benford’s Law as a default testing device.
I re-read the article and you're right. The article seems to imply Madoff as an exception rather than the common case, but I still feel that if data is being manipulated at such a vast scale where an entire countries economic activity is misrepresented, you will be able to find people as smart as Madoff to 'fit' the data as per requirements.
You certainly can find people smart enough to make sure that their fraudulent statistics will meet Benford's Law (or any other data model you wish). The article points out that Greece didn't find people this smart.
Not meeting Benford's Law is a good indicator that there has been some dodgy dealing. However, meeting Benford's Law is not a good indicator that everything is above board.
Not sure if you skimmed the linked article and missed this or not:
statistics from the Czech Republic, Sweden and the UK meet the Benford distribution particularly close and hence appear unsuspicious. All those countries have no interest in joining the Euro and therefore do not have to meet the convergence criteria. On the other hand, statistics from Latvia, Belgium and Romania (the latter notorious for its corruption) appear very suspicious, as well.
"They looked at 130 different values per country and year", they don't say how many years were studied but I assume the greece book fixing lasted only a few years so we are looking at ~500 samples. I honestly don't know if that's enough or not, on wikipedia they show examples of the distribution on less than 100 samples so I guess I'm on the wrong.
Although in this instance I agree, I think the bigger point is that you shouldn't call anything "arcane" on HN. Because it'll be well known to someone and that someone—who is probably someone who could contribute a lot to the HN discussion—will think poorly of you.
Come on, this is not even a real law, and the data are really shaky to justify the title of this post. 5% significance threshold means that Luxembourg or Austria satisfy the criteroa. So what? The EU is a large political organization, not a NYSE index. The real trouble for greece is not the fake accession data (Italy had similarly manipulated data), but the squandering of public money that followed.
I get it, geeks love to play with empirical laws, but this paper is at best interesting trivia. I could go on analyzing what could really prevent the disaster forever ...
It's nice that they made sure the data set contained the key data to their collapse. The pensions of retirees. Would the results have been the same if it weren't included? I don't have access to the paper but I'd be curious what their justification for their choice was.
Among other things, they looked at the total level of debt, the cash reserves of the government and the pensions of retired civil servants.
> According to statisticians, it is almost impossible to manipulate data in a way that a certain outcome is guaranteed and Benford’s Law is met at the same time. Hence, tax authorities in several countries are using Benford’s Law as a default testing device.
I'd be very interested in evidence to support this. This doesn't sound correct. "Almost impossible?"
Nice geek porn, but the title and the premise is basically nonsense.
EU politicians knew for many years that Greece was scamming things. Maybe not so much when the Euro started, but well before it all blew up. All kinds of circumstances, however (such as the need for Greece's vote on some other hot issue of the moment) made sure that no one had the will to really do something about it.
Statistics is nice and dandy, but it doesn't help a dime in fixing the broken mess that is EU politics.
ps. Fun trivia: if the EU applied for EU membership, it wouldn't get in because it's too undemocratic.
32 comments
[ 2.5 ms ] story [ 64.9 ms ] thread1. Spotting odd things in MPs' expenses: http://blog.jgc.org/2009/06/its-probably-worth-testing-mps.h...
2. Spotting odd things in BBC executives' expenses: http://blog.jgc.org/2009/06/running-numbers-on-bbc-executive...
3. The Iranian election: http://blog.jgc.org/2009/06/benfords-law-and-iranian-electio...
4. New Age mumbo jumbo: http://www.jgc.org/blog/2008/02/any-sufficiently-simple-expl...
Benford's law only applies to things that experience exponential growth, things that obey a power law distribution, and things that are totals of random processes.
Expenses are none of those.
This is pretty obvious: the probability of the number 1 will be greater than the number 2, and so on up to 9.
The probability of the range 10-19 will be greater than the range 20-29, and so on.
The probability of the range 100-199 will be greater than the range 200-299, and so on.
And so on... The probability of the ranges starting with "1" will always be higher.
Also, I have no idea why anyone would think this is "arcane". It's very simple and obvious and a good example of where intuition can go wrong.
jordan shoes $32nike shox $32Christan Audigier bikini $23 Ed Hardy Bikini $23Smful short_t-shirt_woman $15ed hardy short_tank_woman $16Sandal $32christian loubo utin $80 Sunglass $15 COACH_Necklace $27handbag $33AF tank woman $17puma slipper woman $30
===== http://www.netetrader.com =====
===== http://www.netetrader.com =====
===== http://www.netetrader.com =====
===== http://www.netetrader.com =====
===== http://www.netetrader.com =====
http://news.ycombinator.com/item?id=3008848
The similarly mentioned Ben Goldacre article was also submitted, but that got no discussion:
http://news.ycombinator.com/item?id=3007964
A detailed discussion by Terry Tao of Benford's Law (and more) can be found here:
http://terrytao.wordpress.com/2009/07/03/benfords-law-zipfs-...
The previous article by Tim Harford mentioned that Madoff's financial data was Benford compatible. So I doubt that using it would have actually prevented the Greek or any other disaster.
According to statisticians, it is almost impossible to manipulate data in a way that a certain outcome is guaranteed and Benford’s Law is met at the same time. Hence, tax authorities in several countries are using Benford’s Law as a default testing device.
Not meeting Benford's Law is a good indicator that there has been some dodgy dealing. However, meeting Benford's Law is not a good indicator that everything is above board.
The reason Greece didn't find people smart enough to satisfy Benford's is that Benford's wasn't being used in the audit.
If it was common knowledge that audits utilize this Law, you can bet that Greece's data would have been compatible.
statistics from the Czech Republic, Sweden and the UK meet the Benford distribution particularly close and hence appear unsuspicious. All those countries have no interest in joining the Euro and therefore do not have to meet the convergence criteria. On the other hand, statistics from Latvia, Belgium and Romania (the latter notorious for its corruption) appear very suspicious, as well.
And from Tim Harford's article in the FT: http://www.ft.com/cms/s/2/171aaa36-d8f1-11e0-aff1-00144feabd...
Romania, Latvia and Belgium also have abnormally distributed data, while Portugal, Italy and Spain have a clean bill of health.
I get it, geeks love to play with empirical laws, but this paper is at best interesting trivia. I could go on analyzing what could really prevent the disaster forever ...
Among other things, they looked at the total level of debt, the cash reserves of the government and the pensions of retired civil servants.
http://www.radiolab.org/2009/nov/30/from-benford-to-erdos/
> According to statisticians, it is almost impossible to manipulate data in a way that a certain outcome is guaranteed and Benford’s Law is met at the same time. Hence, tax authorities in several countries are using Benford’s Law as a default testing device.
I'd be very interested in evidence to support this. This doesn't sound correct. "Almost impossible?"
EU politicians knew for many years that Greece was scamming things. Maybe not so much when the Euro started, but well before it all blew up. All kinds of circumstances, however (such as the need for Greece's vote on some other hot issue of the moment) made sure that no one had the will to really do something about it.
Statistics is nice and dandy, but it doesn't help a dime in fixing the broken mess that is EU politics.
ps. Fun trivia: if the EU applied for EU membership, it wouldn't get in because it's too undemocratic.