Why can't you get the data used to create the Consumer Price Index (CPI)?
The Consumer Price Index (CPI) measures inflation and, in the words of the US Bureau of Labor Statistics, "directly affects hundreds of millions of Americans".
I would have guessed the data used to construct the CPI would be open to everyone.
But no, it appears to be unavailable and it is instead a black box due to rules of confidentiality. What is strange is, it appears much of the data is publicly available data. For example, 8% comes from e-commerce sites, the USPS public website, etc. If you browse through the categories, you can also see that almost all the data could be collected from public sources - APIs, websites, etc.
So, does anyone know why this data cannot be obtained?
Reference: https://www.bls.gov/opub/hom/cpi/pdf/cpi.pdf
52 comments
[ 5.3 ms ] story [ 116 ms ] threadhttp://www.thebillionpricesproject.com/
Spoiler: Billion Prices Project has mostly tracked CPI within 1-2%
http://www.thebillionpricesproject.com/datasets/
Also, I think the inclusion of things like tuition and medical care kind of adulterates the data. I have read that people go to the hospital for non-emergency reasons less often than before the pandemic, and also that several schools slashed tuition due to the pandemic. That offsets price increases in other areas.
You can criticize the Fed for doing reckless policies that fail to increase inflation and only drive a bubble. Why would you even need an unproven conspiracy theory like CPI when the problems with the Fed are undeniable? You can even complain to the fed that inflation is too low and they should either calm down or do something else since the Fed is clearly not capable of driving inflation. Considering the mission statement of the Fed that's a huge blow by itself.
1. I'm not arguing that the fed is artificially inflating inflation. I'm saying that if they want to suppress the appearance of inflation in the short term so that they can keep rates low and prevent any panic about the dollar until they can react to it, they have some ability to do so because of the opaqueness of the data.
2. Bias in a signal doesn't have to be persistent to be problematic. The average reading does not have to be biased in the long term for there to be a problem with the spot reading. That's a major reason why more granular data is always better.
3. I'm not stating any conspiracy theory. The Fed is driven by data. Biases in data affect data-driven decisions. CPI includes various inputs, not all of which are experiencing the same demand. CPI is an agglomerate reference.
But something seems to have changed for the better when they changed to average inflation targeting recently.
Squinting at the graph, for 2008=100, in 2016 CPI was ~108 and BPP was ~109. (Couldn't find a newer graph with a quick look.)
The point is the BPP is proof there is no nefarious conspiracy to manipulate CPI to hide “true” inflation. Amongst the gold-bug hyperinflationist crowd there is a commonly held belief that inflation is much higher than CPI or that CPI is manipulated to hide inflation. The fact that a simple model scraping a few online retailers results in nearly the same annual percentage change demonstrates that this conspiracy theory is a myth.
The current inflationary model is mostly driven by salaries. Thus in the 70s unions drove up inflation due to oil shocks.
The concern for me is that models do not include asset inflation for goods (housing) that most people would like to own and where there is little productivity improvement. Using an equivalent rent model ignores this preference.
Rent is the price of housing consumption.
No, rent and owner's equivalent rent make up a quarter of the inflation basket - just in exact proportion to their share of aggregate consumption expenditure in the US.
The San Francisco tech bro who spends 70% of his take-home on rent is not representative of the US population at large...
(Only the proportion of property taxes that falls on the buildings etc causes deadweight losses. The proportion that falls on the value of the land is free of deadweight losses.
See Land Value Taxes.)
Here’s a graph from Zillow which does not show a huge increase in last 30 years: http://www.doctorhousingbubble.com/wp-content/uploads/2016/0...
According to the American Housing Survey, housing as a percentage of income only increased from 19 to 22% from 1985 to 2005.
Overall, wages have exceeded or kept pace with inflation since 1979: https://www.politifact.com/factchecks/2019/jan/30/kamala-har...
From 1979 - 2018 incomes for 90% of the population is up 23.9%. 90-95% up 45.7%, 95-99% up 71.3%. Top 1% is up 157.8%. https://www.epi.org/blog/top-1-0-of-earners-see-wages-up-157...
So chop off the top 10% and numbers are going to look very different. From housing to CPI it’s tricky to adjust for such things to understand how most of the population is doing.
Because of that the CPI value is significant lower than actual inflation
If you are interested in house prices, look for the Case/Shiller phouse price index, if you are interested in health care costs, look for the PPI components of drugs, hospital services, medical devices etc.
To counter your implied assertion - no, CPI is not being manipulated, but house price/CPI ratio has risen because mortgage rates are low, and medical care costs as a whole have risen mainly because of (a) proportionately more fat and old people, (b) agency problems in the health insurance model (overprescription/overtreatment out of defensiveness even if it is not cost effective).
Generally the government can’t Show high inflation because doing so would have all sorts of second and third order consequences like increased SSI payout and higher government salaries as well as affect contracts and treasury bond yields.
The entire market would crash if you got a 3-5% hot inflation number. That basically can’t be allowed
I think they're trying to prevent people from calculating the real CPI. there's alot at stake: the bond market, social security, federal interest payments, perception of real wages, real productivity per capita number, etc.
https://www.bls.gov/cpi/quality-adjustment/new-vehicles.pdf
Certainly debatable since it is impossible to actually buy a new car that doesn't have a bunch of those features now.
And it becomes even more preposterous when you look 50+ years back. CPI, says x3 increase in cost of car, but it's actually about x10. So, how does a car return +233% of it's value to a user when MPG hasn't even doubled over that time.
i mean sure, monetary rewards are not the only advancements but they should the primary basis for the inflation discount model. those auto-dimming mirrors and countless other features just don't mean very much.
Fundamentally, though, the problem is that CPI does not track changes in the perceived minimal acceptable quality of the various baskets of goods. If the house you "need" to fulfill a specific lifestyle is twice as big and twice as expensive, this has zero impact on CPI. Same with healthcare quality - if your insulin works 10% better and costs 10% more, the CPI-centric view says that people are simply using more real dollars to buy a higher quality of life. Education has a similar problem, where job requirements shifts from high-school diplomas to undergrad degrees simply do not show up in any way in the CPI; we're just choosing to be better educated to end up working the same jobs.
I think this all boils down to the CPI measuring things as if all goods were absolute goods, while nominal GDP per capita would be the appropriate measure if all goods were positional. I don't know what fraction of spending is on each, though, but you can reasonably put the "real" inflation number somewhere between the two.
The correct metric would be life expectancy, as it sidesteps the zero priors problem. Using that, the healthcare you are talking about looks a lot more like the diminishing returns you get on higher priced modern cars. Before anyone says anything about "quality of life", I'd recommend looking at the way assisted living operations are run. Anyway, we really started to lose steam after dentistry... dunno what the analog would be for cars, power steering?
> If the house you "need" to fulfill a specific lifestyle...
This is another thing I've been dealing with recently. I've delayed entry into the housing market way past what I'd have liked because the pricing makes no sense to me, and I've found that usually if something doesn't make sense to me - it is because it is misrepresented. After watching the market for a few years in two very different parts of the country I noticed something interesting: sub-million dollar homes were quickly getting more expensive, and multi-million dollar mansions were relatively static. I wonder how much of that has to do with home-loan accessibility, similar to the cost of educating students magically increasing in lock step with student loan program scope and availability.
It's because the price of the average house is due to the cost of urban/suburban land - which is scarce and therefore tracks affordability (i.e. rises steeply with median incomes, population growth and especially falling mortgage rates).
Meanwhile the price of multi-million dream homes are (a) often dominated by building costs which rise slower than incomes (b) don't rise due to falling mortgage rates as the NFL player/movie star/exiting startup founder pays in cash rather than over a 30yr mortgage term.
> and I've found that usually if something doesn't make sense to me - it is because it is misrepresented
That is a statement full of hubris. You might find that if something doesn't make sense, you might just not be in possession of all the relevant information. (In this case, the trajectory of mortgage rates and their effect on affordability of homes, demographics and population growth in a metro area, income growth and wealth distribution.)
That maybe true, but it would be unrelated to my observations.
> ...rises steeply with median incomes...
Whoops! For that to be true there would have to be a non-linear relation that would itself be far more interesting than the housing prices issue.
> ...often dominated by building costs which rise slower than incomes...
Not a lot of new mansion construction going on. Oh, and those construction costs are pretty ridiculous right now due to a jump in lumber and appliance costs - income would have had to nearly double to match construction's rate of increase. I collected bids a few months ago for a grading and paving, jobs dominated by labor costs. The quotes were laughably overpriced.
> ...as the NFL player/movie star/exiting startup founder pays in cash...
lol, your conception of today's millionaire is way off - it is a distinction that has never been more easily attained.
> That is a statement full of hubris.
No, that is - and presumptuous :) Also, that final sentence agrees with the mortgage issue I mentioned, gets the direction of migration wrong, and finally overlooks the steadily growing income disparity - which is somehow resulting in mansions doing poorly on the real estate market. So you did a pretty good job of unintentionally proving my point with regard to the situation being misrepresented.
https://www.bls.gov/cpi/questions-and-answers.htm
https://tradingeconomics.com/united-states/inflation-cpi
> Multiple confidentiality issues arise in the production of the CPI. One set of issues arises out of the need to prevent unauthorized access of data that are embargoed, or yet to be released to the public. because the CPI data can affect financial markets, it is essential to ensure that no one without authorization has access to the data before release. BLS personnel who do see the data ahead of time are restricted by law from engaging in certain financial transactions during the period where they have seen data that are not public. Pre-release data are encrypted and always kept on secure servers, with any hard copies locked in secured areas.
I know somebody who got a CPI survey last year -- maybe I can ask her.
https://www.chicagobooth.edu/research/kilts/datasets/nielsen...
Good luck, this type of data is really really guarded as it is ridiculously valuable.
I hear you. I tried recreating the CPI for India and couldn't get the data needed. (see https://arun.tanksali.com/topic/rpi/). I eventually thought I can come with a Personal Price Index which reflects how inflation is when seen from my personal perspective. I took a basket of goods derived from our household consumption and scraped prices for a year (till last month). See (2) for data. Obviously, CPI represents the average citizen and no individual is likely to fit that definition well. I am no economics expert but I expected some correlation between the published CPI and my own observations but there were times they were moving in opposite directions. 1: https://arun.tanksali.com/topic/rpi/ 2: https://realpi.tanksali.com/#/app/dashboard
They have an API with documentation here: https://www.bls.gov/developers/
And a few other options here: https://www.bls.gov/data/
But I suspect that what you mean is that the raw data used to calculate these metrics isn’t available. Such as the “worker X on 25 Feb found that the price of a pound of flour at the Kroger store #357 is $1.” I suspect there are two reasons. (I can’t say for sure because I don’t work at the BLS, but I am a statistician, work for the government, and have been involved in decisions not to release data).
1. The BLS almost certainly contracts the leg work for the CPI. While the calculations are relatively simple (arguments about methodology excluded), the work to collect prices nationwide is nontrivial. Including permission to release the data publicly likely increased the cost of the contract. These data vendors also sell their price data to other “businesses intelligence” companies, and allowing the public release hurts their other business prospects, so they would charge a premium for it. And ironic as it may seem, government agencies/workers are interested being good stewards of the public’s tax dollars and had to make a value decision to not buy the rights to release the data.
2. For anything that isn’t legally/contractually prohibited from release, there may have been a conscious decision that the data at its most granular level either wasn’t useful to the public at all, or was likely to be misleading/misused because of critical gaps in what can be released. If the vendor for Alabama prices allowed release but the vendor for California prices did not, they may have just made the decision that releasing incomplete data set wasn’t a good idea.
Beyond the plethora of data that is published, you could submit a FOIA request for anything you feel is in the possession of the BLS, but not being published.
https://catalog.data.gov/dataset/consumer-price-index-cpi