58 comments

[ 3.7 ms ] story [ 98.0 ms ] thread
I don't really understand removal of graduation rates. Obviously that's meant to act as a proxy for learning, especially for those 38 states without proficiency exams.
Right, surely these states have some sort of standard that must be met for graduation even if it is not passing some exam.
the standards for graduation vary wildly so isn't really comparable
The standard to graduate isn’t uniform, so using graduation rates biases towards states with weak requirements.
> Obviously that's meant to act as a proxy for learning

That is the INTENT, sure. But does it work? If you have someone that graduated from a school A and someone that failed to graduate from school B that is in an entirely different area, can you draw any meaningful conclusions? Particularly knowing that school A has no particular requirements on why you graduate, while school B does?

I'm far more likely to complain about "But when we disaggregate student performance scores by racial categories (white, black, Hispanic, and Asian), the rankings change dramatically." I wouldn't want to go too far with that before ensuring that I wasn't trying to curb swimming deaths by keeping Nick Cage out of movies: http://tylervigen.com/view_correlation?id=359

The data is pretty useful in penalizing states that don't educate their students well enough to even get to the standardized testing stage.
First a point about bias. They are quick to point out the bias of traditional scores. But they do not point out that their bias is for libertarian causes. Had they not found that more libertarian states perform well by their measure, they would never have published this.

Removing graduation rates is definitely troublesome. Any school system could then improve its numbers by encouraging the worst students to drop out. Problem students both take more resources to educate and produce worse scores. That's bad for efficiency, but it is not obviously bad for society.

I am also troubled that they are using measures of performance in grades 4 and 8, but not something like SAT scores. I understand that SAT scores could be hard to get. But a lot of what people care about in an education system is how well prepared you are for college, and not how far along you were in grade 8.

> But they do not point out that their bias is for libertarian causes. Had they not found that more libertarian states perform well by their measure, they would never have published this.

The first statement is true. The piece would be a bit stronger if the researchers noted that by publishing in Reason, they're at the very least giving the appearance of potential confirmation bias. Your second statement makes an unsupported assumption though.

RE: SAT scores - not all students take the SAT, some students take it multiple times, the SAT is taken at different points in time per student, and it's a test that is so influential in college admissions that success on it is strongly correlated with wealth, as scores are best improved by tutoring on test taking strategy, not increasing general knowledge. I agree that grade 4 and 8 are probably not optimal to get a complete picture of education quality, but there is no reward to the student for doing well, so they're less likely to have spent substantial time investing in beating the test.

There is no reason given to why they take per-student expense out of the picture, other than 'we just don't think it should count'. And that seems to be the entire basis of their finding that rankings are riddled with flaws.

There is much research (once you sort through the partisan fluff from research institutions with obvious bias) that supports the positive impact of revenue per student on academic achievement on standardized tests. It's actually funny this article popped up, I legit just read a paper from 2015 called "A Cost-Benefit Analysis for Per-Student Expenditures and Academic Achievement". I genuinely read that this morning. Weird. The authors found that "There was a significant correlation between revenues available per student and ACT scores as one outcome measure of achievement." And just to drive that point home, they replicated the findings from a 2002 study, further solidifying that sentiment.

And that's just the most recent one I've read. I'm sure there are more recent. And that is definitely just one in the series of research related to per-pupil expenditures.

Also, I'm afraid this piece serves no purpose other than to be self-congratulatory to the 'lower taxes at all costs' group and right-to-work proponents. Why I say this: research pieces probably shouldn't include snide comments like

"high-tax, high-spending progressive utopias."

"punishing taxes"

Maybe that's just me? Am I off base?

edit: also, the comment section on that article is awful. Just awful.

>There is no reason given to why they take per-student expense out of the picture

What do you think would happen to the regression if per-student expense was both an independent and a dependent variable?

My experience with educational research is that researchers find per-pupil expense is directly correlated to improved achievement on standardized academic measures across all areas and for all variables in race/ethnicity, gender, age, and (obviously) socioeconomic status.

I don't know what variables the authors are talking about when they say "we ran multiple regression analyses on our data, which included several other variables", so I hesitate to make a claim on what their outcomes could be. There are so many things they could be looking at; it's almost impossible to know what they're really evaluating without being able to look at the data.

The main goal of this article was to refute such research. So, if you want to show that their methodologies are superior, I think that would be great.

Also they were assessing efficiency, and since the cost component of efficiency (performance/cost) is measured as per-pupil spending, then we wouldn't get very useful results if performance=f(per-pupil spending).

> There is much research (once you sort through the partisan fluff from research institutions with obvious bias) that supports the positive impact of revenue per student on academic achievement on standardized tests.

The article's methodology looks at academic achievement directly, so there is no need to use spending as a proxy.

Sort-of.

Per my reading of it, they used only one measure, the NAEP. The GP mentions a paper that used the ACT as a guide. In the least, we should resolve the difference between the NAEP and the ACT, if there is any. But, like the SAT or College Acceptance rates, those are just a few measures. Achievement is not easy to define in this context, let alone measure. All the test scores are just proxies for 'achievement' as a general term. Should we measure household income at the 10yr post mark too, College graduation rate, number of pregnancies, marathon runners? It's all just a proxy in the end for trying to determine, in granular detail, if education is worth spending tax money on.

> There is no reason given to why they take per-student expense out of the picture, other than 'we just don't think it should count'. And that seems to be the entire basis of their finding that rankings are riddled with flaws.

What? Why should you count expenditure as a positive? The only thing that ought to matter is achievement. If you hold achievement constant, but spend more, that is objectively worse. Which is the whole point of the article.

> There is much research (once you sort through the partisan fluff from research institutions with obvious bias) that supports the positive impact of revenue per student on academic achievement on standardized tests. It's actually funny this article popped up, I legit just read a paper from 2015 called "A Cost-Benefit Analysis for Per-Student Expenditures and Academic Achievement". I genuinely read that this morning. Weird. The authors found that "There was a significant correlation between revenues available per student and ACT scores as one outcome measure of achievement." And just to drive that point home, they replicated the findings from a 2002 study, further solidifying that sentiment.

Yes, but did they adjust for the confounding factors the authors of this article point out? I'll bet they didn't.

> Why should you count expenditure as a positive? The only thing that ought to matter is achievement.

Because achievement is hard to measure.

> If you hold achievement constant, but spend more, that is objectively worse.

Which again depends on what you mean by achievement.

Consider e.g., Lakeside's teletype terminal in 1960s. If computer access didn't increase aggregate ACT scores (which, it probably didn't...) was it then "objectively a waste of money"?

Having grown up in post-NCLB school system, I'm extremely skeptical of standardized tests as a measure of educational achievement. We're a couple decades into confusing the map for the territory on this one. It's Lockhart's Lament [1] on steroids and in every subject.

Funding levels are at least positively correlated with paying people well, which -- in every field I've worked in at least -- is typically correlated with higher quality employees.

[1] https://www.maa.org/external_archive/devlin/LockhartsLament....

> Because achievement is hard to measure.

Except that they already are measuring achievement. If you count expenditure as a positive, you assume your conclusion that spending is good.

> Which again depends on what you mean by achievement.

No, it does not. It is always worse to spend more for the same level of achievement. If your argument that achievement is mismeasured, that is a completely separate issue.

The intention is to measure post-treatment effects. You can't include a treatment effect as a measure of itself, that defeats the purpose of measuring the benefit.
I'm not 100% sure that you read the whole article. There's absolutely a reason given why per student expense wasn't considered: "intentions to raise performance are not the same as raising performance." That's a good argument for expenditures not to be included regardless of the impact it may have. Why use a questionable proxy when you could find better ways of actually measuring performance?

The "high-tax, high-spending progressive utopias" remark was mirroring the Paul Krugman op-ed that was mentioned in the introduction to the article.

To be honest, refuting Paul Krugman's opinion pieces isn't the most difficult thing to do.

;P

I only have 2 data points of experience on this issue. Which I know doesn't mean much, but I'm left scratching my head after looking at their list.

I spent the first 14 years of my life in Las Cruces, New Mexico (the state's second biggest city), and the schools I went to were not very good. I did high school in Maryland and the school system was a lot better (I initially struggled because I was so far behind everyone else - a friend of mine who moved to North Carolina for High School told me he had the same problem). In the author's ranking list, they place Maine at #48 and New Mexico at #41. I find this very hard to believe. There are so many problems that New Mexico has that Maine doesn't seem to have (gang problems, drug problems - at least at the magnitude that I saw in New Mexico). Unless Maine has some pretty bad schools, this just doesn't add up. US News lists Maine at #6, a ranking drop of 42 spots seems really significant.

I suggest you read the article if you're curious how they created their list. They explain their methodology pretty thoroughly.
I think part of it is the variability between school systems with in the same state. Las Cruces schools maybe below average for NM schools.

Off Topic: I went to Las Cruces a few years ago when my alma mater played New Mexico State. The scenery was breathtaking.

Even though my comment was kind of negative, I love Las Cruses. The scenery and weather are the two things I miss the most.

It's certainly possible the rest of the state has better schools, but I never heard anything about them while I lived there.

I'm confused by your comment, would you care to elaborate? Your anecdote has you moving from (the article's) rank 41 in quality to rank 6. You comment that the rank 6 state is much better than the rank 41 in education quality. However, you find their ranking hard to believe?
Disaggregating the results by race is real cute but also misses the point: blacks and asians and hispanics are residents of the state too and so leaving them in to evaluate the state's overall achievement (relative to other states) in education is fair. If anything it ought to be more problematic that Texas fails so badly at educating hispanic students when a fifth of them live in Texas.

Besides, the race in big-gap states is an unwitting proxy to income, so disaggregate by household per capita income (relative to local cost of living) if you want to disdain. This methodology would work well in all states and expose the challenges of anxiety and the quality of a student's home life in their education attainment.

> race here is an an unwitting proxy to income, so disaggregate by household per capita income

Poor white students do better academically than well off black students. So splitting by race is more informative when discussing academic performance.

Do you have some evidence to back that up? There are some dumb rich people, no doubt of every color, however there is a clear correlation between income and performance in every reputable source I’ve ever seen.

I find it hard to believe that on average rich black kids are doing worse than on average poor white kids.

"Disaggregation" doesn't mean taking them out--the ranking includes everyone:

> Think about that: White students do better in Texas than in Iowa. Black students do better in Texas. Hispanic students do better in Texas. Asian students do better in Texas. Given these facts, it is absurd for U.S. News to rank Iowa higher than Texas in terms of educational performance.

This, of course, makes sense. Students from disadvantaged backgrounds face unique challenges. Under the traditional methodology, states can do a crappy job of educating these students so long as they have few of them. States that have large populations of disadvantaged students, by contrast, are penalized in the rankings even if they do a relatively better job educating those students.

I'm surprised to find no mention of SES or income in the article. There can be similar Simpson's paradox effects within racial groups between states (for instance, Iowa might have more low SES white people than Texas). I would have expected this to be mentioned in the article for why they chose not to account for it in some form.
> If anything it ought to be more problematic that Texas fails so badly at educating hispanic students when a fifth of them live in Texas.

Texas doesn't do badly at educating Hispanic students -- it does a superior job at that. It has low overall performance because Hispanic students have terrible performance, and Texas, even though its Hispanic students are better than average, has a very large number of them.

This kind of thing is generally known as Simpson's Paradox. https://en.wikipedia.org/wiki/Simpson%27s_paradox

All good points, and clarifies my wording about Texas doing a good job (relative to other states) of educating Hispanic students that suggested otherwise. But the widely divergent results between the aggregated and disaggregated view isn't sufficient to declare, like the article does, that the ranking using aggregated view are using a flawed methodology. They're measuring different factors than the authors of the article, and the nuances of both are equally squashed under the pressure of producing a simple metric.

Cohorts are useful because they capture a group with some shared attribute, and racial cohorts are particularly useful because they're overwhelmingly consistent through time and datasource for a particular person. The disaggregation into racial cohorts in a ranking like this would be a useful indicator of how the outcomes of a particular person would change solely by moving states. But it's not an effective indicator of evaluating a state taxpayer's contribution to the quality of students coming out of the state's schools -- students who presumably perpetuate the results of the quality of their education in their life, producing a myriad of poorly-understood impacts on the state as they do so. Both of them are worthwhile questions to answer, but they're very different studies. But one calling another flawed on the basis of this is jarring.

Why isn't it an effective indicator for evaluating how much state taxpayers are getting for their contribution to the state schools? Can you elaborate on the difference between the two questions?
> But it's not an effective indicator of evaluating a state taxpayer's contribution to the quality of students coming out of the state's schools -- students who presumably perpetuate the results of the quality of their education in their life, producing a myriad of poorly-understood impacts on the state as they do so.

The opposite is true: the disaggregated ranking is the only good measure of "a state taxpayer's contribution" because the taxpayer has little to do with e.g. the fact that Hispanic students face additional challenges because they're from an ESL household.

What you're really saying is that the composite metric is a better measure of the average "quality of students coming out of the state's schools," which reduces to the assertion that it's better to have a school system full of privileged white kids than minority kids who face challenges. I don't think we should be ranking school systems that way.

It depends on what question you're trying to answer.

"How well does the average student do?" vs "If you raise a family there, how well will your children do?"

For the former question, you don't want to control for racial mixture. For the latter question, you do. Most parents should care more about the latter question than the former.

Education depends on population, their average IQ and their learning culture, everybody knows it and that's why the main consideration when people with kids buy a house is the population around it. No money or learning methods or good teachers is going to change this basic truth. It doesn't matter in which state you live, what matters is how much money you got to get to the right neighbourhood or the right private school.
I have several thoughts on this:

The race adjusted performance is interesting and useful but they're basically using it as a proxy for wealth. Why can't they just use some metric that directly represents wealth?

If Massachusetts schools are efficiency with money than the system nation wide is far more screwed up than anyone is willing to admit.

That state ranking depends in part on money spent is abhorrent. That should be tracked for reasons that should be obvious but not part of a composite score meant to represents results.

The bit about unions lines up with my anecdotal experience. I worked with the education department in college and they did not hold teachers unions in high regard. If your unions are so bad that college professors at a state school in a blue state gripe about how they hold back progress then I think it's fair to say your union is pretty bad.

They're not using race as a proxy for wealth, because the two things are different. There are unique challenges faced by minorities compared to white americans of the same income level. (Remember, Trayvon Martin was a middle class black kid whose dad was a white-collar government worker. A big part of #blacklivesmatter is educating people that black folks face unique problems that aren't subsumed within the larger issue of income inequality. And as to Hispanic students the challenges there are obvious--they're much more likely to be in families that are English as a second language, and also much more likely to move around a lot.)
I’m not convinced that ESL families is the main difficulty facing Hispanic children. I’ll accept that there are disadvantages to Hispanics but having ESL parents seems similar to those faced by almost any other child of immigrants. Perhaps one could argue that due to the visa system, non-Hispanic parents of Americans tend to have higher earning potential?

I have no idea whether this is true or not

Due to the visa system, immigrants in the U.S. are not identically situated to each other. Non-Hispanic immigrants are much more likely to already speak English and have advanced degrees because they come here through visa categories that specifically select for those things.
Sure, but their children aren’t. Children of non-Hispanic immigrants that are born in the US do better than US-born children of Hispanic immigrants. While English ability is clearly important in education outcomes, it’s not what makes the biggest difference.
What unique problem do you think the Trayvon Martin case proves black people face?

I'm a middle aged white male and I've been confronted by neighborhood watch and by the police while walking in other neighborhoods.

If I had attacked someone who confronted me, as Zimmerman claims Trayvon Martin did, it could easily have ended with a gunshot.

I agree that a big part of #blacklivesmatter is claiming that black folks face unique problems, but in order to make that claim they ignore the cases in which white people have the same experiences.

As a white person from an impoverished family who grew up in a mixed race inner city neighborhood, I see a lot of wealthy white people who claim my life experiences don't happen to white people. I see Hollywood and the news media making the same claims. I see BLM making the same claims.

That's a distorted perspective of life in America that ignores the existence and life experiences of a lot of people.

No, it's doing worse than ignoring them. It's actively trying to silence them.

Ranking is nearly meaningless. eg compare these:

School A - full of underachieving kids that are pushed to get avg grades

School B - full of avg kids getting above avg grades

School C - full of smart kids getting top grades

School D - full of smart kids getting nearly top grades, but soft skills, nice personalities and VIP friends

School E - Mix of kids getting relevant grades

Which is the best school for each kid type? They're impossible to rank. Many schools with top grades are only there because they kick out failing kids, not because they're better at teaching. Many schools are rated good because they can improve grades, but you wouldn't want to send a smart kid there. Many private schools sell themselves on creating well rounded kids, not necessarily good grades.

I went to school F - full of average kids who knew how to play the system to get top grades, as well as to get into high/elite level colleges.
Yes, but all the different 'types of kids' should normalize across a state-level population.
Is it just me that has a hard time understanding the racial aggregation thing. They claim to have split Iowa and Texas results by race and age. And found that Texas did better than Iowa for whites blacks and hispanics - but then aggregating the results ala USNews, Iowa comes out on top

I am struggling to imagine which category is the flip category. If iowa is not doing better for white black or hispanic, which racial category is somlatge as to flip the average?

Example using two races and just ACT scores:

Texas:

50% Hispanic who average 25

50% white who average 29

State average: 27

Iowa:

10% Hispanic who average 24

90% white who average 28

State average: 27.6

texas has 50 % hispanic population ?

https://en.m.wikipedia.org/wiki/Demographics_of_Texas It has 84% white population, not noticeably different to UK makeup at 87%

Iowa has 88% White (https://en.m.wikipedia.org/wiki/Iowa) or maybe 91 not too clear what they mean as difference between white and not hispanic

So them is some pretty small numbers to play with - somehow the "whites do better, and iowa has more whites" argument has only got 4-7% to play with

I would like to see the actual figures but it smells to me

The numbers used in the parent were for illustration only, not necessarily the actuals. Also, student population != general population.
The person you are replying to clearly is giving made up data to explain to you how Simpson paradox works.

Second, instead of looking at overall racial makeup of a state, you should be looking at the racial makeup of public schools. For Texas it’s been 52% Hispanic and 28% white. The former percentage is steadily rising year over year, while the latter is steadily falling.

The efficiency measure sure seems dangerously misleading. My SC high school didn't have the budget to replace broken tables (which were simply overturned until a handy-ish student managed to unsteadily prop them up) and sent seniors home home early via free periods to cover for a lack of teachers.

Was that efficient? I suppose. Was it a desirable education? Not really.

I don't have enough familiarity with this sort of research to evaluate it quickly, but I was a little curious about the authors' backgrounds:

> Stan J. Liebowitz is the Ashbel Smith Professor of Economics at the Colloquium for the Advancement of Free Enterprise Education in the Jindal School of Management at the University of Texas at Dallas.

> Matthew L. Kelly is a research fellow at the Colloquium for the Advancement of Free Enterprise Education in the Jindal School of Management at the University of Texas at Dallas.

Here's another piece by them on net neutrality: https://www.insidesources.com/state-governments-drop-net-neu...

Here's the site for the "Colloquium for the Advancement of Free Enterprise Education": https://jindal.utdallas.edu/centers-of-excellence/capri/cafe...

Here's a press release about the founding of this entity, which includes some information on where the funding for it comes from: https://jindal.utdallas.edu/news/new-program-at-the-jindal-s...

I think this is either an over-reaction or a bit disgenuine. I was careful not to suggest (and certainly not argue) their research be dismissed for any ad-hominem reason.

The fact that they are members of some brand-new academic* unit that claims to be "for the Advancement of Free Enterprise Education" suggests that there's some non-zero chance they knew the conclusion about public spending on education before they started.

I raised a small set of facts without making any claim/appeal/accusation of anything sensational or conspiratorial.

If researchers at the Colloquium for the Advancement of Drink Your Damn Milk publish a study concluding you should drink your damn milk, it's reasonable to weight their conclusions accordingly until you see how others with more domain knowledge assess them.