They’re normalized against previous tests: when was taking ACTs, they would always say “one section is not counted, but you don’t know which one it is, so do your best on everything.” Weird thing to say, but anyway, I’d been told that one section is used to test upcoming questions, and includes previous test questions to correlate performance cross-tests. (Remember, there’s multiple test dates per year, with multiple versions per date. They have to be consistent within the year as well as across time)
I don’t recall the SAT doing this, but I’d assume they’re doing something similar.
These days they count all four sections and add a fifth calibration one. I'm sure some people screw around on it, but they probably developed some heuristic to select only the students who tried on that section.
The SATS at some point had an experimental section that was undisclosed: in other words students had to take it seriously because they didn't know what section was experimental and which one counted. I don't know how long this practice ran.
Since the ACT tests reuse test questions, I'm guessing they aren't able to be transparent and you ultimately have to trust that the test makers know what they are doing.
Supposedly, the SAT and ACT use statistical testing to ensure that their scores are consistent across years.
Among other measures of difficulty, they have students preview the questions of future tests in a fifth, ungraded section, to make a correlation between their scores.
That’s a great question that definitely doesn’t deserve downvotes.
It appears the test method did start to change in 2020 [1], but it doesn’t seem that these changes would obviously affect performance. Perhaps many/most have moved to the computer based testing, and there could be some disadvantage there?
> More graduating students taking the ACT are also doing so through statewide and districtwide testing programs. For the class of 2022, 60% of students who took the ACT did so at least once through those programs, compared with 27% in 2015.
> Fifteen out of the 23 states participating in the program require students to take the ACT in their classrooms during regular school hours, regardless of whether they plan to apply to college.
> The aim is to increase access to the college admissions exam to students who may not otherwise have the opportunity.
Makes sense to me… require more students to take an exam they otherwise wouldn’t have and you will get an average lower score. Students that choose to take the exam are more likely to be prepared for taking it, and thus will study more and achieve higher scores.
Regardless. Agreed that this is potentially a good thing for helping highlight which schools need more assistance.
There's an interesting opposite effect at the university level: undergraduate institutions have been dropping SAT requirements, and then their average score goes up, as students who did or would do relatively badly on it don't submit it.
Makes sense. If no student discloses their SAT then the university would have to assume that any given student achieved the average SAT score. If you're a student who got an above average SAT it's therefore in your interests to disclose your score.
Although these incentives should theoretically cause everyone to voluntarily disclose their SAT scores.
I get that as a profession journalism exists to report happenings largely uncritically but I wish we could get the teeniest amount analysis because just a little thought would have made this a non-story or a very different headline, "students, don't assume your ACT scores being shown as above average means they'll be above average in the pool of applicants at university" with an explanation in the article that this has always technically been true but that the gap between the average as reported by the ACT and the average as seen by schools will be wider this year and here's why.
Unfortunately this drive to testing has had very concrete negative effects on teaching real skills - even the best schools now ‘teach to the test’, and have been skipping learning of practical applications or fundamentals pretty heavily. At least from what I’ve been seeing in person.
Some of us were aware of this as students two decades ago. One of my friends had a weird way of looking at it: ignoring the assigned work and jumping ahead, he felt like he was cheating by having an easier time on tests.
GPA, letters of rec, honors, and personal essays are all subject to the inherent privilege richer students and students in stable households have. Got to do something cool over the summer because your parents can afford to give you that luxury? Teacher writes you a letter of rec because they connect with you more. Have a stable household? GPA goes up and you take honors classes because your focus is uninterrupted.
I was the kid who every year when asked what they did for summer said "nothing". I was the kid who missed homework assignments because of yet another argument or separation between parents at home last night where I played peacemaker. On my personal essays I had to lie and make up experiences I had never had because we simply did not have the money for a vacation or takeout.
GPAs, letters of rec, honors only prove that a student has a safe and secure environment and they are able to excel over a long term. That is a reflection of privilege.
For those of us who grew up in poor and broken households, standardized testing is the only chance for us to get a better life. Everyone who speaks out against it does so from a position of privilege because they have 0 idea what it's like to grow up in that type of environment.
GPAs, letters of rec, honors only prove that a student has a safe and secure environment and they are able to excel over a long term. That is a reflection of privilege.
For those of us who grew up in poor and broken households, standardized testing is the only chance for us to get a better life.
The suggestion that somehow there is a single path out of this sort of hell is overselling the point and over generalizing.
I grew up in a house where my parents were two warring alcoholics, one of which didn't want to have kids at all and the other didn't want the responsibility of raising them. And true enough, I didn't do well in the grades department... but I graduated, had some collected honors at graduation time, had the letters of recommendation I needed, and I absolutely participated in extra-curricular activities if anything to escape being home. Standardized tests? I took the ACT on a whim when my friend said he was going to go in the morning and asked if I wanted to join. I scored well (much better than I expected), not spectacular, just did well. Didn't really move the needle for me nearly so much as my ability to "work the room" with those that could help me get through school and onward as a sub-standard student. In the end I did fine.
My wife hits your points even closer. She was the child of poor immigrants whose mother was very severely schizophrenic. Between having no money and dealing with constant delusional and violent outbursts at home she managed to graduate high school as the school valedictorian (in a large metropolitan school), graduated with full honors, and received a full ride scholarship to her first choice school. She did the work because she knew if she didn't she wouldn't escape the hell of home she was in. Know what she didn't do well with? Her standardized tests. By the time she had reached high school she was prone to anxiety attacks and formal, timed tests were just too much. In fact later in life she wanted to take the GRE and go to graduate school and she couldn't get through the test without a full break down. Even so, she's done well in life despite it all.
But the point is that your expression of your experience is only that: your experience. For both my wife and myself, the standardized tests were simply not factors despite the hardships we faced. This is why anecdote isn't a good way to come to real knowledge... it might have been true for you but other's mileage will vary.
A good friend of mine grew up (as far as I know) in a loving and stable household, though her parents eventually divorced. She had rotten SATs, probably because she attended a terrible high school. There should have been three high schools, but that would have impaired recruiting for the football team, and this was Texas in the 1970s. Oh, and the football coach was also the physics teacher. If he wanted to get an early start on practice prep, there was no physics class that day, and in those days there was not likely to be an appeal beyond the football coach.
AP is another form of standardized testing. GPA is totally arbitrary, there are schools that graduate students with elementary school skills with 4.0s, same with letters of recommendation. Personal essays can be gamed by hiding editors.
All of those are more hackable than tests. Studies show, for example, that essay scores are highly correlated with income. You think Lori Loughlin would’ve needed to risk jail time to get a stellar letter of recommendation from someone famous?
You can definitely do well in SAT with just books. Definitely better than other metrics where the rich families can hire guys from outside to do it or arrange it for the rich kids.
> Regardless. Agreed that this is potentially a good thing for helping highlight which schools need more assistance.
Is it? There are likely many opportunities best pursued through vocational training rather than preparation for college. Given finite learning time, keeping students out of vocational training to improve test scores on a college preparation test is probably not the ideal outcome.
We should stop stigmatizing vocational training, and skilled trade work. The ACT doesn't test for student outcomes. Improving test scores may be counter-productive.
It’s interesting to consider how one would design experimental controls for cognitive deficits after COVID-19 illness when studying academic performance at population scale.
You know, the same guidelines you point to also say " Please don't comment about the voting on comments. It never does any good, and it makes boring reading. "
Is this a calculated likelihood given certain established premises? How do you figure that injury
to the physiology that performs cognition has a trivial impact compared to not showing up to school for some time? From what basis do we derive that certainty?
We haven’t done so, have we?
We’d want to be sure, right?
And: In what way were lockdowns unnecessary? It isn’t and it never was safe for children to be infected. There is irreversible harm to children from it at an indefensibly high rate. It is obvious in children in my vicinity. I’m fairly sure my own son has lingering digestive issues after his bout. I may never know because society will not look.
Nonetheless, lockdowns certainly were a bad fit to the main mode of transmission which is infectious aerosol.
"How do you figure that injury to the physiology that performs cognition has a trivial impact compared to not showing up to school for some time?"
There are plenty of pre-pandemic studies about truancy and poor curricula leading to lower test scores. It's practically a tautology that if you aren't going to school and studying for the tests that you will score poorly on those tests (the measure here). Could there be some trivial (in aggregate) physiological impact at play? Maybe, but that's not the biggest factor based on this established research.
"I may never know because society will not look."
It's not that society won't look, but rather that minor issues are deprioritzed. If you have a new disease, there are likely to be tens of thousands unanswered questions around it. It will take decades to get to most of the questions. Some may never be answered or explained. It's unlikely anyone cares enough about an individual with digestive issues (or recurring intermittent swelling after vaccination) to do a study looking at it. If it's sufficiently rare, they may not be able to even identify it as an issue worth looking into.
There are plenty of countries that did not force unnecessary school closures and other useless covid theatre onto their students, and students in those countries do not exhibit the same amount of unprecedented learning loss as students in the US have, despite similar levels of infections across countries.
Pandemic policies are a huge elephant in the room, and it's extremely annoying to see people like you trying to pin the blame on the pandemic itself. School closures were a disastrous policy, full stop. This is the price of that choice of policy.
I am talking about an effect. (You are talking about… people like me? Being extremely annoying? Because I’m not talking about a different aspect than the one I am talking about?)
How can we measure this effect?
How do you know that it’s not the measured cognitive deficit after the infection and only school closures? How was it measured?
We’d want to analyse and investigate the effect, right?
Do you have any relevant data? We can what-if all day. If you have data pointing to this, I would like to see it.
Your first was a legitimate comment for discussion. The recent one doesn't seem interested in a reasonable discussion. The prior commenter makes a solid point and you get hand-wavey because there's not a study supporting it... when your initial comment was just as theoretical (does your study show this is applicable to kids, at what rate, in which countries, what are their test score results, etc).
I’d invite you to take a look at the multitude of scientific results on cognitive impairment after infection and the numerous physiological mechanisms which are known to be at play. A number of them pop up here on Google Scholar: https://scholar.google.com/scholar?hl=en&as_sdt=7%2C39&q=cov...
Note that this is the only data which has been provided in this discussion as far as I can see right now. What made the commenter’s solid point a solid point? Was there data?
Of course shutting down schools did impact grades.
Of course it did!
In part: Closing schools was poorly executed, and the lockdown/closure style of mitigation was never based in the scientific facts of the main mode of transmission, so it was always a blunt and stupid instrument.
For the avoidance of doubt: I’m raising the point that we should look at other things too. And interested in thinking about how we
would look.
—
I don’t have pediatric studies on cognitive impairment.
I’ve seen quite a bit of literature on the mechanisms that cause it in adults, and only poor reasoning for children being exempt from severe and/or long-term effects. (Is this disputed?) So I’m for monitoring and studying and looking. And thinking about how to do it.
Essentially you can't. Multiple other factors have changed that would affect grades.
If you strictly want intelligence and not just grades, then you'll have to identify kids that had recent intelligence tests before the pandemic and compare them to others. This is problematic because types of intelligence can vary, so some areas might not be covered by the tests. In general, few kids recieve IQ tests and so there's likely to be observation bias with a higher number of kids being tested for some reason. Finally, it's unlikely you could track pre-/post-infection since kids haven't typically been testing every week and mild to asymptomatic cases are fairly common I'm children.
Best case, you can do very small studies on an individual level without control groups, essentially individual case studies. Even if you can, it's not going to answer the questions related to this article/thread.
The effect is not present in countries that did not close schools. The effect strongly correlates with the level of disruption to schools and schooling caused by policy choices. The effect does not correlate with infection levels among that cohort of the population.
There is a strong political desire for politicians who enacted these disastrous policies to avoid blame for them, one strategy is to frame all the policy choices as unavoidable, i.e. claiming that you "had to close schools". Another political strategy is to blameshift all the bad effects onto the virus itself, which is exactly what you're doing.
Your post is a prime example of "JAQing off"; you are "just asking questions", saying things like "shouldn't we study this?", or "there are no studies showing it's because of <insert obvious explanation here>".
> Another political strategy is to blameshift all the bad effects onto the virus itself, which is exactly what you're doing.
What’s up with this “you are like this and that”? What’s with the accusations and insults? What’s your problem? Take it elsewhere!
—
On topic, we’re diverging on the word effect. I am using the word effect to mean “something that the illness is known to do in certain measure” – an effect of the disease on the population. You’re using the word effect to mean effect size in a particular study. It’s an understandable divergence.
Just to make sure: I am not saying. That the grades dropping. Is caused by post COVID cognitive impairment.
I said that it was interesting to think about how you’d design a study so it would control for such an effect.
I see studies about cognitive impairment on the individual level. I see studies about this and that on a population level. I see lots of people saying there’s no connection. I don’t see how they know that.
Would our infection rates have been similar, had we left schools open? I'm skeptical. How about other measures to limit transmission (voluntary and mandatory), were they also similar in those countries?
FWIW, I tend to downvote comments that discuss voting very mechanically. It's not any position relative to the content... I find discussion of how we vote on comments very uninteresting, and I vote accordingly. I don't think I'm alone, and I suspect that I'll see a downvote or two for this one, but I felt it might be kind to point this out just the same.
Although it may be unpopular, tests shouldn't be weighted based on extrinsic factors like race, gender, ethnicity, socio-economics, or health condition. If particular students or certain demographics aren't achieving, direct help to them rather than lower the bar. Build achievement and excellence, not participation trophies. If stats decline for a few years due to widespread neurological damage and/or a lack of pedagogical impact, then it is what it is. The field of education should set goals to get back there rather than fudge the numbers.
You're conflating multiple issues and misapplying the data.
If we're trying to study a problem, we should identify possible root issues. The problem of lower test scores have much better options to investigate. This is why the studies are focusing elsewhere. It's like saying the boat is sinking and we should study the rain instead of the hole in hull or the broken bilge pump.
I’m not conflating anything or misapplying anything! You’re reading things into my words that are not there!
I’m interested in how to study a known side effect of a disease which has been a global pandemic for two and a half years. How to study it at scale.
I’m not conflating it with the whole host of other reasons for the grades dropping, most of which are obvious factors
and obviously bigger factors. The conflation is happening inside your head, somewhere from between your eyeballs and your fingertips. Respectfully.
You are of the opinion that it is obviously a trivial effect and not worth investigating especially. I disagree: I think we should look into it. I do not disagree in the way that I think cognitive deficits from COVID is the main driver of grades dropping.
Our opinions do differ, but not where you are saying we differ.
I think we should make sure we aren’t spilling solvents into the boat as they can degrade the hull. Why don’t you take a look at the bilge pump?
Absolutely, like an NTSB investigation. There is no benefit in assuming any particular conclusion beforehand. Qualitative and quantitative analysis to discover the factors. Perhaps something is amiss or something is beyond human reach. All-in-all, the US education system has steadily declined since the 1960's. A foreign language and Latin were required of everyone, even those not going to college. Financial skills, driving skills, physical education, workshop, drafting, and so on were standard. There's no need to revisit the culture of the past, but there is room to insist on excellence, preparedness, and instilling curiosity rather than drown students in counter-productive standardized tests and throw teachers on food stamps into classrooms with 50 students without resources. The Finish approach is worth copying.
The tests aren’t weighted based on extrinsic factors. If you’re Black and take the SAT you don’t get extra points or more time. In fact, you’re much more likely to get more time if you’re an affluent white parson who has requested and received accommodations for ADHD, etc.
Linking a google scholar query with results does not imply a broad base of valid evidence. It just means people went looking and published.
The top three, which are the only ones I bothered to skim, have quality problems.
The third one seems to be the most cited so I spent the most time with it. It in effect reduces to “if you were hospitalized with COVID” you now have significant cognitive impairments, but from my brief reading there is no “before” so it could just as easily be titled “cognitively impaired people make bad life choices that lead to pre-existing conditions that lead to (exposure to and) hospitalization with COVID.”
Additionally the sample of N at 84k is misleading as only roughly 200 people are in their “significantly impaired” set.
It was a quick read, as I said, and I may have oversimplified, so if I’m wrong, please feel free to respond with corrections.
And at 200 in 84k, the percent of affected individuals are unlikely to affect aggregate data in any practical way (if those numbers even apply to peds, which I'm skeptical of). Nor would there be any actionable solution.
If – if! – we take it as a valid premise that cognitive impairment exists after COVID-19. In adults. And if we take it as a valid premise that this is exhibited in younger people too.
Then:
I find it interesting to think about how one would study progress or decline in an education system, i.e. to look at that system itself to maintain and increase quality of the system. And: if and how one can reveal or disprove a population-scale effect on cognition due to this particular illness.
Regarding cognitive deficits or impairment, I’m not focusing solely on brain damage as the cause, although it is known to occur. There’s also vascular and endothelal dysfunction, endocrine issues like abnormally low cortisol and blood glucose dysregulation, and immune dysregulation – T-cells in the brain (not good, not usual), and elevated inflammatory cytokines which are known to affect behavior. (This is all unsourced; There exist sources for it.) A lot of this is reversible. It’s not known that all of it is.
I think we ought to look at these issues. In this context. I’m interested in thinking about and discussing the scientific mechanisms with which to look.
I’m curious why you think any of this should be novel to the coronavirus COVID-19. We are affected by every infection.
There are a number of studies that show if you ask people about the long-term effects of any infection they will report similar numbers to what is being reported about coronavirus. You can take this to support the previous paragraph, or as an instance of placebo effect.
If you’re interested in how viral infection impacts brain function there’s a lot of cool information out there. I would not start in a speculative area like COVID-19. Maybe start with viral meningitis.
Alternatively if you want to stay speculative some people think that mutations related to our rapid evolutionary increase in brain size may have been in part down to zoonotic virus exposure a couple million years ago.
edit: Maybe you would enjoy The Tangled Tree by David Quammen.
Thanks for the book recommendation! Much appreciated!
> I’m curious why you think any of this should be novel to the coronavirus COVID-19. We are affected by every infection.
This review paper might summarize my inclination to consider SARS-CoV-2 different in this regard: “The Neurobiology of Long COVID” — M. Monje, PhD, Neurology at Stanford & A. Iwasaki PhD, Immunobiology at Yale. Publication status: In Print, Journal Pre-Print Available Online October 7, 2022 — https://doi.org/10.1016/j.neuron.2022.10.006
One paragraph mentions effects in children:
“Similarly, a 2-year a retrospective cohort study using the TriNetX electronic health records network of 1,487, 712 individuals with COVID-19 and a similar number of matched controls with a different respiratory infection found continued risk of cognitive impairment at 2-years from diagnosis (Taquet et al., 2022). In contrast to risk of cognitive impairment, the risk of anxiety and mood disorders normalized within 2 months following SARS-CoV-2 infection (Taquet et al., 2022). This TriNetX study provides some of the only available data about cognitive deficits in children and found in this retrospective cohort study that children did not exhibit an increased risk of anxiety or mood disorder diagnoses, but did exhibit increased risk of cognitive deficits (HR 1.2) and insomnia (HR 1.29) compared to children with different respiratory infections (Taquet et al., 2022). Unlike the adult cohort in this study, the risk of cognitive deficits in children normalized to the level observed in controls with a different respiratory infection by 491 days after COVID-19 (Taquet et al., 2022).”
Referencing table S9 (for children) in the appendix/supplementary there are only four categories of illness where the p values are still possibly meaningful after correction: cognitive deficits, epilepsy or seizures, intracranial hemorrhage, insomnia
The two common conditions, cognitive deficits and insomnia, are only significant because of the arbitrary follow-up period selected of six months. If you examine the charts shown in Figure S1 in the same appendix, you can see that cognitive deficit has no long-term effect, and that actually COVID-19 seems to be protective in the longest term (as is the case with anxiety). This to me is suggestive of the fact that social aspects/expectations are dominating the reported symptoms for these common experiences. None of the symptoms, even insomnia, are quantified.
I can't find if/where they break it out by sub-classification but they seem to lump together real measurements of cognitive deficit with self-reports of brain fog. Similarly anxiety is not well-quantified.
For intracranial hemorrhage and seizures the effect is more pronounced, but as you can probably infer from the pixelation of the charts in S1 for these two, the sample sizes here (children with these symptoms), are tiny, and so the results are underpowered. If I wanted to keep looking this is the only avenue I'd consider.
As far as effects in other populations, they compare e.g. sinusitis and laryngitis on equal footing with lung infections resulting in hospitalization and possibly ventilation for COVID-19. That to me suggests there is the problem of confounding based on the impacts of reduced perfusion and oxygenation of brain tissues due to actual major lung infections that is more present in the COVID-19 data than the control dataset. A respiratory infection for this purpose ought to be something that can seriously impact perfusion and brain oxygenation -- they don't seem to have realized that and instead used "respiratory infections" without further qualification.
The Taquet et al study is the best study I've seen on it and I'm still totally unconvinced this is anything beyond hypoxic brain injury (and only minorly so -- in most survivors it seems to be primarily fear, hypochrondria or provider nudging).
The passage on the effect in children is a tiny part of the whole work. From the introduction:
COVID-19 may affect the central nervous system in (at least) six main ways (Figure 1). First, the immune response to SARS-CoV-2 in the respiratory system may cause neuroinflammation - increasing
cytokines, chemokines and immune cell trafficking in the brain and inducing reactive states of resident microglia and other immune cells in the brain and brain borders. Second, SARS-CoV-2 rarely may directly infect the nervous system. Third, SARS-CoV-2 may evoke an autoimmune response against the nervous system. Fourth, reactivation of latent herpesviruses like Epstein-Barr virus may trigger neuropathology. Fifth, cerebrovascular and thrombotic disease may disrupt blood flow, disrupt blood-brain-barrier function, and contribute to further neuroinflammation and/or ischemia of neural cells. Lastly, pulmonary and multiorgan dysfunction occurring in severe COVID can cause hypoxemia, hypotension and metabolic disturbances that can negatively affect neural cells. It is important to recognize that these mechanisms of nervous system injury are not mutually exclusive, and a combination of mechanisms may occur in some individuals, with varying frequency and timing.
It has been shown that there are many markers for Long COVID, many of which present as neurological and cognitive issues. Two markers have been shown to be reliable and consistent: Patient reports – and unusually low cortisol levels. The other markers are things such as elevated inflammatory cytokines. Fear and hypochondria and provider nudging do not manifest in those ways. Also: Provider nudging? I have heard of nothing but universal and unscientific dismissal of anything COVID-related when people go to health providers for long-term issues. I don’t think that’s it.
I can’t see how this could be anything less than real, and how children could possibly be exempt from the effects. And we aren’t looking because we think we know it’s nothing.
Most kids did not suffer significantly if they got Covid. The vast majority of the kids who got Covid had just mild symptoms. I'm not saying there were no kids who had it bad, but they formed a very small minority, and could not have dragged down the average ACT score.
Does the article talk about the fact that so many universities stopped requiring the ACT or SAT for admissions? It sounded like this was a trend pre-COVID (because of the prevailing belief that standardized tests discriminated against minorities and poor people) and like many other things was accelerated by COVID when test-taking became impossible. This must have had some interesting effects.
Colleges are addicted to the new normal and will never go back. Applications surge by tens of thousands -> colleges make more of the $75 application fees. Acceptance rates go down -> your ranking in USNews goes up. Average ACT score increases (because why would you submit an under average score now) -> you can make your school look more prestigious than it actually is. Is any of this good for students? Standards can only continue to drop so low.
Just look at Harvard. Since they don't require any additional essays applying there is now literally just a button press away - wouldn't you?
I suspect the new normal of not requiring test scores has at least as much to do with university admissions feeling more free to mold their incoming classes without the pressure of maintaining a ranking stat (test scores).
I never understood the pressure for highest SAT average. There are way more people with lower scores who want an education, and they don't want to go to a school where they have the lowest "aptitude" or preparation.
I want to play professional basketball with players of extremely high aptitude but I’m 5 foot 7 and can’t shoot or dribble. Somehow the Bulls aren’t interested in me
It's the difference between a revenue stream and a sum of assets under management. The $50B isn't liquid and can't be used to distribute bonuses. The yearly $75,000 however can be distributed among admissions office admin with impunity, for the labor of accepting file uploads.
More applicants at the same number of acceptances makes the school literally more exclusive. They think that a smaller reported % makes them look better.
Yeah, it's not my personal opinion, but this seemed to be the reason given publicly by various universities.
The thread from last week showing that "Asian" kids' SAT scores crushed "White" kids' scores in Michigan was pretty eye-opening alongside the current Supreme Court cases regarding admissions and Asians.
One of my favorite opinion pieces about the SAT, from a few years ago, came from a black guy who grew up in rural Mississippi or somewhere like that. He argued that the SAT was the only thing that gave him a level playing field, where he could prove his equivalence, in college admissions against kids from elite private schools and wealthy public school districts. His position made sense to me.
Under a democratic (or let's say, public-opinion-based) system, who is rewarded for trying to uphold some kind of standards, when it doesn't benefit the majority to do so? I'm just commenting aside from the pandemic-related issues here, on the long progression of anti-testing, anti-standards pressure in the US.
The majority of people aren't happy with having standardized tests because they block kids from feeling they have a shot at the top schools. But of course that's their function -- to sort.
So, public pressure calls for dropping objective measurements so that people feel better about it... but they aren't doing better. And muddying the waters to say that everyone should have an equal shot at going to a top school -- for reasons other than academics and using the resource efficiently. And the kids just stagnate in performance and everyone feels good about it, until you get kicked in the ass by reality and you find other countries who don't care so much about making people feel better are actually passing you by.
But into what categories are they sorting? What if they are mostly measuring who is good at taking a test?
At my kids high school, they were in a program where they got to write a whole bunch of practice tests. From test-to-test, they did better and better. Some of the tests were close enough in time that their knowledge didn’t significantly change, but their scores improved. I attribute that to learning answers to questions that pop up frequently and to being less anxious after writing the nth practice exam.
Both of my kids did great at the tests eventually.
The ACT like all standardized testing is worthless. I git an 18 twice while my brother who took it drunk got a 21 and therefore received scholarships. It's a sham.
I tend to agree but also acknowledge that TV and radio did have engagement feedback loops. Viewer numbers and focus groups were/are used to “optimize” the content. Those are obviously not as extremely personalized as what’s available in modern forms of media but I struggle to identify a specific tipping point at where such feedback loops become problematic.
This is funny to me because this particular distinction between the two forms of the drug is a decades out of date bit of racist drug war propaganda that was widely spread through TV news. I'm not sure if that says anything about the relative risks of these media formats, but it's grimly funny.
I have many times yes. I was a homeless addict for years and neither was my preference but any port in a storm you know.
They are definitely different experiences, but nearly all drugs have different delivery mechanisms with different onset times and intensities and long-term consequences of using them. The crack-vs-powder cocaine difference is nowhere near, for example, the difference between smoking and injecting heroin. But that is not considered a different, more dangerous, more brutally punished drug if you inject it.
And more importantly the subjective experience of the delivery doesn't matter, the origin of the distinction is in the historical record. Crack cocaine was given different, much harsher, sentencing guidelines for unambiguously racist reasons, that then needed to be justified in the public consciousness.
Possibly? Is that so incomprehensible? It's based on a relationship between drugs that only ever existed to justify a racist sentencing policy. Not a strong basis for an effective metaphor imo, but I'm not your editor.
Historically college applications have gone up during recessions as the opportunity cost of college drops due to worse job prospects (https://www.communitycollegereview.com/blog/why-student-enro...). With that said, significant changes in average scores are much more likely to be related to changes like who is taking the test, and massive shocks like the shutdowns during COVID that had a major impact on teaching and student outcomes across all age groups (https://www.theguardian.com/education/2022/sep/01/math-readi...).
Well sure, but TikTok, Netflix and Fortnite scores are at their all-time high!
More seriously, I'm not sure that we should worry so much that students didn't learn what we wanted them to learn for the last two years. Maybe they learned something else that might turn out to be beneficial someday.
On the other hand, I'm not sure how many "influencers" our society actually needs.
113 comments
[ 3.3 ms ] story [ 432 ms ] threadCouldn't it be that tests after the pandemic became harder?
I don’t recall the SAT doing this, but I’d assume they’re doing something similar.
Since the ACT tests reuse test questions, I'm guessing they aren't able to be transparent and you ultimately have to trust that the test makers know what they are doing.
Among other measures of difficulty, they have students preview the questions of future tests in a fifth, ungraded section, to make a correlation between their scores.
It appears the test method did start to change in 2020 [1], but it doesn’t seem that these changes would obviously affect performance. Perhaps many/most have moved to the computer based testing, and there could be some disadvantage there?
1. https://www.winwardacademy.com/act-announces-big-changes-ret...
> Fifteen out of the 23 states participating in the program require students to take the ACT in their classrooms during regular school hours, regardless of whether they plan to apply to college.
> The aim is to increase access to the college admissions exam to students who may not otherwise have the opportunity.
Makes sense to me… require more students to take an exam they otherwise wouldn’t have and you will get an average lower score. Students that choose to take the exam are more likely to be prepared for taking it, and thus will study more and achieve higher scores.
Regardless. Agreed that this is potentially a good thing for helping highlight which schools need more assistance.
Although these incentives should theoretically cause everyone to voluntarily disclose their SAT scores.
Journalism exists to sell newspapers, and "just a little thought would have made this a non-story" is the reason the headline is the way it is.
I was the kid who every year when asked what they did for summer said "nothing". I was the kid who missed homework assignments because of yet another argument or separation between parents at home last night where I played peacemaker. On my personal essays I had to lie and make up experiences I had never had because we simply did not have the money for a vacation or takeout.
GPAs, letters of rec, honors only prove that a student has a safe and secure environment and they are able to excel over a long term. That is a reflection of privilege.
For those of us who grew up in poor and broken households, standardized testing is the only chance for us to get a better life. Everyone who speaks out against it does so from a position of privilege because they have 0 idea what it's like to grow up in that type of environment.
For those of us who grew up in poor and broken households, standardized testing is the only chance for us to get a better life.
The suggestion that somehow there is a single path out of this sort of hell is overselling the point and over generalizing.
I grew up in a house where my parents were two warring alcoholics, one of which didn't want to have kids at all and the other didn't want the responsibility of raising them. And true enough, I didn't do well in the grades department... but I graduated, had some collected honors at graduation time, had the letters of recommendation I needed, and I absolutely participated in extra-curricular activities if anything to escape being home. Standardized tests? I took the ACT on a whim when my friend said he was going to go in the morning and asked if I wanted to join. I scored well (much better than I expected), not spectacular, just did well. Didn't really move the needle for me nearly so much as my ability to "work the room" with those that could help me get through school and onward as a sub-standard student. In the end I did fine.
My wife hits your points even closer. She was the child of poor immigrants whose mother was very severely schizophrenic. Between having no money and dealing with constant delusional and violent outbursts at home she managed to graduate high school as the school valedictorian (in a large metropolitan school), graduated with full honors, and received a full ride scholarship to her first choice school. She did the work because she knew if she didn't she wouldn't escape the hell of home she was in. Know what she didn't do well with? Her standardized tests. By the time she had reached high school she was prone to anxiety attacks and formal, timed tests were just too much. In fact later in life she wanted to take the GRE and go to graduate school and she couldn't get through the test without a full break down. Even so, she's done well in life despite it all.
But the point is that your expression of your experience is only that: your experience. For both my wife and myself, the standardized tests were simply not factors despite the hardships we faced. This is why anecdote isn't a good way to come to real knowledge... it might have been true for you but other's mileage will vary.
A high SAT score is like a ripped body: you can spend money on coaches, and they can help, but ultimately you need to lift the weight yourself.
I will get an sat tutor for my kids for sure and help them focus on it when the time comes. Opportunity requires a prepared mind.
Is it? There are likely many opportunities best pursued through vocational training rather than preparation for college. Given finite learning time, keeping students out of vocational training to improve test scores on a college preparation test is probably not the ideal outcome.
We should stop stigmatizing vocational training, and skilled trade work. The ACT doesn't test for student outcomes. Improving test scores may be counter-productive.
https://www.nationsreportcard.gov/highlights/ltt/2022/
There’s a pretty broad base of evidence for this btw: https://scholar.google.com/scholar?hl=en&as_sdt=7%2C39&q=cov...
"Curious about those who seem to disagree: Am I missing something? Maybe one of my premises or assumptions is bad?"
Is this a calculated likelihood given certain established premises? How do you figure that injury to the physiology that performs cognition has a trivial impact compared to not showing up to school for some time? From what basis do we derive that certainty?
We haven’t done so, have we?
We’d want to be sure, right?
And: In what way were lockdowns unnecessary? It isn’t and it never was safe for children to be infected. There is irreversible harm to children from it at an indefensibly high rate. It is obvious in children in my vicinity. I’m fairly sure my own son has lingering digestive issues after his bout. I may never know because society will not look.
Nonetheless, lockdowns certainly were a bad fit to the main mode of transmission which is infectious aerosol.
There are plenty of pre-pandemic studies about truancy and poor curricula leading to lower test scores. It's practically a tautology that if you aren't going to school and studying for the tests that you will score poorly on those tests (the measure here). Could there be some trivial (in aggregate) physiological impact at play? Maybe, but that's not the biggest factor based on this established research.
"I may never know because society will not look."
It's not that society won't look, but rather that minor issues are deprioritzed. If you have a new disease, there are likely to be tens of thousands unanswered questions around it. It will take decades to get to most of the questions. Some may never be answered or explained. It's unlikely anyone cares enough about an individual with digestive issues (or recurring intermittent swelling after vaccination) to do a study looking at it. If it's sufficiently rare, they may not be able to even identify it as an issue worth looking into.
Pandemic policies are a huge elephant in the room, and it's extremely annoying to see people like you trying to pin the blame on the pandemic itself. School closures were a disastrous policy, full stop. This is the price of that choice of policy.
How can we measure this effect?
How do you know that it’s not the measured cognitive deficit after the infection and only school closures? How was it measured?
We’d want to analyse and investigate the effect, right?
Your first was a legitimate comment for discussion. The recent one doesn't seem interested in a reasonable discussion. The prior commenter makes a solid point and you get hand-wavey because there's not a study supporting it... when your initial comment was just as theoretical (does your study show this is applicable to kids, at what rate, in which countries, what are their test score results, etc).
Note that this is the only data which has been provided in this discussion as far as I can see right now. What made the commenter’s solid point a solid point? Was there data?
"In sum, there is clear evidence for a negative effect of COVID-19-related school closures on student achievement"
Here's one example of shutdown harm.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8481663/
So now who has the only pertianate data? Don't you have a study pointing the school test scores being lower due to physiological impairment?
Please stop trolling this topic now.
Of course it did!
In part: Closing schools was poorly executed, and the lockdown/closure style of mitigation was never based in the scientific facts of the main mode of transmission, so it was always a blunt and stupid instrument.
For the avoidance of doubt: I’m raising the point that we should look at other things too. And interested in thinking about how we would look.
—
I don’t have pediatric studies on cognitive impairment.
I’ve seen quite a bit of literature on the mechanisms that cause it in adults, and only poor reasoning for children being exempt from severe and/or long-term effects. (Is this disputed?) So I’m for monitoring and studying and looking. And thinking about how to do it.
Essentially you can't. Multiple other factors have changed that would affect grades.
If you strictly want intelligence and not just grades, then you'll have to identify kids that had recent intelligence tests before the pandemic and compare them to others. This is problematic because types of intelligence can vary, so some areas might not be covered by the tests. In general, few kids recieve IQ tests and so there's likely to be observation bias with a higher number of kids being tested for some reason. Finally, it's unlikely you could track pre-/post-infection since kids haven't typically been testing every week and mild to asymptomatic cases are fairly common I'm children.
Best case, you can do very small studies on an individual level without control groups, essentially individual case studies. Even if you can, it's not going to answer the questions related to this article/thread.
The effect is not present in countries that did not close schools. The effect strongly correlates with the level of disruption to schools and schooling caused by policy choices. The effect does not correlate with infection levels among that cohort of the population.
There is a strong political desire for politicians who enacted these disastrous policies to avoid blame for them, one strategy is to frame all the policy choices as unavoidable, i.e. claiming that you "had to close schools". Another political strategy is to blameshift all the bad effects onto the virus itself, which is exactly what you're doing.
Your post is a prime example of "JAQing off"; you are "just asking questions", saying things like "shouldn't we study this?", or "there are no studies showing it's because of <insert obvious explanation here>".
What’s up with this “you are like this and that”? What’s with the accusations and insults? What’s your problem? Take it elsewhere!
—
On topic, we’re diverging on the word effect. I am using the word effect to mean “something that the illness is known to do in certain measure” – an effect of the disease on the population. You’re using the word effect to mean effect size in a particular study. It’s an understandable divergence.
Just to make sure: I am not saying. That the grades dropping. Is caused by post COVID cognitive impairment.
I said that it was interesting to think about how you’d design a study so it would control for such an effect.
I see studies about cognitive impairment on the individual level. I see studies about this and that on a population level. I see lots of people saying there’s no connection. I don’t see how they know that.
But we certainly want to do analyses on why grades are dropping.
Yes, and there's tons of that research out there if you care to look for it.
It would be nice to disprove it, right?
You're conflating multiple issues and misapplying the data.
If we're trying to study a problem, we should identify possible root issues. The problem of lower test scores have much better options to investigate. This is why the studies are focusing elsewhere. It's like saying the boat is sinking and we should study the rain instead of the hole in hull or the broken bilge pump.
I’m interested in how to study a known side effect of a disease which has been a global pandemic for two and a half years. How to study it at scale.
I’m not conflating it with the whole host of other reasons for the grades dropping, most of which are obvious factors and obviously bigger factors. The conflation is happening inside your head, somewhere from between your eyeballs and your fingertips. Respectfully.
You are of the opinion that it is obviously a trivial effect and not worth investigating especially. I disagree: I think we should look into it. I do not disagree in the way that I think cognitive deficits from COVID is the main driver of grades dropping.
Our opinions do differ, but not where you are saying we differ.
I think we should make sure we aren’t spilling solvents into the boat as they can degrade the hull. Why don’t you take a look at the bilge pump?
https://www.washingtonpost.com/education/2019/08/30/what-fin...
The top three, which are the only ones I bothered to skim, have quality problems.
The third one seems to be the most cited so I spent the most time with it. It in effect reduces to “if you were hospitalized with COVID” you now have significant cognitive impairments, but from my brief reading there is no “before” so it could just as easily be titled “cognitively impaired people make bad life choices that lead to pre-existing conditions that lead to (exposure to and) hospitalization with COVID.”
Additionally the sample of N at 84k is misleading as only roughly 200 people are in their “significantly impaired” set.
It was a quick read, as I said, and I may have oversimplified, so if I’m wrong, please feel free to respond with corrections.
If – if! – we take it as a valid premise that cognitive impairment exists after COVID-19. In adults. And if we take it as a valid premise that this is exhibited in younger people too.
Then:
I find it interesting to think about how one would study progress or decline in an education system, i.e. to look at that system itself to maintain and increase quality of the system. And: if and how one can reveal or disprove a population-scale effect on cognition due to this particular illness.
Regarding cognitive deficits or impairment, I’m not focusing solely on brain damage as the cause, although it is known to occur. There’s also vascular and endothelal dysfunction, endocrine issues like abnormally low cortisol and blood glucose dysregulation, and immune dysregulation – T-cells in the brain (not good, not usual), and elevated inflammatory cytokines which are known to affect behavior. (This is all unsourced; There exist sources for it.) A lot of this is reversible. It’s not known that all of it is.
I think we ought to look at these issues. In this context. I’m interested in thinking about and discussing the scientific mechanisms with which to look.
There are a number of studies that show if you ask people about the long-term effects of any infection they will report similar numbers to what is being reported about coronavirus. You can take this to support the previous paragraph, or as an instance of placebo effect.
If you’re interested in how viral infection impacts brain function there’s a lot of cool information out there. I would not start in a speculative area like COVID-19. Maybe start with viral meningitis.
Alternatively if you want to stay speculative some people think that mutations related to our rapid evolutionary increase in brain size may have been in part down to zoonotic virus exposure a couple million years ago.
edit: Maybe you would enjoy The Tangled Tree by David Quammen.
> I’m curious why you think any of this should be novel to the coronavirus COVID-19. We are affected by every infection.
This review paper might summarize my inclination to consider SARS-CoV-2 different in this regard: “The Neurobiology of Long COVID” — M. Monje, PhD, Neurology at Stanford & A. Iwasaki PhD, Immunobiology at Yale. Publication status: In Print, Journal Pre-Print Available Online October 7, 2022 — https://doi.org/10.1016/j.neuron.2022.10.006
One paragraph mentions effects in children:
“Similarly, a 2-year a retrospective cohort study using the TriNetX electronic health records network of 1,487, 712 individuals with COVID-19 and a similar number of matched controls with a different respiratory infection found continued risk of cognitive impairment at 2-years from diagnosis (Taquet et al., 2022). In contrast to risk of cognitive impairment, the risk of anxiety and mood disorders normalized within 2 months following SARS-CoV-2 infection (Taquet et al., 2022). This TriNetX study provides some of the only available data about cognitive deficits in children and found in this retrospective cohort study that children did not exhibit an increased risk of anxiety or mood disorder diagnoses, but did exhibit increased risk of cognitive deficits (HR 1.2) and insomnia (HR 1.29) compared to children with different respiratory infections (Taquet et al., 2022). Unlike the adult cohort in this study, the risk of cognitive deficits in children normalized to the level observed in controls with a different respiratory infection by 491 days after COVID-19 (Taquet et al., 2022).”
There’s a lot more to the paper.
Referencing table S9 (for children) in the appendix/supplementary there are only four categories of illness where the p values are still possibly meaningful after correction: cognitive deficits, epilepsy or seizures, intracranial hemorrhage, insomnia
The two common conditions, cognitive deficits and insomnia, are only significant because of the arbitrary follow-up period selected of six months. If you examine the charts shown in Figure S1 in the same appendix, you can see that cognitive deficit has no long-term effect, and that actually COVID-19 seems to be protective in the longest term (as is the case with anxiety). This to me is suggestive of the fact that social aspects/expectations are dominating the reported symptoms for these common experiences. None of the symptoms, even insomnia, are quantified.
I can't find if/where they break it out by sub-classification but they seem to lump together real measurements of cognitive deficit with self-reports of brain fog. Similarly anxiety is not well-quantified.
For intracranial hemorrhage and seizures the effect is more pronounced, but as you can probably infer from the pixelation of the charts in S1 for these two, the sample sizes here (children with these symptoms), are tiny, and so the results are underpowered. If I wanted to keep looking this is the only avenue I'd consider.
As far as effects in other populations, they compare e.g. sinusitis and laryngitis on equal footing with lung infections resulting in hospitalization and possibly ventilation for COVID-19. That to me suggests there is the problem of confounding based on the impacts of reduced perfusion and oxygenation of brain tissues due to actual major lung infections that is more present in the COVID-19 data than the control dataset. A respiratory infection for this purpose ought to be something that can seriously impact perfusion and brain oxygenation -- they don't seem to have realized that and instead used "respiratory infections" without further qualification.
The Taquet et al study is the best study I've seen on it and I'm still totally unconvinced this is anything beyond hypoxic brain injury (and only minorly so -- in most survivors it seems to be primarily fear, hypochrondria or provider nudging).
The passage on the effect in children is a tiny part of the whole work. From the introduction:
COVID-19 may affect the central nervous system in (at least) six main ways (Figure 1). First, the immune response to SARS-CoV-2 in the respiratory system may cause neuroinflammation - increasing cytokines, chemokines and immune cell trafficking in the brain and inducing reactive states of resident microglia and other immune cells in the brain and brain borders. Second, SARS-CoV-2 rarely may directly infect the nervous system. Third, SARS-CoV-2 may evoke an autoimmune response against the nervous system. Fourth, reactivation of latent herpesviruses like Epstein-Barr virus may trigger neuropathology. Fifth, cerebrovascular and thrombotic disease may disrupt blood flow, disrupt blood-brain-barrier function, and contribute to further neuroinflammation and/or ischemia of neural cells. Lastly, pulmonary and multiorgan dysfunction occurring in severe COVID can cause hypoxemia, hypotension and metabolic disturbances that can negatively affect neural cells. It is important to recognize that these mechanisms of nervous system injury are not mutually exclusive, and a combination of mechanisms may occur in some individuals, with varying frequency and timing.
It has been shown that there are many markers for Long COVID, many of which present as neurological and cognitive issues. Two markers have been shown to be reliable and consistent: Patient reports – and unusually low cortisol levels. The other markers are things such as elevated inflammatory cytokines. Fear and hypochondria and provider nudging do not manifest in those ways. Also: Provider nudging? I have heard of nothing but universal and unscientific dismissal of anything COVID-related when people go to health providers for long-term issues. I don’t think that’s it.
I can’t see how this could be anything less than real, and how children could possibly be exempt from the effects. And we aren’t looking because we think we know it’s nothing.
Just look at Harvard. Since they don't require any additional essays applying there is now literally just a button press away - wouldn't you?
I never understood the pressure for highest SAT average. There are way more people with lower scores who want an education, and they don't want to go to a school where they have the lowest "aptitude" or preparation.
If everyone's acceptance rate went down, then no one's ranking went up.
(if you define "minorities" to exclude the country's second largest, and fastest growing minority group)
I define Nevadans as a minority in the U.S. It is obvious that this standardized test discriminates against this minority.
Don't even get me started on people taller than two meters. ACT won't even show that data because it is so telling.
The thread from last week showing that "Asian" kids' SAT scores crushed "White" kids' scores in Michigan was pretty eye-opening alongside the current Supreme Court cases regarding admissions and Asians.
One of my favorite opinion pieces about the SAT, from a few years ago, came from a black guy who grew up in rural Mississippi or somewhere like that. He argued that the SAT was the only thing that gave him a level playing field, where he could prove his equivalence, in college admissions against kids from elite private schools and wealthy public school districts. His position made sense to me.
The majority of people aren't happy with having standardized tests because they block kids from feeling they have a shot at the top schools. But of course that's their function -- to sort.
So, public pressure calls for dropping objective measurements so that people feel better about it... but they aren't doing better. And muddying the waters to say that everyone should have an equal shot at going to a top school -- for reasons other than academics and using the resource efficiently. And the kids just stagnate in performance and everyone feels good about it, until you get kicked in the ass by reality and you find other countries who don't care so much about making people feel better are actually passing you by.
But into what categories are they sorting? What if they are mostly measuring who is good at taking a test?
At my kids high school, they were in a program where they got to write a whole bunch of practice tests. From test-to-test, they did better and better. Some of the tests were close enough in time that their knowledge didn’t significantly change, but their scores improved. I attribute that to learning answers to questions that pop up frequently and to being less anxious after writing the nth practice exam.
Both of my kids did great at the tests eventually.
I recall with AP History back in the 90s, DBQ questions required some training to the form before sitting for the exam.
Or, their function is to evaluate macro trends in education.
They are definitely different experiences, but nearly all drugs have different delivery mechanisms with different onset times and intensities and long-term consequences of using them. The crack-vs-powder cocaine difference is nowhere near, for example, the difference between smoking and injecting heroin. But that is not considered a different, more dangerous, more brutally punished drug if you inject it.
And more importantly the subjective experience of the delivery doesn't matter, the origin of the distinction is in the historical record. Crack cocaine was given different, much harsher, sentencing guidelines for unambiguously racist reasons, that then needed to be justified in the public consciousness.
Should I never use crack cocaine as a metaphor ever again?
More seriously, I'm not sure that we should worry so much that students didn't learn what we wanted them to learn for the last two years. Maybe they learned something else that might turn out to be beneficial someday.
On the other hand, I'm not sure how many "influencers" our society actually needs.