“In most machine scoring states, any of the randomly selected essays with wide discrepancies between human and machine scores are referred to another human for review”.
And “between 5 to 20 percent” of essays are randomly selected for human review.
So the takeaway is that if you’re one of the 80-95% of (typically black or female) people who the machine scored dramatically lower, but are not selected for human review, your education future is systematically fucked and you have no knowledge of why or how to change it.
Absolutely reprehensible. Anyone involved in the creation or adoption of these systems should be ashamed.
The thing is, you could be similarly screwed by a biased human whose grading is not checked by a less biased human.
At least the machines offer the following hope: even if unbiased humans are rare among paper-grading teachers, those humans can be used to train the machines, so then bias-free or lower-bias grading becomes more ubiquitous.
Basically, the system has the potential for systematically identifying and reducing systematic bias. A computer program can be retrained much more readily than nation-wide army of humans. Humans can be given a lecture on bias, and then they will just return to their ways.
AI has a lot more potential for bias than humans. It depends on the input data which is likely heavily biased based on other data set results like face detection. It will only amplify any small bias present in the data.
So will humans. With AI, the model and training data are auditable and, if necessary, modifiable. You can't audit a person's life history and you can only really modify a grader by replacing them, and it's much less socially damaging to retrain a model than to fire an employee.
AI certainly has a lot of potential for bias, but claims that AI bias is somehow worse than good old human bias always seem shoddily supported (Note I'm not claiming it's untrue. Just that it's never been shown to my satisfaction, which is not surprising given how quickly AI is changing. It may well be true.)
> but claims that AI bias is somehow worse than good old human bias always seem shoddily supported
Well, AI bias can combine sampling bias with human bias. Like say we train the AI with the output of only 10 human paper graders, all chosen from the same school district.
Due to the sampling bias, that data could create markedly more (or less!) bias than the entire population of human paper-graders.
The resulting AI will ideally mimic those 10 humans, though; it shouldn't show more bias than that group. If those 10 are flaring racists, and grade accordingly, the AI will be the same. (In fact, we hope that it will be the same, if the algorithm actually works in mimicking human grading.)
It's amazing to see how the general opinion of CS people has completely shifted in the last few years from "algorithmic scoring is important in removing the bias from human graders" to the exact opposite.
That was the hope, but all the most effective methods suffer from data collection bias and the studies show that makes them worst than implicitly biased humans.
If we can quantify the bias in the machine, that gives us an opportunity to close the feedback loop and control the bias.
The bias comes from the human-generated training data in the first place; the machine isn't introducing its own. For instance, the machine has no inherent concept of disparaging someone's language because it's from an identifiable inner city dialect. If it picks up that bias, at least it will apply it consistently. When we investigate the machine, the machine will not know that it's being investigated and will not try to conceal its bias from us.
On the other hand, eliminating bias from humans basically means this: producing a new litter of small humans and teaching them better than their predecessors.
>So the takeaway is that if you’re one of the 80-95% of (typically black or female) people who the machine scored dramatically lower
It's quite funny how are some people manipulated to think that society and especially education is somehow biased against minorities or women when opposite is true[1][2].
Did you read the article this thread is discussing? It cites multiple specific results wherein minority and female writers were underscored by the machine graders compared to human ones.
The idea that black folks and females score dramatically lower on anything is a commonly held racist opinion that is foundational for the continuation of institutional oppression. When you assume the worst about people it ends up hindering their personal progress.
That's not what the article is saying, at all. There ARE general differences in style and word choice between minority groups or women and the average white male in writing. The corpus of training examples used in making this AI grader are at least biased towards the average white male. When the AI grades an essay harshly, it is not saying "this essay by a black woman is bad writing", it is saying "this essay differes from my training set by an aggregate score of ____", then sorts those results and (maybe) applies a curve.
An essay could be different from the reference standard because the standard is an example of good writing and the essay is not. Or it can be different because the author has a cultural, regional, gender, or developed background that imparts a different style than anything in the training corpus. Mistaking the two is very, very bad.
> It's quite funny how are some people manipulated to think that society and especially education is somehow biased against minorities or women when opposite is true
I think you have misunderstood the parent, who is asserting that the machines scoring these essays typically give lower scores to members of these demographics. This does not, by itself, mean the entire system is biased against those groups.
Whether or not the groups are, overall, systematically benefitted or harmed is not relevant to the injustice this article says exists.
That link talks about how black women are attending college in highest proportion due to extremely recent growth.
It also says that despite their recently high educational attainment, black women are still underrepresented by a factor of 2 in private sector jobs, as compared to their college graduation rates.
>Anyone involved in the creation or adoption of these systems should be ashamed
That's the problem - there is seemingly no shame these days. People involved "saved time and money", got paid and that's it. "If I didn't do it someone else would" and all of that.
Unlike a multiple choice test where the primary audience is automated graders, the primary audience for an essay is other humans. If even Google and Facebook with their billions of dollars and billions of posts worth of data, still cannot always understand the intent and purpose of written content, what hope do these algorithms have?
If it is cost-prohibitive for every essay to be graded by humans, then they should be dropped from the tests. Otherwise, we are missing the whole point of essays which is to communicate effectively with another human, not just match certain text patterns.
Finally, but then none of these new test takers will know what it feels like to get near perfect scores on the other sections of the test but then completely bomb the written portion and ruin your overall score.
> Otherwise, we are missing the whole point of essays which is to communicate effectively with another human, not just match certain text patterns.
I agree, this is traditionally the purpose of an essay. But to play devil's advocate, consider the rising number of people who are writing SEO or ASO content which is actually targeted at machines.
While this is already terrible, I’m aware of a few project that are trying to do the same with scientific literature. Basically they are trying to train models for scoring literatures based on their quality, novelty and what not. At the current rate and state of AI, I cannot ever imagine this is going to work.
It was a few weeks ago that someone shared “The Dark Age of AI” on HN [1]. I think we are promising way over what Drew McDermott thought we would not going to promise. This is to the extend that we are applying AI on assessing Art, Creativity and even quality and novelty of Science, something that in a way we don’t even understand (or trying to understand) ourselves at the time that we are publishing it.
I imagine that any student that experimented with the form of the essay or wrote an exceptionally well argued piece in simple language would not have their test graded appropriately either.
Any essay writing test which could be adequately graded by a machine is not testing anything of value.
Edit: I’ll further add that as soon as people’s careers depend on a metric, the metric becomes useless as a metric, because it will be gamed and manipulated by everyone involved. Almost nobody involved is incentivized to accurately measure student’s writing ability.
I think machines could be valuable in giving feedback on writing, like that grammarly.com.
A lot of what students write is actually garbage from that point of view. Even if they happen to have a good basic idea about what they want to say, the point of essay writing is to master the mechanics of expression so that you get the idea across effectively.
Whether the student has a brilliant idea isn't even so important, and it wouldn't even be fair; imagine if high school computer science expected students to turn in a best-selling app for a term project. Not everyone can come up with something brilliant to say; and even relatively mundane lines of reasoning can be given a good treatment in writing to develop the skill.
I remember when I had essays graded in school, a lot of the comments were low-grade fluff like "run on sentence", "wrong word", "faulty parallelism", "missing colon before 'for example'" and such points having nothing to do with the content being original, well-considered and well-argued. That sort of thing might as well be done by machine, at least as a preprocessing step to improve a student's rough draft.
Almost nobody involved is incentivized to accurately measure student’s writing ability
It's the same reason you see keyword posters in math education. "Together" means "plus", that kind of thing. It's completely worthless, except for one-step problems, and even then it doesn't always work. What is happening is collusion between teachers and testmakers. You can't teach understanding, but you can teach test-passing techniques because the way the test is set permits this.
You see the same thing here, in English you can get away with not teaching quality writing if you teach techniques to score well.
If I have 3 left shoes colored blue green and red, and you have 2 right shoes colored black and white, how many pairs can we make if our lefts and rights are put together?
I feel like the mistake is assuming that essay writing is about the content. It's just a thing to give the student something barely non-trivial to write about.
When your essays are graded they're marked down for mechanical and wording problems. There's really no point in trying or grade 'good ideas' on a subject piece you had maybe 10 minutes to skim.
That's a travesty, and you know it because when the kids are in college and they have as much time as they like to write their assignments they all use the wrong words and then misapply them.
There is value in the ability to produce correct English 'off the cuff'. You could argue essays are the best way to get students to produce off the cuff written text. Hence, it makes some sense to ask students for essays, and then judge those essays only for form.
However, it is rather important that students know their essays are not judged as essays, but only judged on the content. Otherwise you teach students that form trumps content in essays.
When judging an essay as an essay correct English barely matters. What matters is how convincing you are, and how interesting of a read the essay is. This is a great skill to have, and testing it also makes sense. Really though, we should separate these two forms of testing.
This is my first time learning that AI-graded essays are a thing. Am I the only one who thinks that's insane? I feel like you'd probably have to have an AGI to meaningfully evaluate an essay.
I work in AI, and was very surprised when I heard about this (a few years ago). I don't think anyone who works in the area thinks the tech is ready for this kind of deployment. There is research on the subject [1], and NLP systems can do better than baseline methods, but the error rates are still pretty high.
A thing you quickly find if you try to download off-the-shelf NLP tools and apply them to anything is how little is reliable at all, unless you can constrain the domain. Even basic topic identification only works with low error rates when constrained to something like NYT stories, or PubMed abstracts, not arbitrary text by arbitrary writers. And I would bet ETS is using worse tech than research state-of-the-art.
Hmmm. I also work in AI, in fact professionally in information retrieval and NLP. I disagree strongly with what you say. Basic topic summarization and keyword / named entity extraction on unstructured sources of text works reasonably well. It’s easy to modify BERT and GPT on smaller problems, language classification is borderline totally solved by extremely easy to train neural network models.
I still agree that automatic essay grading is beyond the reach of SOTA NLP models today, but youmake it sound like virtually nothing can be done in a production-grade manner that solves real world unconstrained NLP problems. This is manifestly false.
It's completely possible I'm not fully up on recent progress, especially since a bunch of stuff seems to have moved in the past 6 months. But I haven't seen any general models that can solve open-domain problems, without specifically retraining on each domain. Do you have any pointers? E.g. a single pretrained BERT model that can reliably extract topics from: tweets, paragraphs from 19th-century novels, mathematics journal articles, and Wikipedia articles? All the systems with very low error rates that I know of target one specific domain. The last time I looked into sentiment analysis (a year or so ago), it wasn't even that great on many individual domains, e.g. it would get tripped up by sentences from novels that used "negative" keywords in a humorous or ironic way.
In production problems that I work on, we don’t even really use things from within the past year. These problems are just incredibly well-solved with fairly vanilla LSTM networks from 2-3 years ago. Enough so that while it’s probably premature for fully automated essay grading, it’s not _crazy_ to make a product from models trained to solve this problem.
I have a grant where were are doing just that. Implementing more or less SOTA research using fairly vanilla LSTM networks from 2-3 years ago (primarily Taghipour & Ng) to provide low stakes feedback to students on their essays in one of our teaching tools at Purdue. It’s based on research using the Kaggle ASAP database and we have found it to be pretty accurate across a variety of domains in early testing. Though some essay prompts seem to do better with CNNs vs. RNNs. I doubt many of the systems in TFA are based on LSTMs or neural nets at all. They are probably doing regression on hand-crafted features.
Very interesting. Are there any meta-analyses / reviews that summarize progress in this area? Would it be possible to share your grant proposal -- I'd be curious to get an idea of what is being attempted.
It's an internal grant and I'm not sure I'd be allowed to share it. We are adding AES to our peer-review app. Currently as an additional "grader" to the peer reviews since that's what the PI requested. Since the tool allows unlimited submissions until the review date, I hope to add it as a "pre-flight" estimate to give students a chance to get a rough prediction of the score they will receive and a metric they can use as they revise until the due date.
I'm not aware of any meta-analyses myself. I have been keeping up with the ASAP competition and various attempts to improve on the initial systems for a number of years. The two papers I believe are having the most success are [1] and [2]. [3] seems promising for balancing the opposing forces of high accuracy for true positives and the risk of false positives via adversarially crafted inputs.
I'm also vaguely aware of research happening around extracting features from neural nets. I'd love to be able to help students understand why the system is predicting a particular score.
You've noticed though that the AI con is on. This damages your work as people get burned and will bring about the second "AI winter"
People making big decisions with a lot of money around computing know nothing about it and are marks for con-artists. Think big consulting firms selling to senior public servants in washington. "For a successful technology reality must take precedence of public relations." But reality just gets in the way when conning a mark for a successful snake oil sale, right?
Call it out, publically, cite your credentials. Encourage colleagues, your competition and everyone with a clue to pour scorn on whoever is selling this evil, toxic waste as drinkable.
In a forum of CS people I'm surprised this is one of the top opinions. Our field is full of super surprising results like this -- that you don't have to actually understand the text at beyond basic grammar structures to reasonably accurately predict the score a human would give it.
Like this kind of thing should be cool, not insane. I mean wasn't it cool in your AI class when you learned that DFS could play Mario if you structured the search space right?
Like how I felt when I was given low grades for my ugly handwriting. It was stupid to grade it, but it guaranteed that I will never get a top score on any literature class.
Adding further bias against the underprivileged is not "cool". Implenting this while avoiding publicity or providing a means to publically audit the results is doubly not cool.
It is fine to play with "cool" techniques when you are doing consequence free stuff like playing Mario. When you are creating systems that have significant and long term effects of people's lives a different standard applies.
When I hear a result like "software which understands basic grammar structures can predict what grade a human would give an essay" I think my views are roughly:
* 5% - cool, we could make a company that grades essays
* 15% - cool, we could make a company that grades essays and sell our source code to the test-prep industry
* 80% - fascinating, it sounds like the exam designers need to reevaluate what they are trying to measure with essay questions
Whatever we decide to measure, it needs to scale to millions of essay responses each year in a way such that scores are consistent across entire states or countries. With that in mind I'd imagine it's difficult to do much more than grade on grammar and basic semantics.
And if you succeed you will simply be measuring an uninteresting but manageable subset of the problem which will then become in some people's eyes the definition of the problem.
Education is supposed to be about teaching people to think, to give them the tools with which to do it, to be able to evaluate, criticise, invent, etc.
I came first in English for my school, many moons ago. Leading up to the finals, I regularly finished ahead of the hard core the English essay people, generally to my amusement. My exam essay responses were generally half the length (sometimes even shorter) than the prodigious writers. Although I've an ok vocabulary, I always made sure I made the right choice of word to hit a specific meaning, rather than choosing words with a high syllable count.
I'd find it highly interesting to see what kind of result I'd get using an automated system.
Why?
Because, I once asked a teacher (also an examiner) why I got good grades above the others, and the answer surprised me: my answers were generally unique /refreshingly different, to the point/ not too long and easy to read.
I suspect with this new system, I'd be an average student. It'd also be interesting to find out, several years down the road, if the automated system could be gamed at all -- I suspect it could, and teachers would help students 'maximise' their scores as a result of that.
It seems plausible that, under this system, you would eventually have learned to write longer essays.
To my mind, that would be a school teaching you to be worse.
In fact, throughout the article I kept being surprised by the idea that long is good. When writing, I tend to prefer being brief.
"...that you don't have to actually understand the text at beyond basic grammar structures to reasonably accurately predict the score a human would give it"
That only really shows that the humans they're training on are terrible at grading essays.
This is sort of like discovering the Excel spreadsheet at the heart of a system responsible for handling hundreds of millions of dollars of transactions for your bank.
Yeah, it's cool, but what about your savings account?
Teaching human-human communication by removing human inputs and having computers decide about quality... call me a skeptic. I feel bad for the students. Essay grading was bad enough before this
Narrowly for grammar however - is even that a good thing? It probably helps scale grammar help to more students, but if those tools became ubiquitous in grading and editing then unique voices would just disappear and a lot of potentially “great writers” might choose different careers because the machines don’t like them
This problem is a first class demonstration of the difference between "can we?" and "should we?"
The fact that it's being implemented in society is insane because anyone who is paying attention to the state of AI today already knows how it will go wrong: without reading the article I already guessed that it systematically discriminated against certain demographics. Which was in fact what the article claimed.
It's interesting that it's possible to predict what the scorer would decide, but the moment you actually implement it is when all of the known problems become relevant, and the intellectual wonder must take a backseat to the human problems.
I remember when I was in middle school 16 years ago, my English classes would have us submit some of our work to a web app. It would then grade the submission. I remember this distinctly because I asked my teacher to intervene on at least two occasions. The app failed to recognize the words "squirrelly" (as in "That guy in the corner has been acting squirrelly.") and "defragment". My teacher decided to subvert the app's recommended grades because she, as a human, understood the intent of my use of those weird words.
I'm surprised people are surprised by it. I guess it just hasn't gotten talked about it a lot? When I took the GRE in 2011 the rule was that my essay would be graded by one human and one automated grader, and a second human would become involved if the computer and the human differed by one point or more iirc.
Maybe nobody really makes a big deal about it because it is pretty much irrelevant anywah. Applicants provide a letter of intent that the grad dept people can, y'know, actually read for themselves, so I think unless you totally bombed the writing section nobody cared.
I totally agree that "AI" grading is totally bullshit. But, I also have plenty of experience teaching/TAing large courses, and after reading too many essays they all become semanticically saturated meaninglessness. One can not help but skim them, and grade according to a few quick heuristics. At that point one tries to be self-consistent and defensible in one's grading, but careful consideration is right out. I suspect state graders are dealing with way more than 100 essays per person and are probably on a tight schedule too. It's quite possible that a ML model is better than an exhausted human grader, as their cognitive strategies are mostly identical.
The solution isn't to do a better job at grading 'meaninglessness' but to stop requiring the production of it in the first place.
One major problem with algorithmic approaches, whether automated or not, is that they become the definition of good in the context and therefore become something that cannot be argued against. And of course it makes 'teaching to the test' an even more likely outcome.
If I were a conspiracy theorist I'd attribute this to wanting a dumbed down population. Unfortunately I think it is probably the other way round, the population is already dumbed down and a belief in AI unicorns is the result.
As Aristotle said to Alexander: 'There is no royal road to geometry', and so it is with education; it's hard work for both the student and the educator and no amount of AI/ML/algorithmic snake oil will change that without also changing the meaning of the word education.
We had this in my school for 8th and 9th grade so 2008-2010. We had to type the essays in class and submit by the end of the hour. I would only get maybe 3 paragraphs in before time was up because I was trying to build a strong argument for the prompts. Despite that I would usually get 3-4/6 and my teacher said she would read the essays and regrade but she never actually did. My friend literally copy and pasted the pledge of allegiance 20-30 times and scored a perfect 6/6. Later we found out if you repeated the words in the writing prompt you would get a guaranteed 5/6 and with a high enough word count you’d get 6/6. The essays were all bullshit and just a way for the teachers to get an extra free period once a week.
> I feel like you'd probably have to have an AGI to meaningfully evaluate an essay.
So the reason this isn't the case, is because there are very simple metrics that tend to highly correlate with essay quality. It doesn't mean the grading-bot is actually evaluating essay quality. It's just looking for properties that are statistically associated with good essays. Remember, at the end of the day as long as the bot's ranking is close enough to the human grader's ranking, nobody really cares about the internal logic.
A very straightforward example is spelling mistakes. People who make spelling mistakes aren't necessarily bad writers. And vice versa, there may be great speller who can't write for shit. But by and large the people who spell poorly also tend to write poorly. Easily detectable grammatical issues, like misplaced modifiers, subject verb disagreement, or inconsistent tense, are also correlated indicators.
A very simple metric is essay length. Especially if its a timed exam. Good writers tend to have verbal fluidity, with words easily flowing to paper. They don't struggle converting thoughts too sentences. So they tend to end up with the most words written down within a fixed time period. By and large the longer a timed essay is, the more likely that its actual quality is high.
Grading bots basically rely on these statistical relationships. They're not measuring anything intrinsic to good writing. But at the end of the day, their student rankings are usually pretty close to that of a typical human grader. In some cases the bot will have a closer ranking to a random human grader, than two random human graders will have to each other.
The biggest flaw here is Goodwin's law. When the test takers become aware of the kludges that the bots use, they can exploit it. For example just dump a bunch of verbal diarrhea with as many correctly spelled words as possible. But even then it doesn't really hurt the bot's ranking accuracy too much. Because the kids who do the most test-prep and learn all the tips and tricks, are usually high-achievers who do well on essays anyway.
Strongly (but respectfully) disagree with a lot of this!
This is related to current fairness-in-AI discussions. In many cases the basic problem is ML systems leverage correlations for making causal decisions. Here, there is a huge ethical difference between scoring a person based on "is this a good essay" and "do the features of this essay correlate with features of good essays". Just like there is a huge fairness and discrimination difference between "is this person qualified for a loan" and "do the features of this person correlate with features of people who qualify for loans" (algorithmic redlining). Your last sentence has a big discrimination/fairness issue also, since you are testing even more for parental income and parental free time.
Your last paragraph, and particularly the last sentence, epitomizes what is wrong with your whole thesis: the ultimate goal of the testing (and education itself, for that matter) is not to find people who can "do well on essays"; it is to develop analytical thinking.
That assumption lacks justification when the scoring does not actually measure analytical thinking. Any statistical evidence for it is suspect as a predictor of future outcomes when a high score can more easily be gamed than 'honestly' achieved.
Scoring is not the point here; the analytical thinker is gaming the test to pump the score, thus proving they are an analytical thinker. Not a statistical argument; a suggestion that the screen works, when it is abused. Because it is abused.
>Remember, at the end of the day as long as the bot's ranking is close enough to the human grader's ranking, nobody really cares about the internal logic.
This isn't true at all. Imagine you got a B or C on an essay that a human would have given an A to because you wrote it concisely and in plain language, or because you used language that's statistically correlated with being black. Does the fact that this is rare console you? "Sorry, but it's usually very close to the human grader's ranking." Close enough isn't good enough when you get the short end of the stick. "Sorry, you aren't going to get to go to the college you wanted because you use language statistically correlated with poor writing." Or just because you're different, so the statistical correlation doesn't apply to you, you filthy outlier. Just because it's a rare event doesn't make it okay.
In adulthood, this is like hiring or firing for work statistically correlated with good work. Remember when amazon rolled out the resume scorer? [0] Sure it was biased towards women, but it was close enough to human scores, so who cares about the internal logic?
>Grading bots basically rely on these statistical relationships. They're not measuring anything intrinsic to good writing.
At the end of the day, our goal here is to measure good writing. If the bots aren't measuring anything intrinsic to good writing, we shouldn't use them.
The problem with the bots is that while they average agreement with the humans they can produce very different results. Fine if you're seeing how a school is doing, horrible if you're testing how a student is doing.
It is absolutely insane. By no definition does the system understand what is written.
You could ask a student to write an essay taking a firm opinion on some subject, and they could change standpoint every paragraph and there's no way these systems would know.
If I was a student I would be extremely offended at people wasting my time like this.
Because a (likely unsophisticated) algorithm is grading the essays, there's probably a deterministic method to do score well.
This seems like a terrible idea.
It's not a stretch to imagine the opportunity for nefarious behavior this allows - think of the recent college admission scandals, and how happy they'd be to have a guise of algorithmic indifference'.
If used long-term, it could offer a big advantage to the wealthy in other avenues. Another hypothetical, probably not far from reality: the algorithm becomes solved (almost or completely) by some premier 'tutoring' company. Said company can charge a pretty penny given its stellar track record, offering yet another hidden advantage to the wealthy/elite.
There's definitely a deterministic way to score well on HS level proofs. Also, I think you are overestimating the requirements for an essay on a standardized test.
My mother worked grading standardized tests. It was a hellish job for many reasons (limited breaks, etc.)
One question she had to grade was essentially, "What's something you want your teacher to know about you?"
It was an essay answer, and she was supposed to grade it on grammar, etc. Just the mechanical aspects of writing. (The real question explained the details more, but that was the core of the question.)
She saw answers that would make you weep.
"My daddy touches me."
"I haven't eaten today. I don't know when I'm going to eat again."
Stuff like that.
And my mother was going to be the only human who ever saw their responses. Their teacher had no chance to see their responses, just my mom.
So she goes to her supervisor and asks, "What can we do to help these kids?"
The supervisor said there was nothing you can do. Just grade the answers.
School contractor is someone who is hired by the school district on via a contractual relationship. Think temporary teachers, or custodian staff. It’s not a transitive relationship to every employee of every company who has some sort of contract, however small, with a school.
So you're arguing that only individuals can be a contractor? That wouldn't make much sense, not only because such relationships are rare in schools. Most common are contractors that have been outsourced something like food service. The law would make no sense if it included practically no one. It would mean a company that provides, say, temp staffing within the school, and those temp staffers saw abuse, they too wouldn't be required to report. I have a hard time believing a court would rule the definition to be so narrow. Both the common language understanding of the term and legal literalism would point against that. There's no transitive property here. We're not talking about contractors hired by contractors hired by contractors. We're talking about a contractor and its employees. There is no way for it to exercise this reporting requirement save through it's individual employees.
It takes a five minute phone call with the company's legal department or a warrant to find out who the kid was. Either way it would need to be escalated to involving law enforcement.
Collect or photograph all the evidence, record every conversation with supervisors, escalate as much as possible internally, then contact local police, and at the same time go to the media. Don't quit, but if necessary let them fire you and then sue. None of this is easy.
When escalating I'm sure it'll be effective to say it'll be an interesting story for the news and how the incident is being blocked by supervisors that encourage child abuse.
> With a written accusation from a child? Is that enough to get a warrant to force the company to release the demographic information?
Yes, she should have absolutely went to the local police. A child's first hand account in writing of child abuse and neglect is slam dunk evidence to secure a warrant to link the essay ID to the individual child.
> Everything she took in and out of there was monitored, too. So it's not like she can go to the Xerox, and walk out of there with a copy.
Doesn't matter. She could have went to the police herself as a witness. That alone would be enough for a probable cause warrant to retrieve the essays.
It is very sad she saw these signs of abuse and did not report it.
Do you suspect this is a one off mistake or do you think you make repeated mistakes of this kind? I'm always worried I'm doing some particular thing, interpersonally, repeatedly, but I fail to notice.
I meant that your self-consciousness and constant worry is probably harming your social interactions more than anything. Or not. It was for me in the past.
Some of these will be 100% true as well. But don't make the mistake that there are no kids who go for shock value or are wantonly manipulative when they know it can't come back to them.
So how many are true and how many false? I have no clue. Literally none. And no it doesn't make me feel any better about the screams of existential agony even if that were a low percentage. Could be high too.
> But don't make the mistake that there are no kids who go for shock value or are wantonly manipulative when they know it can't come back to them.
In the US, school funding is based upon standardized test results, and bad results can shut a poorly performing school down.
It's drilled into every kid's head that these tests are very important, super strict and if they accidentally mess up, it can ruin their academics, because retesting and regrading are expensive.
As a kid I would go out of my way to fail those tests. The whole curriculum was designed around them, meaning that even if we did score high, any funding gains would just be put towards training us to take the test.
I thought the state was holding the school hostage, threatening to cut funds or shut them down if they ever stopped. We never learned anything about civics or American history. Until I was out of highschool, content regarding atrocities like slavery and the trail of tears was not on the test and that was enough to whitewash the whole curriculum.
Standardized testing is to the U.S. what lead waterpipes were to the roman empire.
I don’t know about calling it manipulative. I remember taking the ACT, and struggling to plan out one of my essays. It was something like “tell us about a book that inspired you”. So I changed details about the plot so it all fit nicely and was easy to write. I can see something similar here, where someone takes on a persona when writing in order to effectively communicate.
That's entirely fair. Manipulation is kind of what a writer does yet the word manipulative has perjoritive connotations. Many types of writing don't have literal truth as any kind of pre-requisite. Others make a pretence of literal truth to achieve greater effect then basically lie, many autobiographies fall into this trap to some degree. All these things. Differential empathy. Data quality matters.
This is absolutely the case. In fact, my SAT prep class taught us that the factual veracity of our essays is irrelevant. Essay scores are almost entirely correlated with essay length as long as spelling, grammar, and basic paragraph structure (intro, body paragraphs, conclusion) is followed.
Of course some kids are manipulative, going for shock value, continuing an "in-joke", or just plain trolling. But would a teacher just look the other way, or would they talk to the kid? What would you want for your kids? This is why teachers assigning homework like "what do you want your teacher to know about you" and then not even seeing it is dehumanizing.
For the not eating, it's pretty easy to get data. It's like 1 in 5 children live in food-insecure households in the US and maybe 1 in 20 of those very insecure, so not eating before school provided lunch is common enough that if you're grading tons of papers you'll run into kids like that.
Food Security Status of U.S. Households with Children in 2017
Among U.S. households with children under age 18:
84.3 percent were food secure in 2017.
In 8.0 percent of households with children, only adults were food insecure.
Both children and adults were food insecure in 7.7 percent of households with children (2.9 million households).
Although children are usually protected from substantial reductions in food intake even in households with very low food security, nevertheless, in about 0.7 percent of households with children (250,000 households), one or more child also experienced reduced food intake and disrupted eating patterns at some time during the year.
When I was a high school student, we had some state administered test in health class that tasked us with analyzing advertisements for liquor and tobacco and seeing if we could recognize harmful behavior that the ads might be promoting. This test had no impact on our class grade...
..which means students wrote whatever the hell we wanted. I was assigned a Captain Morgan (rum) ad. I wrote that the ad was glorifying maritime piracy and was likely responsible for pirate activity in Somalia.
False accusations can actually be the result of prior abuse. They may substitute one person for another. Or do things as a result of mental illness caused by abuse. Kids think differently to adults and may behave inexplicably. And unfortunately that means that an abused child is a terrible witness.
I was abused as a child (not sexually however) and I can attest to this. Many of my memories are highly charged and don't really hold up - they're very confused. Some of the scariest stuff that happened to me I don't even remember, and my siblings have had to let me in on it (and they were even younger at the time).
As a child you're really not prepared for the concept that your parents are treating you badly. So that realization doesn't come until much later.
The US has federal child abuse mandatory reporting requirement laws which include teachers and school staff and personnel, as well as additional state requirements which vary but include, for 11 states, faculty, staff, and volunteers at public or private higher education institutions. Computer and IT professionals are also covered in cases.
Faculty, administrators, athletics staff, or other employees and volunteers at institutions of higher learning, including public and private colleges and universities and vocational and technical schools (11 States).
I'd believe that ML could spot abuse that humans miss pretty well from signals like non-overt references in homework and school records, if one could come up with an adequate training set.
Much more likely than teaching ML to score reasoned and creative activity in any reasonable way.
Machine learning techniques are going to be absolutely awful in detecting something like this, the reason being it's exceedingly rare (at least I'm guessing it is; if we're talking about child sexual abuse by one's own parents, it sure sounds extremely unlikely- but even child abuse in general is probably rare [1]). Machine learning systems are awful at identifying rare events. Like the OP seems to suggest, the false positive rate would most likely be very high.
"Spooky" machine learning results happen when a correlation is abundant in a dataset [2]. Otherwise, machine learning techniques will probably miss it altogether.
[2] The archetypal spooky machine learning story is surely the one about Target sending baby item coupos to a girl in high school before her father knowing she was pregnant:
Humans are awful at rare events and vigilance tasks, too. That's part of why we're seeing machine vision and machine learning starting to outperform humans in e.g. grading radiology screening scans.
The total incidence of child abuse of all types from infancy to adulthood is on the order of 1 in 3. This is not terrifically rare-- it's of higher prevalence than pregnancy and of positive screening events.
A much bigger concern is non-causative correlations. It'd be pretty easy to train ML to be racist or look for e.g. indicators of class, which are correlates of abuse.
As to false positive rates-- you can pick your false positive rate to be whatever you want it to be, by twiddling the threshold for a positive result. I'm not sure false positives are of that great of a concern, if the output from a system is a notification to school administrators that they may want to keep an eye out for this student.
> The people whose lives are ruined by being mis-identified by the system.
When a child writes "daddy touches me between the legs" in an essay, it doesn't matter if a human spots it or an AI that forwards it to a human, this needs to be investigated either way.
> Those same people whose lack of competence people are bemoaning throughout these comments.
It's not a lack of competence that's bemoaned, it's a massive amount of understaffing (and resulting overwork) in teachers and other school resources, as well as a drastic lack of financing because it's easy to cut budgets for schools for politicians as the effects only show up two decades afterwards.
Sprinkling some AI over it won't fix those issues, I'd argue it will make them worse as people blindly accept the results.
There were some cases in the UK about a decade ago where bugs in software the Royal Mail was using led to incorrect accusations of fraud. People actually went to jail over this, it took years to resolve.
> When a child writes "daddy touches me between the legs" in an essay, it doesn't matter if a human spots it or an AI that forwards it to a human, this needs to be investigated either way.
When a child writes a set of things that individually are not very concerning, they may have cues that could say "hey, this kid, you should maybe keep an eye out for evidence of abuse."
Particularly attuned, experienced individuals might spot these cumulative cues, but we all know that this is not all people dealing with children.
Society's bigotry is going to flood that bad boy so quick you might as well name it Gobbels.
I love ML. I want children to be safe. This is not the place for ML or AI or Quantum or any tech.
What needs to exist is better resources for those children, that mother grading the tests, the teachers of those children, and social services that are meant to support them. If you want to make a difference about this, look there.
Don't go building a automaton King Solomon who decides why this kid should be taken from these parents because speaking Spanish was worth -0.1 on some goddamn weight trained on data generated from a racist society.
This isn't a "spooky" correlation a cool algorithm can detect, it's a serious, layered social problem.
> Don't go building a automaton King Solomon who decides why this kid should be taken from these parents because speaking Spanish was worth -0.1 on some goddamn weight trained on data generated from a racist society.
Totally what I advocated for and not a strawman attack /s. Indeed, the chance that such an algorithm could be racist or classist and there being needs to avoid bad correlations and have appropriate controls is important.
I think there are opportunities here. Ideally ed-tech doesn't take humans out of the loop, but asks schoolteachers and administrators questions like, "Hey, are you sure students A, B, and C are being supported correctly for subject Z? Are you sure students D, E doesn't have some kind of abuse or other significant home problem? It sure looks like student F is in this subpopulation that research shows benefits from educational intervention Y. You might want to keep your eye out for that."
And then the teacher goes "Oh, crap. Now that I think about D, there were always these little things 1, 2, and 3 that seemed off... maybe this is worth a referral to social services to check on what's up."
Or "Oh, ... maybe F's struggles in reading really are a speech problem and we should handle that"
That is not how the law work. The law states that if people at a school is made aware or suspect abuse then they must act on that knowledge. A ML scoring system is obviously unable to be made aware or having suspicions, but the administrators could be help responsible if they happen to see something and chooses to not act.
It would be interesting to know if a child psychiatrist could be held liable if incompetence prevented them from seeing obvious sign of abuse, but I doubt that is covered under the cited law above.
It seems like these accumulated errors in the educational system and filters needed to get through it would create a market inefficiency that could be exploited by a firm willing to ignore degrees, grades, and test scores and judge for themselves whether a candidate can do the job they're being hired for.
To me this brings up the absurdity of having essays on standardized tests. What about an essay is standardized? It's a totally nonsensical premise.
This always gets made into some kind of techluminati conspiracy for the machines to ingrain structural racism whereas it's pretty clear all the algorithms fail to do is improve an already bad situation stemming from a flawed premise.
A number of states found out their schools were graduating students who genuinely could not read or write effectively. If you want to quantify that, you're forced to test it somehow. How would you test writing ability without asking them to write something?
It's nuts when you put it that way. To really standardize an essay, you'd have to give the prompt and argument to be made and just test their ability to turn it into prose.
From the article: "All essays scored by E-rater are also graded by a human and discrepancies are sent to a second human for a final grade. Because of that system, ETS does not believe any students have been adversely affected by the bias detected in E-rater."
That particular company seems to do a not-horrible job. But they’re not the only game in town, so presumably most or many essays are graded by another company’s system.
> Of those 21 states, three said every essay is also graded by a human. But in the remaining 18 states, only a small percentage of students’ essays—it varies between 5 to 20 percent—will be randomly selected for a human grader to double check the machine’s work.
Any state that relies on the AI as the primary grader does not understand the current state of AI.
It would make sense to use the AI as a first pass, and then not randomly grade the essays with a human, but specifically choose all the essays that are on the cusp of the pass fail line. Then use all those human generated scores to update the model, especially if someone moves from pass to fail or fail to pass. Then maybe throw in a few of the really high and really low outliers to make sure those are right, and throw away your entire model if the human scores are drastically different (and obviously don't tell the humans what the computer score was so they have no idea if they're reading a "cusp" essay or an outlier essay).
But putting the educational fate (and therefore future earnings) in the hands of an AI is unconscionable.
I'm normally pretty open minded but this is just stupid. AI is nowhere near literate enough for this task.
What kind of world is it when humans create merely for the consumption of machines. The product of our creativity deserves better.
I would support any student who refuses to consent to their work being used in this fashion.
This is a natural development of industrialized education. Treating children as individual thinkers would require for more resources and manpower than our system would like to provide.
I've heard stories from others in the industry of companies using tools like this on their human-facing documentation and requiring a certain score from them. Imagine using Microsoft Word's spelling and grammar checker, not being able to add or override its decisions (without following an extremely lengthy and bureaucratic process), and being required to have less than X "defects" per 100 words. Naturally, this results in documentation that is perfectly grammatical and free of spelling errors, but verbose, full of unusual phrasing, and next to useless for its actual purpose of informing a human.
Grading students' code using a machine is not such a bad idea in contrast, because in that case there is [1] no exceptions possible in a programming language, [2] the machine (compiler) has to understand it anyway, and [3] it does save time verifying correctness. But communication in a human language really needs to be assessed by humans. Anyone who thinks "AI" can accurately assess human language is either severely delusional, or trying to make $$$ from it.
The sooner we get it out of our heads that this education system of ours is a meritocracy the closer we’ll get to actually creating a quality universal system.
I wish this machine bias wasn't always presented in such divisive terms as race and "disadvantaged groups". It can affect anybody. If you happened to develop a writing style that looks like typical bad essay writers' style, then you could be hurt by bias in the grading.
Anyone can be hurt by bias, but minority groups are the obvious ones to be most likely statistically affected by it, making them the most obvious red flag for this kind of situation.
There are many classes of people who have problems of discrimination. Short, ugly, ginger, etc. The intersections of all those classes are so numerous that everybody will have some disadvantage. But it won't be apparent unless you define their class and measure it.
In terms of the technical effect that causes it as opposed to sociological.
The point then isn't that the algorithm hates black people or the programmers are racist (even if they were they would likely find it hard to train it to specifically exclude accurately without major side effects), but that their training set or analysis is flawed - potentially even if it is representative. Even if the results are problematic trying to address hate which isn't there won't be helpful. Call for better vetting procedures instead or that it clearly isn't ready for the proposed application.
Say it gets the highest accuracy on a set of photos of non-people images and a set of people which represents the exact ethnic make up all tagged just as "has people" or "doesn't have people". Going with something biased towards the most populous groups would get the best results quickest.
Talking about say image tagging to ensure automated performance checking across various characteristics could be productive on the other hand.
Good lord what a terrible design. Rather than determine if the writer has a coherent understanding and a complex prompt, the system grades based on writing patterns. This is actually my biggest fear of AI. Deploying wide scale systems like this that have very clear flaws
They seem to have figured out being political gets them clicks over substance. So much for the days they just reported stories other media wouldn't cover.
Any sources backing anything here?
Is NLP actually used to primarily grade papers anywhere in the USA?
It seems way beyond current technology.
Checking if a human marker doesn't need their score checked, perhaps could possibly be implemented.
I think machine learned grading of papers is insane, but at the same time I don't think we should be training or encouraging students to speak in AAVE (as the article suggests).
I think the right approach for machine learned systems is to automatically "whitelist" essays rather than "blacklisting" them. Students in the middle of the distribution of essays aren't really interesting, so whitelist them, give them a pass. Those at the extremes can be either exceptional or terrible, but usually terrible. The judgement of those at the extremes should be decided by a human, not a machine. You wouldn't want to blacklist the Einstein of essays because he did something genius that is indistinguishable from insanity.
However, I think there are some essays that can automatically be blacklisted. For example, those with:
1. Plagiarism (perhaps human moderated)
2. Extremely low word count
3. Extremely high count of fake words
And at the end of the day, these essay assignments aren't there to judge whether a student is the next writing sensation; they are given to judge whether the student can write legible sentences and words, to ensure they are prepared for the future. So perhaps it is at least possible to automatically blacklist on sentence structure and spelling (you should just lose points for invalid structure or invalid words, you shouldn't gain points for big words or complicated sentences). To make this fair, the student should be informed of this requirement. If they are informed and still fail, then they need to be remediated. If we discover that a disproportionate number of minorities are getting blacklisted, then we should investigate why the school is failing to teach them proper sentence structure and spelling, not pretend we can change the world to make AAVE an acceptable dialect of english in the workplace.
301 comments
[ 3.9 ms ] story [ 233 ms ] threadAnd “between 5 to 20 percent” of essays are randomly selected for human review.
So the takeaway is that if you’re one of the 80-95% of (typically black or female) people who the machine scored dramatically lower, but are not selected for human review, your education future is systematically fucked and you have no knowledge of why or how to change it.
Absolutely reprehensible. Anyone involved in the creation or adoption of these systems should be ashamed.
At least the machines offer the following hope: even if unbiased humans are rare among paper-grading teachers, those humans can be used to train the machines, so then bias-free or lower-bias grading becomes more ubiquitous.
Basically, the system has the potential for systematically identifying and reducing systematic bias. A computer program can be retrained much more readily than nation-wide army of humans. Humans can be given a lecture on bias, and then they will just return to their ways.
AI certainly has a lot of potential for bias, but claims that AI bias is somehow worse than good old human bias always seem shoddily supported (Note I'm not claiming it's untrue. Just that it's never been shown to my satisfaction, which is not surprising given how quickly AI is changing. It may well be true.)
Well, AI bias can combine sampling bias with human bias. Like say we train the AI with the output of only 10 human paper graders, all chosen from the same school district.
Due to the sampling bias, that data could create markedly more (or less!) bias than the entire population of human paper-graders.
The resulting AI will ideally mimic those 10 humans, though; it shouldn't show more bias than that group. If those 10 are flaring racists, and grade accordingly, the AI will be the same. (In fact, we hope that it will be the same, if the algorithm actually works in mimicking human grading.)
The bias comes from the human-generated training data in the first place; the machine isn't introducing its own. For instance, the machine has no inherent concept of disparaging someone's language because it's from an identifiable inner city dialect. If it picks up that bias, at least it will apply it consistently. When we investigate the machine, the machine will not know that it's being investigated and will not try to conceal its bias from us.
On the other hand, eliminating bias from humans basically means this: producing a new litter of small humans and teaching them better than their predecessors.
It's quite funny how are some people manipulated to think that society and especially education is somehow biased against minorities or women when opposite is true[1][2].
[1] https://www.studyinternational.com/news/record-high-numbers-...
[2] https://www.independent.co.uk/news/world/americas/black-wome...
An essay could be different from the reference standard because the standard is an example of good writing and the essay is not. Or it can be different because the author has a cultural, regional, gender, or developed background that imparts a different style than anything in the training corpus. Mistaking the two is very, very bad.
I think you have misunderstood the parent, who is asserting that the machines scoring these essays typically give lower scores to members of these demographics. This does not, by itself, mean the entire system is biased against those groups.
Whether or not the groups are, overall, systematically benefitted or harmed is not relevant to the injustice this article says exists.
That's the problem - there is seemingly no shame these days. People involved "saved time and money", got paid and that's it. "If I didn't do it someone else would" and all of that.
If it is cost-prohibitive for every essay to be graded by humans, then they should be dropped from the tests. Otherwise, we are missing the whole point of essays which is to communicate effectively with another human, not just match certain text patterns.
Apparently it is. But everyone still wants writing to be assessed…
If you want to grade on form to test the ability to write correct rather than coherent sentences, make those separate questions, and mark them so.
I agree, this is traditionally the purpose of an essay. But to play devil's advocate, consider the rising number of people who are writing SEO or ASO content which is actually targeted at machines.
It was a few weeks ago that someone shared “The Dark Age of AI” on HN [1]. I think we are promising way over what Drew McDermott thought we would not going to promise. This is to the extend that we are applying AI on assessing Art, Creativity and even quality and novelty of Science, something that in a way we don’t even understand (or trying to understand) ourselves at the time that we are publishing it.
[1]: https://news.ycombinator.com/item?id=20546503
Any essay writing test which could be adequately graded by a machine is not testing anything of value.
Edit: I’ll further add that as soon as people’s careers depend on a metric, the metric becomes useless as a metric, because it will be gamed and manipulated by everyone involved. Almost nobody involved is incentivized to accurately measure student’s writing ability.
A lot of what students write is actually garbage from that point of view. Even if they happen to have a good basic idea about what they want to say, the point of essay writing is to master the mechanics of expression so that you get the idea across effectively.
Whether the student has a brilliant idea isn't even so important, and it wouldn't even be fair; imagine if high school computer science expected students to turn in a best-selling app for a term project. Not everyone can come up with something brilliant to say; and even relatively mundane lines of reasoning can be given a good treatment in writing to develop the skill.
I remember when I had essays graded in school, a lot of the comments were low-grade fluff like "run on sentence", "wrong word", "faulty parallelism", "missing colon before 'for example'" and such points having nothing to do with the content being original, well-considered and well-argued. That sort of thing might as well be done by machine, at least as a preprocessing step to improve a student's rough draft.
It's the same reason you see keyword posters in math education. "Together" means "plus", that kind of thing. It's completely worthless, except for one-step problems, and even then it doesn't always work. What is happening is collusion between teachers and testmakers. You can't teach understanding, but you can teach test-passing techniques because the way the test is set permits this.
You see the same thing here, in English you can get away with not teaching quality writing if you teach techniques to score well.
Hint: together does not mean plus.
When your essays are graded they're marked down for mechanical and wording problems. There's really no point in trying or grade 'good ideas' on a subject piece you had maybe 10 minutes to skim.
That's a travesty, and you know it because when the kids are in college and they have as much time as they like to write their assignments they all use the wrong words and then misapply them.
However, it is rather important that students know their essays are not judged as essays, but only judged on the content. Otherwise you teach students that form trumps content in essays.
When judging an essay as an essay correct English barely matters. What matters is how convincing you are, and how interesting of a read the essay is. This is a great skill to have, and testing it also makes sense. Really though, we should separate these two forms of testing.
A thing you quickly find if you try to download off-the-shelf NLP tools and apply them to anything is how little is reliable at all, unless you can constrain the domain. Even basic topic identification only works with low error rates when constrained to something like NYT stories, or PubMed abstracts, not arbitrary text by arbitrary writers. And I would bet ETS is using worse tech than research state-of-the-art.
[1] e.g. https://www.aclweb.org/anthology/P15-1053
I still agree that automatic essay grading is beyond the reach of SOTA NLP models today, but youmake it sound like virtually nothing can be done in a production-grade manner that solves real world unconstrained NLP problems. This is manifestly false.
I'm not aware of any meta-analyses myself. I have been keeping up with the ASAP competition and various attempts to improve on the initial systems for a number of years. The two papers I believe are having the most success are [1] and [2]. [3] seems promising for balancing the opposing forces of high accuracy for true positives and the risk of false positives via adversarially crafted inputs.
I'm also vaguely aware of research happening around extracting features from neural nets. I'd love to be able to help students understand why the system is predicting a particular score.
[1] https://www.aclweb.org/anthology/D16-1193 [2] https://arxiv.org/pdf/1606.04289.pdf [3] https://arxiv.org/pdf/1804.06898.pdf
People making big decisions with a lot of money around computing know nothing about it and are marks for con-artists. Think big consulting firms selling to senior public servants in washington. "For a successful technology reality must take precedence of public relations." But reality just gets in the way when conning a mark for a successful snake oil sale, right?
Call it out, publically, cite your credentials. Encourage colleagues, your competition and everyone with a clue to pour scorn on whoever is selling this evil, toxic waste as drinkable.
Like this kind of thing should be cool, not insane. I mean wasn't it cool in your AI class when you learned that DFS could play Mario if you structured the search space right?
It took me much longer to pass my English language O level (exam taken at 16).
It is fine to play with "cool" techniques when you are doing consequence free stuff like playing Mario. When you are creating systems that have significant and long term effects of people's lives a different standard applies.
* 5% - cool, we could make a company that grades essays
* 15% - cool, we could make a company that grades essays and sell our source code to the test-prep industry
* 80% - fascinating, it sounds like the exam designers need to reevaluate what they are trying to measure with essay questions
And if you succeed you will simply be measuring an uninteresting but manageable subset of the problem which will then become in some people's eyes the definition of the problem.
Education is supposed to be about teaching people to think, to give them the tools with which to do it, to be able to evaluate, criticise, invent, etc.
I'd find it highly interesting to see what kind of result I'd get using an automated system.
Why?
Because, I once asked a teacher (also an examiner) why I got good grades above the others, and the answer surprised me: my answers were generally unique /refreshingly different, to the point/ not too long and easy to read.
I suspect with this new system, I'd be an average student. It'd also be interesting to find out, several years down the road, if the automated system could be gamed at all -- I suspect it could, and teachers would help students 'maximise' their scores as a result of that.
You know, those average writers like Hemingway /s
In fact, throughout the article I kept being surprised by the idea that long is good. When writing, I tend to prefer being brief.
That only really shows that the humans they're training on are terrible at grading essays.
GIGO is our God now.
https://en.wikipedia.org/wiki/Garbage_in%2C_garbage_out
Yeah, it's cool, but what about your savings account?
Narrowly for grammar however - is even that a good thing? It probably helps scale grammar help to more students, but if those tools became ubiquitous in grading and editing then unique voices would just disappear and a lot of potentially “great writers” might choose different careers because the machines don’t like them
The fact that it's being implemented in society is insane because anyone who is paying attention to the state of AI today already knows how it will go wrong: without reading the article I already guessed that it systematically discriminated against certain demographics. Which was in fact what the article claimed.
It's interesting that it's possible to predict what the scorer would decide, but the moment you actually implement it is when all of the known problems become relevant, and the intellectual wonder must take a backseat to the human problems.
To emphasize, this was 16 years ago.
Maybe nobody really makes a big deal about it because it is pretty much irrelevant anywah. Applicants provide a letter of intent that the grad dept people can, y'know, actually read for themselves, so I think unless you totally bombed the writing section nobody cared.
One major problem with algorithmic approaches, whether automated or not, is that they become the definition of good in the context and therefore become something that cannot be argued against. And of course it makes 'teaching to the test' an even more likely outcome.
If I were a conspiracy theorist I'd attribute this to wanting a dumbed down population. Unfortunately I think it is probably the other way round, the population is already dumbed down and a belief in AI unicorns is the result.
As Aristotle said to Alexander: 'There is no royal road to geometry', and so it is with education; it's hard work for both the student and the educator and no amount of AI/ML/algorithmic snake oil will change that without also changing the meaning of the word education.
So the reason this isn't the case, is because there are very simple metrics that tend to highly correlate with essay quality. It doesn't mean the grading-bot is actually evaluating essay quality. It's just looking for properties that are statistically associated with good essays. Remember, at the end of the day as long as the bot's ranking is close enough to the human grader's ranking, nobody really cares about the internal logic.
A very straightforward example is spelling mistakes. People who make spelling mistakes aren't necessarily bad writers. And vice versa, there may be great speller who can't write for shit. But by and large the people who spell poorly also tend to write poorly. Easily detectable grammatical issues, like misplaced modifiers, subject verb disagreement, or inconsistent tense, are also correlated indicators.
A very simple metric is essay length. Especially if its a timed exam. Good writers tend to have verbal fluidity, with words easily flowing to paper. They don't struggle converting thoughts too sentences. So they tend to end up with the most words written down within a fixed time period. By and large the longer a timed essay is, the more likely that its actual quality is high.
Grading bots basically rely on these statistical relationships. They're not measuring anything intrinsic to good writing. But at the end of the day, their student rankings are usually pretty close to that of a typical human grader. In some cases the bot will have a closer ranking to a random human grader, than two random human graders will have to each other.
The biggest flaw here is Goodwin's law. When the test takers become aware of the kludges that the bots use, they can exploit it. For example just dump a bunch of verbal diarrhea with as many correctly spelled words as possible. But even then it doesn't really hurt the bot's ranking accuracy too much. Because the kids who do the most test-prep and learn all the tips and tricks, are usually high-achievers who do well on essays anyway.
This is related to current fairness-in-AI discussions. In many cases the basic problem is ML systems leverage correlations for making causal decisions. Here, there is a huge ethical difference between scoring a person based on "is this a good essay" and "do the features of this essay correlate with features of good essays". Just like there is a huge fairness and discrimination difference between "is this person qualified for a loan" and "do the features of this person correlate with features of people who qualify for loans" (algorithmic redlining). Your last sentence has a big discrimination/fairness issue also, since you are testing even more for parental income and parental free time.
Thanks for catching the mistake.
Anyway, yeah, not really a correlation.
>Remember, at the end of the day as long as the bot's ranking is close enough to the human grader's ranking, nobody really cares about the internal logic.
This isn't true at all. Imagine you got a B or C on an essay that a human would have given an A to because you wrote it concisely and in plain language, or because you used language that's statistically correlated with being black. Does the fact that this is rare console you? "Sorry, but it's usually very close to the human grader's ranking." Close enough isn't good enough when you get the short end of the stick. "Sorry, you aren't going to get to go to the college you wanted because you use language statistically correlated with poor writing." Or just because you're different, so the statistical correlation doesn't apply to you, you filthy outlier. Just because it's a rare event doesn't make it okay.
In adulthood, this is like hiring or firing for work statistically correlated with good work. Remember when amazon rolled out the resume scorer? [0] Sure it was biased towards women, but it was close enough to human scores, so who cares about the internal logic?
>Grading bots basically rely on these statistical relationships. They're not measuring anything intrinsic to good writing.
At the end of the day, our goal here is to measure good writing. If the bots aren't measuring anything intrinsic to good writing, we shouldn't use them.
https://www.reuters.com/article/us-amazon-com-jobs-automatio...
You could ask a student to write an essay taking a firm opinion on some subject, and they could change standpoint every paragraph and there's no way these systems would know.
If I was a student I would be extremely offended at people wasting my time like this.
This seems like a terrible idea.
It's not a stretch to imagine the opportunity for nefarious behavior this allows - think of the recent college admission scandals, and how happy they'd be to have a guise of algorithmic indifference'.
If used long-term, it could offer a big advantage to the wealthy in other avenues. Another hypothetical, probably not far from reality: the algorithm becomes solved (almost or completely) by some premier 'tutoring' company. Said company can charge a pretty penny given its stellar track record, offering yet another hidden advantage to the wealthy/elite.
One question she had to grade was essentially, "What's something you want your teacher to know about you?"
It was an essay answer, and she was supposed to grade it on grammar, etc. Just the mechanical aspects of writing. (The real question explained the details more, but that was the core of the question.)
She saw answers that would make you weep.
"My daddy touches me."
"I haven't eaten today. I don't know when I'm going to eat again."
Stuff like that.
And my mother was going to be the only human who ever saw their responses. Their teacher had no chance to see their responses, just my mom.
So she goes to her supervisor and asks, "What can we do to help these kids?"
The supervisor said there was nothing you can do. Just grade the answers.
https://www.childwelfare.gov/topics/systemwide/laws-policies...
So do what? Contact her local police?
With a written accusation from a child? Is that enough to get a warrant to force the company to release the demographic information?
And people don't work at a job like that because they want to. They work there because they need the money.
Everything she took in and out of there was monitored, too. So it's not like she can go to the Xerox, and walk out of there with a copy.
It's beyond dehumanizing. For everyone. The kid, the people who work there.
> With a written accusation from a child? Is that enough to get a warrant to force the company to release the demographic information?
Absolutely! My girlfriend works as a counselor at a school and she is required by law to report all serious abuses by parents.
Collect or photograph all the evidence, record every conversation with supervisors, escalate as much as possible internally, then contact local police, and at the same time go to the media. Don't quit, but if necessary let them fire you and then sue. None of this is easy.
> With a written accusation from a child? Is that enough to get a warrant to force the company to release the demographic information?
Yes, she should have absolutely went to the local police. A child's first hand account in writing of child abuse and neglect is slam dunk evidence to secure a warrant to link the essay ID to the individual child.
> Everything she took in and out of there was monitored, too. So it's not like she can go to the Xerox, and walk out of there with a copy.
Doesn't matter. She could have went to the police herself as a witness. That alone would be enough for a probable cause warrant to retrieve the essays.
It is very sad she saw these signs of abuse and did not report it.
I don't find it adds much here.
So how many are true and how many false? I have no clue. Literally none. And no it doesn't make me feel any better about the screams of existential agony even if that were a low percentage. Could be high too.
In the US, school funding is based upon standardized test results, and bad results can shut a poorly performing school down.
It's drilled into every kid's head that these tests are very important, super strict and if they accidentally mess up, it can ruin their academics, because retesting and regrading are expensive.
I thought the state was holding the school hostage, threatening to cut funds or shut them down if they ever stopped. We never learned anything about civics or American history. Until I was out of highschool, content regarding atrocities like slavery and the trail of tears was not on the test and that was enough to whitewash the whole curriculum.
Standardized testing is to the U.S. what lead waterpipes were to the roman empire.
Food Security Status of U.S. Households with Children in 2017 Among U.S. households with children under age 18:
84.3 percent were food secure in 2017. In 8.0 percent of households with children, only adults were food insecure. Both children and adults were food insecure in 7.7 percent of households with children (2.9 million households). Although children are usually protected from substantial reductions in food intake even in households with very low food security, nevertheless, in about 0.7 percent of households with children (250,000 households), one or more child also experienced reduced food intake and disrupted eating patterns at some time during the year.
..which means students wrote whatever the hell we wanted. I was assigned a Captain Morgan (rum) ad. I wrote that the ad was glorifying maritime piracy and was likely responsible for pirate activity in Somalia.
As a child you're really not prepared for the concept that your parents are treating you badly. So that realization doesn't come until much later.
Faculty, administrators, athletics staff, or other employees and volunteers at institutions of higher learning, including public and private colleges and universities and vocational and technical schools (11 States).
https://www.childwelfare.gov/topics/systemwide/laws-policies...
https://www.childwelfare.gov/pubPDFs/manda.pdf
This includes penalties for failure to report in multiple states:
https://www.childwelfare.gov/topics/systemwide/laws-policies...
I'd believe that ML could spot abuse that humans miss pretty well from signals like non-overt references in homework and school records, if one could come up with an adequate training set.
Much more likely than teaching ML to score reasoned and creative activity in any reasonable way.
But this type of thing seems like the exact kind of spooky correlation that ML is good at spotting compared to humans.
"Spooky" machine learning results happen when a correlation is abundant in a dataset [2]. Otherwise, machine learning techniques will probably miss it altogether.
______________
[1] Quick online search: https://www.inquirer.com/philly/blogs/healthy_kids/What-is-t...
[2] The archetypal spooky machine learning story is surely the one about Target sending baby item coupos to a girl in high school before her father knowing she was pregnant:
https://www.forbes.com/sites/kashmirhill/2012/02/16/how-targ...
The total incidence of child abuse of all types from infancy to adulthood is on the order of 1 in 3. This is not terrifically rare-- it's of higher prevalence than pregnancy and of positive screening events.
A much bigger concern is non-causative correlations. It'd be pretty easy to train ML to be racist or look for e.g. indicators of class, which are correlates of abuse.
As to false positive rates-- you can pick your false positive rate to be whatever you want it to be, by twiddling the threshold for a positive result. I'm not sure false positives are of that great of a concern, if the output from a system is a notification to school administrators that they may want to keep an eye out for this student.
Child sexual abuse isn't extremely rare and familial abuse is a very large minority of child sex abuse.
How? Particularly, where do you get training data at the required scale?
You survey those kids in adulthood about whether and how they were subject to abuse and other types of relevant adversity.
You attempt to control the data so that you don't just latch onto other correlates of abuse (e.g. social class).
The people whose lives are ruined by being mis-identified by the system.
The positives must be evaluated by a human anyway.
Those same people whose lack of competence people are bemoaning throughout these comments.
When a child writes "daddy touches me between the legs" in an essay, it doesn't matter if a human spots it or an AI that forwards it to a human, this needs to be investigated either way.
> Those same people whose lack of competence people are bemoaning throughout these comments.
It's not a lack of competence that's bemoaned, it's a massive amount of understaffing (and resulting overwork) in teachers and other school resources, as well as a drastic lack of financing because it's easy to cut budgets for schools for politicians as the effects only show up two decades afterwards.
There were some cases in the UK about a decade ago where bugs in software the Royal Mail was using led to incorrect accusations of fraud. People actually went to jail over this, it took years to resolve.
When a child writes a set of things that individually are not very concerning, they may have cues that could say "hey, this kid, you should maybe keep an eye out for evidence of abuse."
Particularly attuned, experienced individuals might spot these cumulative cues, but we all know that this is not all people dealing with children.
It's an interesting problem.
Society's bigotry is going to flood that bad boy so quick you might as well name it Gobbels.
I love ML. I want children to be safe. This is not the place for ML or AI or Quantum or any tech.
What needs to exist is better resources for those children, that mother grading the tests, the teachers of those children, and social services that are meant to support them. If you want to make a difference about this, look there.
Don't go building a automaton King Solomon who decides why this kid should be taken from these parents because speaking Spanish was worth -0.1 on some goddamn weight trained on data generated from a racist society.
This isn't a "spooky" correlation a cool algorithm can detect, it's a serious, layered social problem.
Totally what I advocated for and not a strawman attack /s. Indeed, the chance that such an algorithm could be racist or classist and there being needs to avoid bad correlations and have appropriate controls is important.
I think there are opportunities here. Ideally ed-tech doesn't take humans out of the loop, but asks schoolteachers and administrators questions like, "Hey, are you sure students A, B, and C are being supported correctly for subject Z? Are you sure students D, E doesn't have some kind of abuse or other significant home problem? It sure looks like student F is in this subpopulation that research shows benefits from educational intervention Y. You might want to keep your eye out for that."
And then the teacher goes "Oh, crap. Now that I think about D, there were always these little things 1, 2, and 3 that seemed off... maybe this is worth a referral to social services to check on what's up."
Or "Oh, ... maybe F's struggles in reading really are a speech problem and we should handle that"
It would be interesting to know if a child psychiatrist could be held liable if incompetence prevented them from seeing obvious sign of abuse, but I doubt that is covered under the cited law above.
This always gets made into some kind of techluminati conspiracy for the machines to ingrain structural racism whereas it's pretty clear all the algorithms fail to do is improve an already bad situation stemming from a flawed premise.
> Of those 21 states, three said every essay is also graded by a human. But in the remaining 18 states, only a small percentage of students’ essays—it varies between 5 to 20 percent—will be randomly selected for a human grader to double check the machine’s work.
So that applies only in a minority of cases.
It would make sense to use the AI as a first pass, and then not randomly grade the essays with a human, but specifically choose all the essays that are on the cusp of the pass fail line. Then use all those human generated scores to update the model, especially if someone moves from pass to fail or fail to pass. Then maybe throw in a few of the really high and really low outliers to make sure those are right, and throw away your entire model if the human scores are drastically different (and obviously don't tell the humans what the computer score was so they have no idea if they're reading a "cusp" essay or an outlier essay).
But putting the educational fate (and therefore future earnings) in the hands of an AI is unconscionable.
I would support any student who refuses to consent to their work being used in this fashion.
Grading students' code using a machine is not such a bad idea in contrast, because in that case there is [1] no exceptions possible in a programming language, [2] the machine (compiler) has to understand it anyway, and [3] it does save time verifying correctness. But communication in a human language really needs to be assessed by humans. Anyone who thinks "AI" can accurately assess human language is either severely delusional, or trying to make $$$ from it.
If you don't talk about the actual problem, how can you possibly expect to solve it?
The point then isn't that the algorithm hates black people or the programmers are racist (even if they were they would likely find it hard to train it to specifically exclude accurately without major side effects), but that their training set or analysis is flawed - potentially even if it is representative. Even if the results are problematic trying to address hate which isn't there won't be helpful. Call for better vetting procedures instead or that it clearly isn't ready for the proposed application.
Say it gets the highest accuracy on a set of photos of non-people images and a set of people which represents the exact ethnic make up all tagged just as "has people" or "doesn't have people". Going with something biased towards the most populous groups would get the best results quickest.
Talking about say image tagging to ensure automated performance checking across various characteristics could be productive on the other hand.
They seem to have figured out being political gets them clicks over substance. So much for the days they just reported stories other media wouldn't cover.
Any sources backing anything here?
Is NLP actually used to primarily grade papers anywhere in the USA?
It seems way beyond current technology.
Checking if a human marker doesn't need their score checked, perhaps could possibly be implemented.
I think the right approach for machine learned systems is to automatically "whitelist" essays rather than "blacklisting" them. Students in the middle of the distribution of essays aren't really interesting, so whitelist them, give them a pass. Those at the extremes can be either exceptional or terrible, but usually terrible. The judgement of those at the extremes should be decided by a human, not a machine. You wouldn't want to blacklist the Einstein of essays because he did something genius that is indistinguishable from insanity.
However, I think there are some essays that can automatically be blacklisted. For example, those with:
1. Plagiarism (perhaps human moderated)
2. Extremely low word count
3. Extremely high count of fake words
And at the end of the day, these essay assignments aren't there to judge whether a student is the next writing sensation; they are given to judge whether the student can write legible sentences and words, to ensure they are prepared for the future. So perhaps it is at least possible to automatically blacklist on sentence structure and spelling (you should just lose points for invalid structure or invalid words, you shouldn't gain points for big words or complicated sentences). To make this fair, the student should be informed of this requirement. If they are informed and still fail, then they need to be remediated. If we discover that a disproportionate number of minorities are getting blacklisted, then we should investigate why the school is failing to teach them proper sentence structure and spelling, not pretend we can change the world to make AAVE an acceptable dialect of english in the workplace.