Being held accountable to metrics will incentivize some behaviors over others. It's not surprising that when Arts and Phys Ed are not on the test, they are dropped from the curriculum. I think student, teacher, doctor and patient happiness should be one of the key metrics that is measured and reported along with test scores and mortality rates. That would incentivize schools and hospitals to take a more balanced approach.
> I think student, ... happiness should be one of the key metrics
I have seen it first hand. Students are always happiest when a charming instructor provides bland infotainment that allows them to sustain a pretense of high achievement without trying very hard. The second hapiest students are those of mediocre teachers who give good grades to everyone so no one is very motivated to rock the boat. As a matter of fact, the teacher has to descend pretty low in terms of incompetence so the students will actually be less happy than if presented with a lecturer that knows their subject and expects them to put an actual effort in learning.
Of course, all things being equal, stern lecturers are more disliked than more relaxed instructors. This is not what I am taking about.
The thing I am always amazed by people who developed metrics is they never seem to stop and think how the metric will be gamed and then take action to prevent this. It should be part of the development of any new metric to have a "evil" team work out all the ways the metric can be gamed so this can be blocked.
For example, it should be impossible to "teach the test" other than by teaching broadly the underlying concepts being tested. A well designed test will touch on so many concepts that only a education that is broad will work.
I'm not even sure the voters/parents want the tests to be made harder like that. Then test scores would go down further and schools would have to devote even more time to passing them.
And FWIW, I've looked at the Texas tests and they didn't seem game-able to me. If they were, the "don't teach to the test" rabblerousers would cite them in a heartbeat. But no one seems to follow the logical implication of this argument.
You can make the tests as easy or as hard as you like and still prevent teaching to the test. The pass/fail rate is a function of the effectiveness of the teaching and the difficulty of the exam. If the fail rate is above the politically acceptable rate then make the test easier, just don’t make the test gameable.
I'm sure the teachers would already prefer a quicker way of teaching that also conveys a more general understand and enables the students to pass the test.
But if they already knew this method and could implement it, they would already be doing it.
No I am not saying the tests are too difficult, but I am saying that if the failure rate is politically too high then the way to solve this by making the tests easier rather than making the tests gameable.
Of course the best solution is to improve the quality of teaching, but this is a much harder problem.
Why do you assume they don't stop to think about that? People who develop assessments usually think quite rigorously about exactly such problems, assuming they're following proper methods. Is it so surprising to you that this is a very difficult problem?
I would compare this to asking, "Why don't software developers think about bugs?" They do, and they attempt to handle them, but it's almost impossible to solve all such problems in programs (tests) of non-trivial size.
Well, then the solution there is to only write trivial programs. Haha.
Though, with enough resources, time, and effort, I think almost any program one would want in practice could be written with a formal proof of correctness. (I won't say every, because that runs up against halting problem / incompleteness things, but I think for almost all that you would actually want)
It's just that there isn't enough demand, and time, for the programs to be proven correct.
If there was a program that we could all stand to wait for a very long time for (maybe even a few generations?), and it was important enough that it be completely free of bugs, that could be done, I think. I'm not sure what it would be that important though. Starship stuff?
If they did then it would not be possible to “teach the test” other than by teaching the full subject. If there are any shortcuts a teacher can take with the class then the test is a failure.
I used to write tests all the time in my subjects and my tests could not be gamed. I would have half multiple choice questions drawn from every area we covered in the semester and half essay questions where the students had to integrate what they learned into a coherent answer. My students would ask me what they had to learn for the final exam and I would always say everything - if it wasn’t going to be examined there was no point teaching it because the students won’t both learning it :)
The problem with applying this sort of measuring regime against education or medicine vs. a sales team or IT service operation is that the outcomes are difficult to assess, because he assessments are bogus.
A high quality general education should lead to a good outcome on a standardized test, (Say the SAT) right?
The reality is the complexity of the assessment requires preparation. Without understanding SAT test strategy, you won't do well. When I took AP US History, the difference between a 4 and a 5 (the highest score) was understanding how to "stage" essays for the graders.
Instead of worrying about red teaming standardized tests, we should think log and hard about the cultures being created in these environments.
It's always possible to "teach the standardized test". The only way to avoid it is to make the test subjective and non-standard. But then students who failed will complain that they were treated unfairly.
The key point is that unfair treatment of some students should be considered an unavoidable risk, because the alternative is unfair treatment of all students.
> The only way to avoid it is to make the test subjective and non-standard.
How so? You can make it multiple-choice (objective) difficult, non-algorithmic math problems comprising multiple concepts each and then grade the whole test on a curve.
Impossible to game except to actually teach the students a broad range of math skills and how to problem-solve, while still objective and standard (in the sense that everyone gets the same test and is graded the same).
Ooh yay, a topic I can speak fairly well-informed on.
>> You can make it multiple-choice (objective) difficult, non-algorithmic math problems comprising multiple concepts each and then grade the whole test on a curve.
You do realize that that's what Olympiads are right?
https://en.wikipedia.org/wiki/International_Mathematical_Oly...
You can be fairly confident the Chinese "teach for the test" to a few handpicked kids who can perform well on those problems (in the same way they select children with an aptitude for gymnastics at an early age). Nearly 20 wins, not one Fields Medal. You can 'teach to the test' at any level up to the 400 levels, at which point you start creating new ways to approach the test, solving problems which weren't solved for a couple hundred years, or founding problems which won't be solved for another couple hundred.
As a former English as a Foreign Language teacher, I can assure you that there are always ways to game exams in English. Because I had to help some of my students get into university, I took a look at the entrance exams. Even though I am a native English speaker, I found that I could not finish the Todai (top school in Japan) entrance exam for English in the required time. It required reading and understanding very large amounts of text and then answering difficult multiple choice questions at the end. Sometimes the questions were difficult to understand and required that you were really careful about concentrating on the wording.
So instead, I decided to try again (with a different year's test) and instead of reading the material ahead of time, I used the questions to direct my efforts. I would look at all the answers and then scan the text for areas that talked about those things. Having identified those, I then looked at the question and ruled out the answers that didn't seem related. Finally I settled on the answers -- mainly by applying grammar rules mechanically (i.e., a wrong answer would be in the wrong tense compared to the supplied text).
I was able to complete the exam in time and get full marks. I had absolutely no idea what the text said. In fact, I didn't even know what half of the questions asked.
Now for my dilemma as a teacher. If my student gets accepted to Todai, it will basically make their life. You are almost guaranteed to get a good job that will give you the kind of life that most Japanese people are looking for. If I ask the student to try to do the exam properly, by reading the text, they will fail. Even I would fail. I can teach them to pass it in a different way.
What should I do? The test is too difficult. There are always going to be ways to game it and the successful students do. Therefore the test gets more difficult and the students are forced to game it even more. As a teacher, should I tell my student that they can't go to Todai, or should I teach them how to game the exam?
As far as I was concerned, I decided that teaching English and passing entrance exams were 2 different things. I taught English for those students who were interested. I gave up a lot of my free time (a few hours every day) to help them do that. When exam time came, though, I taught no English at all. I taught them how to do what they needed to do to be successful. Every one of my colleagues did the same thing. Probably every teacher in the country does the same thing.
Standardized tests that grade you and decide your future are insane.
It sounds like it's gameable because it's predicatable.
Imagine the next year's Todai entrance exam was totally different and unpredictable (say, transcribing an Eminem song or something similarly off-the-wall), then all the people who studied ways to game last year's exam would be SOL and only the people who actually know English would get in.
That's the kind of system I'm advocating for: get a bunch of smart test writers together with the express intent of coming up with novel ways to destroy the people who game the exams.
The reason they can't reasonably make ungameable tests is that they have tens of thousands of applicants. The need for an easy to mark test is more important than the need for an ungameable test. Otherwise you could simply interview applicants in English. (Todai actually considered this approach, but I understand that they abandoned it as being impractical).
In my more cynical periods, I suspect that they don't mind people gaming the exams. After all, most Todai students don't actually need to speak English (or know crazy details of history, or write thousands of kanji, or whatever). They are more interested in clever students who are willing to study like insane people and practice boring stuff over and over again. That's what you need to do to game the exams. In a twisted way, it works...
Making test questions more difficult won't help. More people will make simple stupid mistakes, so it's going to be more like a lottery. If you forgive simple stupid mistakes, the test is becoming subjective again.
Here's an example of a bad question: what is the area of the right triangle with hypotenuse 10 and height 6 (height is measured from the vertex opposite to the hypotenuse)? It does not test understanding. The optimal algorithm for the student is to parse the question looking for keywords - "triangle", "area", "height", recalling the formula, and plugging the numbers into it to get the answer 30.
The question is from the real test, and by the way, this answer is incorrect. Why it is incorrect is a better question.
Here's an example of a good question: prove the Pythagorean Theorem. You can't answer it by picking the right answer from the multiple choice list. You have to explain your line of reasoning to the examiner and be able to answer question like "why you say these two triangles are congruent". Such interactive process is much better at determining student's skill level, and you can't prepare for it by simply memorizing stuff without understanding.
The student might not like the proof in the textbook and prefer another proof. It take a human to understand whether this other proof holds water.
Why don't you Google real Texas or California standardized tests (as opposed to hypothetical badly designed tests) and then list some explicit techniques for "teaching to the test"?
That would be far more convincing than merely claiming teaching to the test is real and can be done in all cases.
Are you suggesting that teachers don’t engage in “teaching to the test” or that they are doing it, but under the mistaken idea that the standardised tests are gameable?
I'm suggesting that I don't think "teaching to the test" (defined as things which increase scores but not understanding) is possible on well designed tests, such as the ones I cited.
I'm sure teachers might do it if they could (shady people teachers are), but don't see any evidence they can.
OK. I certainly agree with you that a good test is not gameable, but the evidence seems to be teachers believe they can game the metric even in CA and TX. This is a problem even if the test is not gameable since the student's time is being wasted.
Edit. I have been meaning to mention that I always enjoy your posts Chris even if I don't always agree with you.
Here's an example of the technique for "teaching to the test": the formula you use should include all the numbers in the question. For example, if the question is "find the angle alpha in this right triangle" and gives lengths of the two sides, you know the answer should be some function of these two numbers (and nothing else).
More importantly, non-interactive nature of the test does not allow for follow-up questions. A few follow-up questions on oral exam let the examiner probe a lot deeper in less time, therefore getting a much better picture of student's ability and skill level.
I think a very important part of learning math is learning that it's not magic or memory, it's just mechanical procedure transforming inputs to outputs (and therefore the output must contain the input).
I.e., you aren't just teaching the test, you are also improving understanding.
Now consider a question like "length of alpha, beta=2, gamma=3, a) sqrt(2^2+3^2) b) sqrt(2+3), c) 2^2+3^2, d) 7". Using your heuristic the student can rule out (d) and has a 33% chance of getting the right answer. The test reveals (averaging this over many questions) that they know a little bit, but nothing close to 100%.
The test would reveal that the student knows a little bit about taking the test. The heuristic would still work even if the test was written in Russian and the student did not understand a single word in the question.
20 years ago, a certain chain of fast food restaurants pushed through new metrics to compare different locations by. One of those was which shifts kept drive through average times under 60 seconds.. We got all sorts of recognition and praise.. Nobody noticed that first thing when opening was we ran through 20 or orders through drive through for water (no cost) and closed within 5 seconds. Then intermixed all day between orders, when it was slow.
This is a excellent example of a poor metric resulting from the test developers not thinking about how the metric could be gamed. If they had they could have made sure that they only counted orders over a certain value or even better counted size adjusted orders.
Education should be taught in a way such that one is exposed to the methodologies by which one should approach problems, rather than solving the problem itself[1]. "Teaching to the test" is awful, I agree. How can one, on a national basis, ensure that every school is affording as equal an opportunity as possible? I'm not being sarcastic - what is the better alternative, the better metric by which we can gauge the performance of both our students and our teachers? I certainly don't think 8th grade math teachers should be penalized for the poor performance of their students if those 7th graders who matriculated into your class on September didn't have a functional understanding of fractions. (Conversely, I think teachers who reached 'tenure'[2] and have clocked out until they've hit that pension-year _should_ be punished). How should those teachers be evaluated? One could suggest "grade the teachers on the improvement of performance of their students compared to their prior years performance" but that brings us back to the same problem -- you have a test. I ask this without snark - how do you test as many concepts at that broad a scope of subject, at that grand a scale (let's start small - 3 districts in NY state chosen at random)? How do you even choose what concepts are important at that broad a level?
[1] This gets into a whole meta-discussion about what the appropriate methodologies are, which I'll leave as an exercise to those doctoral candidates finishing up their dissertations.
[2] Tenure as I understand it serves to allow researchers could undergo extensive, perhaps subversive, study to enrich the human body of knowledge while retaining job security. While educators at a grade/secondary school level perform a great service, I fail (heh) to see the purpose it serves at that level.
Well each year should have a syllabus and what the students are expected to know and be able to do (we used to call them rubrics). You just test that the students perform in all the things in the syllabus. The only way to “teach the test” if testing was done like this would be to teach the syllabus which is exactly what the teachers are supposed to be doing anyway.
Yes this is 100% true, but the point I was making is if the test writes think first about how the metric can be gamed then they can take action to prevent the gaming including changing the metric and test.
You found a singular NEJM paper by the celebrated "checklist champion" Atul Gawande for a method that applies only to one specialty, surgery. How can you be sure that this generalizes to all medicine?
Given that doctors in most other fields spend a ridiculous (by some metrics, 7:1) ratio of time doing paperwork or data entry vs. seeing the patient [1], I would suggest that there is serious opportunity cost in adding more self-measurements and checklists to most doctors' workload. Surgeons are likely an outlier that perhaps benefit most from being regressed to the mean.
I can be sure that it generalizes to all medicine because it's obvious. It generalizes to all fields. It's why we write unit tests. Nobody likes writing tests, but we do it anyway because they work. Like, really, really well. The idea that doctors would object to being asked to do the same when people's lives hang in the balance is crazy to me.
Now, it's certainly possible they're doing a whole bunch of useless paperwork. And it's certainly possible that there is too much in some areas - but that isn't a rejection of metrics and objective criteria. It's a rejection of the particular implementations of those things in particular places.
EDIT: To be clear, I think that what is actually needed is an overhaul of the tech. used to collect these metrics and fill out this paperwork. It's absolutely insane to me that doctors still use paper at all, or that they have to enter the same data multiple times in multiple places, constantly. That needs to be fixed. And they need to start collecting 100x more data on their performance with 100x less friction of collection, by upgrading their technology in this regard from what was available in 1970.
I can totally understand your sentiment. I would just note that writing a unit test for a class and designing a metric that unambiguously measures good clinical care are on very different levels of difficulty. In our research group, we've tried to do some of the latter regarding infection control, and there is definitely a tradeoff between making metrics explainable in plain language (allowing mental buy-in by healthcare workers) and measuring attributable differences in performance, which usually requires a heap of tricky statistical corrections that few people can grok.
It would be a very interesting future where healthcare workers have the reams of stats that professional athletes do, and where everybody in the space is literate enough to intuitively understand each number and its caveats, like hardcore sports fans. For example, an ICU nurse is bound to encounter more hospital-acquired infections (HAIs) per week than a nurse in pre-op assessment, and so averaged cases of HAI per primary unit is the proper context for that stat, much like you'd weigh recent rushing yards for an NFL linebacker against the strength of the defenses faced. Then, managers could optimize their teams to each person's strength or weakness and/or supply training in the correct areas, much like coaches in sports do. This would be a sea change in the medical culture, though, and it would encounter staunch resistance, because some of the metrics already being pushed on doctors (# pts/day) arguably incentivize worse care.
To your last point, part of my research now is analyzing electronic medical record (EMR) data, and coming from any other field you'd be shocked at how messy it can be. In a way, this is to be expected, because the "vocabularies" for medical data are huge, and even widely used ontologies have bizarre properties [1]. I really hope that we can solve the data collection problems with better technology, but as of now, the incentives for EMR vendors to overhaul the tech is pretty low, now that most US hospitals have picked a system and those vendors have everything to lose and little to gain from overhauling their user interface.
>Whatever we do, we have to ask our clinicians and teachers whether measurement is working, and truly listen when they tell us that it isn’t. Today, that is precisely what they’re saying.>
The simplest solution, which very often turns out to be the most effective and least expensive solution, is to ask those personally involved in a system to improve it, themselves...if they have trouble getting started offer incentives for them to do so...
That (first) step is all too often skipped in our "age of impatience"...
I'm not sure I've understood your point, but that has some problems.
My parents are high school teachers in Italy. Right now the government is requiring a basic evaluation system for teachers. Tests here are hardly ever standardized and can vary a lot; tests for things like Latin that is still taught like 100 years ago would be impossible to standardize anyway, so assessing student performance is impossible.
They could not come up with some metrics for whatever reason, and now teachers have to decide these criteria by themselves. Of course nobody agrees on what such criteria should be, and those I've heard from my parents are hardly useful to improve teaching quality.
I think I understand the difficulties you identify...maybe I should have done a better job expanding upon, or explaining, what I'm advocating, which is simply the idea of "continuous improvement"...
In short, I'm in favor of involving those most closely related (those in direct contact, those in the actual environment) to what is perceived as a "problem" in devising a solution, or solutions...versus having an outside entity recommending improvements that might, or might not, improve outcomes...
Improvement is often hard, and when it comes it comes in increments...sometimes those increments are incredibly small, but even marginal improvements contribute to the success of the whole...even without "metrics" I think most teachers know, or sense, the difference in student outcomes their approaches have led to...
I'll bet (although reaching a consensus is difficult, especially across academic disciplines) that each teacher, at the end of a course, has in mind some way that their efforts could have been at least marginally more productive and could quickly answer the question, "What should I do differently next time?" ...
The best teachers do this automatically...and it makes all the difference...
My wife has a great doctor. We lost our first kid and this doctor took tons of extra time helping us in the second pregnancy. From fitting us into her already full schedule to taking 10 extra minutes in an appointment to calm my wife down.
There is nothing about the experience that could be captured in these metrics. But I can tell you that she kept my wife afloat during the bad days and probably kept her from seeing another councilor. Ultimately we had a baby girl the second time around. The kindness and as cited in the article, love, that the doctor showed us really really helped us.
Performance metrics sound like their empirically great, but as we all know, it's more complicated than that.
Personally, I've seen important programs, such as primary care-based depression screening and management that are proven to work then be turned into checklists and given to teams that are about checklist first, and less focused on mission/impact/caring. And I've seen patients respond that they are not helped. The therapeutic value of perceived empathy and attentiveness is unknown but should not be underestimated.
The irony is that in every organization, everyone knows who the performers and who the slackers are. But nobody wants to talk about that in deciding who gets promoted and who gets set aside.
A manager once told me that before the employee performance review cycle he had already decided who was going to get what raise. He'd start from that and work backwards to derive the required "metrics" to support it. The right people got the raises, and the HR department was satisfied with the scientific process :-)
I don't disagree with you that this is how things work, but my god this process is prone to corruption, both deliberate and unconscious. As a manager it is very hard to not reward your allies and punish your enemies and if you have a process that enables this then this is exactly what happens.
1. I wouldn't consider working for a boss who regarded me as an enemy.
2. The manager also gets evaluated, and word will get around if he's promoting cronies, and if his dept gets poor results as a consequence.
3. I mentioned that everyone knows who is naughty and who is nice. A rating system can be as simple as "who do the patients prefer" and "which teachers do the parents want for their kids". I expect that would be pretty accurate and pretty hard to game.
Walter nobody wants to work for a boss that considers them an enemy hence why there is so much politics in the workplace. Almost nobody is honest to their boss about what they think of them.
Word might get around about the boss, but when your job is on the line most people tow the line - as the Japanese say the nail that sticks out is the one that is hammered.
The problem with using metrics of what patients and parents think is this is not a good measure of what makes a good doctor or teacher. You can be an incompetent doctor with a fantastic bedside manner or a teacher all the parents relate to but who can't teach a fish to swim. The metric here is not the one you really want optimised.
On this topic I personally prefer doctors who have very poor beside manner because I assume they must have optimised competence over personal relations.
> The problem with using metrics of what patients and parents think is this is not a good measure of what makes a good doctor or teacher.
I think it's natural to assume this is not a good measure. Let's take teachers. I don't believe a teacher can consistently fool parents into believing they've taught the kid a lot when the kid is obviously not learning. And bluntly, the teacher is supposed to be working for the parent. Why shouldn't the parents decide who they want teaching their kids?
I think in all these discussions of bean counters, the underlying assumptions non bean counters make is that the bean counters are trying to improve things. That, in my experience, is seldom the case. Mostly the act of bean counting is to create positions for bean counters, and insert themselves into the process. There is no end game in this, they will happily have meetings to discuss processes and improve metrics, because to them, bean counting is all there is. Without bean counting they would be unemployed. Once you let them in, you're screwed, cause they're impossible to get out.
Edit: Why the down vote? Perhaps some argument? (Oh, of course, it was probably a bean counter)
I posted this as a very real comment - once you accept that what you are doing is an item measurable by parties that aren't professionals in your field (bean counters) then any qualitative aspects of the profession are lost. Only the quantifiable is left, and because the bean counter doesn't communicate using the language of the profession - you are forced to learn their language. A language of quantifiables and reduction to line items. The rise of the professional manager who has necessarily no expertise in the field they are managing must of necessity reduce all non quantifiables to a 0 value. Thus you end up in the pickle that educators and physicians find themselves in. In the past administration was performed by members of the profession and so they could communicate using the same language.
57 comments
[ 2.5 ms ] story [ 111 ms ] threadI have seen it first hand. Students are always happiest when a charming instructor provides bland infotainment that allows them to sustain a pretense of high achievement without trying very hard. The second hapiest students are those of mediocre teachers who give good grades to everyone so no one is very motivated to rock the boat. As a matter of fact, the teacher has to descend pretty low in terms of incompetence so the students will actually be less happy than if presented with a lecturer that knows their subject and expects them to put an actual effort in learning.
Of course, all things being equal, stern lecturers are more disliked than more relaxed instructors. This is not what I am taking about.
For example, it should be impossible to "teach the test" other than by teaching broadly the underlying concepts being tested. A well designed test will touch on so many concepts that only a education that is broad will work.
And FWIW, I've looked at the Texas tests and they didn't seem game-able to me. If they were, the "don't teach to the test" rabblerousers would cite them in a heartbeat. But no one seems to follow the logical implication of this argument.
But if they already knew this method and could implement it, they would already be doing it.
There is no quick way of really teaching. Teaching well is a really hard problem that many think is easy. It is not.
Of course the best solution is to improve the quality of teaching, but this is a much harder problem.
I would compare this to asking, "Why don't software developers think about bugs?" They do, and they attempt to handle them, but it's almost impossible to solve all such problems in programs (tests) of non-trivial size.
Though, with enough resources, time, and effort, I think almost any program one would want in practice could be written with a formal proof of correctness. (I won't say every, because that runs up against halting problem / incompleteness things, but I think for almost all that you would actually want)
It's just that there isn't enough demand, and time, for the programs to be proven correct.
If there was a program that we could all stand to wait for a very long time for (maybe even a few generations?), and it was important enough that it be completely free of bugs, that could be done, I think. I'm not sure what it would be that important though. Starship stuff?
I used to write tests all the time in my subjects and my tests could not be gamed. I would have half multiple choice questions drawn from every area we covered in the semester and half essay questions where the students had to integrate what they learned into a coherent answer. My students would ask me what they had to learn for the final exam and I would always say everything - if it wasn’t going to be examined there was no point teaching it because the students won’t both learning it :)
So you create an education system that's a mile wide and an inch deep.
A high quality general education should lead to a good outcome on a standardized test, (Say the SAT) right?
The reality is the complexity of the assessment requires preparation. Without understanding SAT test strategy, you won't do well. When I took AP US History, the difference between a 4 and a 5 (the highest score) was understanding how to "stage" essays for the graders.
Instead of worrying about red teaming standardized tests, we should think log and hard about the cultures being created in these environments.
The key point is that unfair treatment of some students should be considered an unavoidable risk, because the alternative is unfair treatment of all students.
How so? You can make it multiple-choice (objective) difficult, non-algorithmic math problems comprising multiple concepts each and then grade the whole test on a curve.
Impossible to game except to actually teach the students a broad range of math skills and how to problem-solve, while still objective and standard (in the sense that everyone gets the same test and is graded the same).
>> You can make it multiple-choice (objective) difficult, non-algorithmic math problems comprising multiple concepts each and then grade the whole test on a curve.
You do realize that that's what Olympiads are right? https://en.wikipedia.org/wiki/International_Mathematical_Oly... You can be fairly confident the Chinese "teach for the test" to a few handpicked kids who can perform well on those problems (in the same way they select children with an aptitude for gymnastics at an early age). Nearly 20 wins, not one Fields Medal. You can 'teach to the test' at any level up to the 400 levels, at which point you start creating new ways to approach the test, solving problems which weren't solved for a couple hundred years, or founding problems which won't be solved for another couple hundred.
So instead, I decided to try again (with a different year's test) and instead of reading the material ahead of time, I used the questions to direct my efforts. I would look at all the answers and then scan the text for areas that talked about those things. Having identified those, I then looked at the question and ruled out the answers that didn't seem related. Finally I settled on the answers -- mainly by applying grammar rules mechanically (i.e., a wrong answer would be in the wrong tense compared to the supplied text).
I was able to complete the exam in time and get full marks. I had absolutely no idea what the text said. In fact, I didn't even know what half of the questions asked.
Now for my dilemma as a teacher. If my student gets accepted to Todai, it will basically make their life. You are almost guaranteed to get a good job that will give you the kind of life that most Japanese people are looking for. If I ask the student to try to do the exam properly, by reading the text, they will fail. Even I would fail. I can teach them to pass it in a different way.
What should I do? The test is too difficult. There are always going to be ways to game it and the successful students do. Therefore the test gets more difficult and the students are forced to game it even more. As a teacher, should I tell my student that they can't go to Todai, or should I teach them how to game the exam?
As far as I was concerned, I decided that teaching English and passing entrance exams were 2 different things. I taught English for those students who were interested. I gave up a lot of my free time (a few hours every day) to help them do that. When exam time came, though, I taught no English at all. I taught them how to do what they needed to do to be successful. Every one of my colleagues did the same thing. Probably every teacher in the country does the same thing.
Standardized tests that grade you and decide your future are insane.
Imagine the next year's Todai entrance exam was totally different and unpredictable (say, transcribing an Eminem song or something similarly off-the-wall), then all the people who studied ways to game last year's exam would be SOL and only the people who actually know English would get in.
That's the kind of system I'm advocating for: get a bunch of smart test writers together with the express intent of coming up with novel ways to destroy the people who game the exams.
In my more cynical periods, I suspect that they don't mind people gaming the exams. After all, most Todai students don't actually need to speak English (or know crazy details of history, or write thousands of kanji, or whatever). They are more interested in clever students who are willing to study like insane people and practice boring stuff over and over again. That's what you need to do to game the exams. In a twisted way, it works...
Here's an example of a bad question: what is the area of the right triangle with hypotenuse 10 and height 6 (height is measured from the vertex opposite to the hypotenuse)? It does not test understanding. The optimal algorithm for the student is to parse the question looking for keywords - "triangle", "area", "height", recalling the formula, and plugging the numbers into it to get the answer 30.
The question is from the real test, and by the way, this answer is incorrect. Why it is incorrect is a better question.
Here's an example of a good question: prove the Pythagorean Theorem. You can't answer it by picking the right answer from the multiple choice list. You have to explain your line of reasoning to the examiner and be able to answer question like "why you say these two triangles are congruent". Such interactive process is much better at determining student's skill level, and you can't prepare for it by simply memorizing stuff without understanding.
The student might not like the proof in the textbook and prefer another proof. It take a human to understand whether this other proof holds water.
That would be far more convincing than merely claiming teaching to the test is real and can be done in all cases.
I'm sure teachers might do it if they could (shady people teachers are), but don't see any evidence they can.
Edit. I have been meaning to mention that I always enjoy your posts Chris even if I don't always agree with you.
More importantly, non-interactive nature of the test does not allow for follow-up questions. A few follow-up questions on oral exam let the examiner probe a lot deeper in less time, therefore getting a much better picture of student's ability and skill level.
I.e., you aren't just teaching the test, you are also improving understanding.
Now consider a question like "length of alpha, beta=2, gamma=3, a) sqrt(2^2+3^2) b) sqrt(2+3), c) 2^2+3^2, d) 7". Using your heuristic the student can rule out (d) and has a 33% chance of getting the right answer. The test reveals (averaging this over many questions) that they know a little bit, but nothing close to 100%.
Education should be taught in a way such that one is exposed to the methodologies by which one should approach problems, rather than solving the problem itself[1]. "Teaching to the test" is awful, I agree. How can one, on a national basis, ensure that every school is affording as equal an opportunity as possible? I'm not being sarcastic - what is the better alternative, the better metric by which we can gauge the performance of both our students and our teachers? I certainly don't think 8th grade math teachers should be penalized for the poor performance of their students if those 7th graders who matriculated into your class on September didn't have a functional understanding of fractions. (Conversely, I think teachers who reached 'tenure'[2] and have clocked out until they've hit that pension-year _should_ be punished). How should those teachers be evaluated? One could suggest "grade the teachers on the improvement of performance of their students compared to their prior years performance" but that brings us back to the same problem -- you have a test. I ask this without snark - how do you test as many concepts at that broad a scope of subject, at that grand a scale (let's start small - 3 districts in NY state chosen at random)? How do you even choose what concepts are important at that broad a level?
[1] This gets into a whole meta-discussion about what the appropriate methodologies are, which I'll leave as an exercise to those doctoral candidates finishing up their dissertations. [2] Tenure as I understand it serves to allow researchers could undergo extensive, perhaps subversive, study to enrich the human body of knowledge while retaining job security. While educators at a grade/secondary school level perform a great service, I fail (heh) to see the purpose it serves at that level.
Secondary infections in hospitals is a good metric. If you game it, it's still a net win for everybody.
Pass rates in schools is a bad metric. I can improve my metric by getting my lowest students to quit.
Rather than trying to "anti-game" a metric, simply observe if the metric is being gamed. If it is, the metric is a priori bad and should be removed.
That is exactly why you should have a review of how a metric might be gamed, before you accept to use a this metric.
And in medicine the problem is even more extreme, and comes down even more obviously in favor of more, not less, measurement. Here's an example:
http://www.nejm.org/doi/full/10.1056/NEJMsa0810119
Given that doctors in most other fields spend a ridiculous (by some metrics, 7:1) ratio of time doing paperwork or data entry vs. seeing the patient [1], I would suggest that there is serious opportunity cost in adding more self-measurements and checklists to most doctors' workload. Surgeons are likely an outlier that perhaps benefit most from being regressed to the mean.
[1]: http://well.blogs.nytimes.com/2013/05/30/for-new-doctors-8-m...
Now, it's certainly possible they're doing a whole bunch of useless paperwork. And it's certainly possible that there is too much in some areas - but that isn't a rejection of metrics and objective criteria. It's a rejection of the particular implementations of those things in particular places.
EDIT: To be clear, I think that what is actually needed is an overhaul of the tech. used to collect these metrics and fill out this paperwork. It's absolutely insane to me that doctors still use paper at all, or that they have to enter the same data multiple times in multiple places, constantly. That needs to be fixed. And they need to start collecting 100x more data on their performance with 100x less friction of collection, by upgrading their technology in this regard from what was available in 1970.
It would be a very interesting future where healthcare workers have the reams of stats that professional athletes do, and where everybody in the space is literate enough to intuitively understand each number and its caveats, like hardcore sports fans. For example, an ICU nurse is bound to encounter more hospital-acquired infections (HAIs) per week than a nurse in pre-op assessment, and so averaged cases of HAI per primary unit is the proper context for that stat, much like you'd weigh recent rushing yards for an NFL linebacker against the strength of the defenses faced. Then, managers could optimize their teams to each person's strength or weakness and/or supply training in the correct areas, much like coaches in sports do. This would be a sea change in the medical culture, though, and it would encounter staunch resistance, because some of the metrics already being pushed on doctors (# pts/day) arguably incentivize worse care.
To your last point, part of my research now is analyzing electronic medical record (EMR) data, and coming from any other field you'd be shocked at how messy it can be. In a way, this is to be expected, because the "vocabularies" for medical data are huge, and even widely used ontologies have bizarre properties [1]. I really hope that we can solve the data collection problems with better technology, but as of now, the incentives for EMR vendors to overhaul the tech is pretty low, now that most US hospitals have picked a system and those vendors have everything to lose and little to gain from overhauling their user interface.
[1]: http://www.healthcaredive.com/news/the-16-most-absurd-icd-10...
The simplest solution, which very often turns out to be the most effective and least expensive solution, is to ask those personally involved in a system to improve it, themselves...if they have trouble getting started offer incentives for them to do so...
That (first) step is all too often skipped in our "age of impatience"...
In short, I'm in favor of involving those most closely related (those in direct contact, those in the actual environment) to what is perceived as a "problem" in devising a solution, or solutions...versus having an outside entity recommending improvements that might, or might not, improve outcomes...
Improvement is often hard, and when it comes it comes in increments...sometimes those increments are incredibly small, but even marginal improvements contribute to the success of the whole...even without "metrics" I think most teachers know, or sense, the difference in student outcomes their approaches have led to...
I'll bet (although reaching a consensus is difficult, especially across academic disciplines) that each teacher, at the end of a course, has in mind some way that their efforts could have been at least marginally more productive and could quickly answer the question, "What should I do differently next time?" ...
The best teachers do this automatically...and it makes all the difference...
There is nothing about the experience that could be captured in these metrics. But I can tell you that she kept my wife afloat during the bad days and probably kept her from seeing another councilor. Ultimately we had a baby girl the second time around. The kindness and as cited in the article, love, that the doctor showed us really really helped us.
Personally, I've seen important programs, such as primary care-based depression screening and management that are proven to work then be turned into checklists and given to teams that are about checklist first, and less focused on mission/impact/caring. And I've seen patients respond that they are not helped. The therapeutic value of perceived empathy and attentiveness is unknown but should not be underestimated.
A manager once told me that before the employee performance review cycle he had already decided who was going to get what raise. He'd start from that and work backwards to derive the required "metrics" to support it. The right people got the raises, and the HR department was satisfied with the scientific process :-)
Word might get around about the boss, but when your job is on the line most people tow the line - as the Japanese say the nail that sticks out is the one that is hammered.
The problem with using metrics of what patients and parents think is this is not a good measure of what makes a good doctor or teacher. You can be an incompetent doctor with a fantastic bedside manner or a teacher all the parents relate to but who can't teach a fish to swim. The metric here is not the one you really want optimised.
On this topic I personally prefer doctors who have very poor beside manner because I assume they must have optimised competence over personal relations.
I think it's natural to assume this is not a good measure. Let's take teachers. I don't believe a teacher can consistently fool parents into believing they've taught the kid a lot when the kid is obviously not learning. And bluntly, the teacher is supposed to be working for the parent. Why shouldn't the parents decide who they want teaching their kids?
Edit: Why the down vote? Perhaps some argument? (Oh, of course, it was probably a bean counter)
I posted this as a very real comment - once you accept that what you are doing is an item measurable by parties that aren't professionals in your field (bean counters) then any qualitative aspects of the profession are lost. Only the quantifiable is left, and because the bean counter doesn't communicate using the language of the profession - you are forced to learn their language. A language of quantifiables and reduction to line items. The rise of the professional manager who has necessarily no expertise in the field they are managing must of necessity reduce all non quantifiables to a 0 value. Thus you end up in the pickle that educators and physicians find themselves in. In the past administration was performed by members of the profession and so they could communicate using the same language.
They hide behind the numbers, which they have rigged, to avoid taking any blame.