The problem at the core is not spyware and "room checks", they are just a symptom - the question rather is how exams are designed and why schools and universities go to such drastic lengths to prevent cheating.
Most tests are essentially nothing more than tests for the ability of rote memorization - they don't (really) check if the students actually understood the material and are able to apply it to new situations ("transfer of learning"), especially because it is way more difficult to create such tests and to fairly grade them. However, rote memorization tests are extremely easy to cheat on, which requires intrusive countermeasures.
In the end, this is a "common thread" across most of modern academia - the focus has long since shifted from taking in the most gifted of society for actual academic research to providing a fresh herd of fungible graduates for large corporations to prey upon each year, and it shows everywhere from how admissions are handled over the sad and sorry state of frontal teaching to, as seen, dehumanizing exams.
this has been my experience with college too... just a waste of time to satisfy the bureaucrats. They think that winning a silly exam proves anything but your willingness to waste your time memorizing answers.
I think I agree and I can't help but think that we (as a society) have our priorities backwards.
Who do these tests protect? Do they bolster the student? If so, why would they want to cheat? Is it because the information isn't relevant to their future and only the letter grade is? If a student cheats and then never encounters the natural consequences (like getting the same subject matter wrong on the job) - what was the point of the test?
The pessimist in me thinks that we're protecting industries. I would guess that an industry wants a plethora of candidates that meet a specific bar to choose from, and the information the students must memorize is inconsequential for the most part.
The point of tests in higher education is for the student to get an assessment of where they are. The whole design kind of assumes that students want to learn, not just get a piece of paper.
But that's the entirety of the higher education experience. It's all about presenting opportunities to learn and grow. People who treat it as mere credentialism are missing about 99% of the point. And yes, that includes a lot of faculty, students, and people pressuring students (to go either direction, really.)
I don't see a problem with tests. I see a problem with people treating degrees as some sort of magic totem.
> Most tests are essentially nothing more than tests for the ability of rote memorization
This a hundred times. If I had to hire someone for a job, I would rather hire someone who knows how to search online the answer to a problem than the one who repeats the book word for word without knowing what he/she is talking about.
Exams are crap like that because better exams would require the use of one professor asking non trivial questions for each student, which costs money.
> If I had to hire someone for a job, I would rather hire someone who knows how to search online the answer to a problem than the one who repeats the book word for word without knowing what he/she is talking about.
Do you have hiring authority or have you ever had it?
How do the candidates know _what_ to search? That's memorisation.
How do you get creativity, how are candidates better able to combine idea together? By forming each idea into a memory 'chunk' that can be handled as a single concept, and then working to combine these chunks.
If you hire people who can search google, then you get people who can search what is easily searchable, and who can create what has been previously posted online. If they somehow have 'internal' knowledge and capability, that's memory.
I used the wrong terms to describe skills like creativity and problem solving beside pure superficial knowledge. Of course I wouldn't hire anyone who can only search Google.
are any of these criticims backed by evidence? Rote memorization is one of the, if not the most important backbone of excellence in any discipline. Pianists play scales, chess players memorize and repeat, athletes do the same, virtually any excellent programmer I've met shares one trait, they program a lot.
frontal teaching to large audiences isn't dehumanizing, it's simply effective reflected empirically by the large amounts of excellent graduates coming out of countries that focus heavily on that style of teaching.
I've never met a world class mathematician coming from a hippie waldorf school, I've met plenty coming out of the Soviet education system.
> Pianists play scales, chess players memorize and repeat, athletes do the same, virtually any excellent programmer I've met shares one trait, they program a lot.
They practice applying the knowledge.
> frontal teaching to large audiences isn't dehumanizing, it's simply effective reflected empirically by the large amounts of excellent graduates coming out of countries that focus heavily on that style of teaching.
LOL have you ever had the fortune of dealing with these "excellent graduates" in offshoring projects? As soon as you go one tiny step beyond what they were taught in "university" shit gets nasty. Applying abstract knowledge to a specific use case is what is important in IT, and those who never had to actually understand the topic fail miserably.
Almost all knowledge is tacit, there's no clear line between applying and learning, it's a feedback loop. In fact obsession with abstract knowledge has always been weird to me. How far does reading about swimming get you?
The offshoring thing is literally just about that. The nastiness and cultural mismatch of that industry. The huge Chinese and Indian domestic tech industries show that they can build successful things, just not necessarily for foreign markets.
Rote memorisation is essential. Not _all_ there is to achievement, but as you say the other parts are difficult to test.
Without memorisation ("Just google", as many comments seem to suggest), you are severely limited in what you can come up with, and how you can contribute.
If you don't have foundational knowledge (memory): 1) you don't know what to google; 2) within a discussion concepts will go over your head so you can't contribute; 3) you don't have the components to play around with and mix and match in your head to generate new ideas.
If your work is dreadfully formulaic, sure, you can probably get by with limited foundational knowledge and just searching. If your work in any way is knowledge work, then you need _knowledge_, not just access to a vast reference.
> The problem at the core is not spyware and "room checks", they are just a symptom - the question rather is how exams are designed and why schools and universities go to such drastic lengths to prevent cheating.
You got the premise wrong. The main way that students cheat in online test is by communicating with other students/tutors and by having other people take the tests for them.
Doesn't matter how perfect your exam is if the student has a friend coaching them during the exam and telling them all the answers.
And it is far more common than you think. I have been tutoring for a for 5 years and there was a massive amount of students who tried to hire me to do online exams for them during the 2 years of covid. I probably would have done it if I was still a student full of debt at that time.
It is terrible, but what is the alternative to prevent cheating? I had quite a few on-line exams by now and I've seen very creative attempts at cheating so far.
I've never had one, but from what I've heard they're the kind of thing that if done properly ends up being a better way to test people, but they're harder for instructors to implement.
I guess so. Most of the tests I;ve taken at college were the same tasks as in previous years, just with changed numbers.
... or not. That also happened.
Because most of teachers didn't teach online class before. They are not used to design exams that would not be way too easy it suddenly becomes open book.
I won't even say they're "lazy", coming up good take-home exams is super hard, especially when you spend your entire career not need to do that.
> if people want to cheat they can find some way to cheat regardless of any countermeasures you deploy.
I agree if it's online exam, it's kinda doomed either way. But it's more to act as a deterrent than total solution. And it does work to some "less-adventurous" students.
Note: above is just directly replying to your questions. I don't support scanning student's room.
Come on now, some of the college instructors weren't even teaching the offline classes. 200 students in a lecture hall and some random TA grading and assigning the tests. Do you honesty believe the TA knows each and ever student and that someone else didn't show up on exam day?!
I'd guess that people who are willing and able to afford hiring someone to cheat for them in an exam, can probably also order a fake ID with their name and the photo of their exam-taker.
Though maybe an extra step of storing a copy of the ID with the exam entry and having a professor who knows them verify that it matches the real student would catch most of that...
> if people want to cheat they can find some way to cheat regardless of any countermeasures you deploy
I was a Academic highschool teacher for 8 years and I agree with your statement but also don't think that we should make it too easy for them. At minimum a screen recorder so the students know even if they were not caught during the exam, they could later be caught
Bonus anecdote
When the first bluetooth phones hit the market we had students use their (pseudo) smart phones (that looked like expensive calculators with greyscale displays back then) to send each other the solutions during the physics exam. No teacher at the time had this as a threat model
At some point, when a student find a truly novel way to gain an advantage, that can be seen as a kind of creativity, and may point to a strong career (hopefully within the law).
However, if lots of students cheat in the same way semester after semester, the school has a credibility problem.
Take home tests that are difficult enough for the questions to not be easily answered on Stack Exchange or paid sites. This is fairly easy to do in upper-division undergraduate courses as well as graduate courses. Not so easy for lower-division courses. The closest I've seen for a lower-division class was a professor who made as much of the exam content based off examples from lecture material, but I always saw that as a way to punish students who ditched lectures.
Students and teachers have been cheating before pencil and paper. Do you ever wonder why some college students have near-perfect SAT scores yet can barely rub two English words together?
No, it is not acceptable that we have to give up privacy for some stupid exam cheating problem. There are going to be lots of problems and inconveniences like this. We have a legacy of 250 years of constitutional protection of privacy.
Solve the fucking problem without violating privacy. If it doesn't get solved, fine, let it be. Accept those consequences and live with it.
The best way to prevent cheating in an online test is to set a strict time limit. That way its harder for them to look up the answers on Google etc. The problem then is that a student may experience technical difficulties out of their control. For cases like that, students should be able to appeal for extra time.
This might work for things you have to remember off the top of your head, where you have e.g. 10 seconds to write answer, but it doesn't really work when you have longer questions requiring some kind of working which take more than a minute to work out.
I also think that the "extra time for technical difficulties" is going to be very hard to prove or disprove and would lead to very arbitrary decisions.
They already do this with some tests, both online and in person. If it isn't generous enough, it generally harms folks that are being careful with answers while cheaters can cheat. The cheaters just have to be prepared.
Design better tests. If you can "cheat" on a test by just googling things/looking at notes then the test isn't testing anything but rote memorization anyway.
A math professor (granted easier with math) at university always said, you can bring anything you want to the exam, if you can answer it with help he made a mistake designing the test.
I see the same when I interview people online and I hear them frantically click their keyboard - but no help, I know before what Google returns.
Trying to evaluate a software engineer without letting them use google strikes me as being similar to trying to evaluate a construction worker by making them dig holes with their bare hands.
I'm not asking things that can be solved by Google [1], because they can be solved by Google, and I only gain the knowledge "Candidate can Google", I see this as given. It doesn't mean that not a lot of work at the job involves googling Stackoverflow - I do it myself all the time.
[1] I don't ask API knowledge, if someone doesn't know some API I tell them just to invent the call they need.
Best technical interview I ever had, they sat me down in front of an internet-connected laptop, gave me some problems on a piece of paper, and said "we'll be back in an hour, do your best". Just left me alone with Google et al.
(Then I spent an hour-ish talking to swardley about 3D printing and its future impact on IP law, etc. -- this was back in early 2007, before it was generally available. That was the weirdest non-technical interview I've had.)
While preparing for a certification, I had someone DM me on the prep forums and offer to ship me an in-ear monitor and physical HDMI mole to coach me in real-time. Cheaters are nuts, there are few test formats that would survive an attack like that.
Exactly, I strongly believe that any test at a secondary education level should be open book. We should be testing for reasoning and capacity to apply knowledge not rote-learning.
It's also why I believe that in most cases multiple choice questions have no place in any well designed tests because it's extremely hard to create multiple choice questions that actually test the application of knowledge versus the regurgitation of rote learning.
When I was in university (grande école) in France, luckily all tests followed that format, they were open book, very few questions that needed deep detailed answers and were about interesting problems. Even finding tests from previous years and doing them was actually fun (tests were never reused).
I did an exchange in the US at RIT and most tests had very boring rote learning questions. For example in a computer graphics class: Which company were voting members of the OpenGL Architecture Review Board? Which year was the first OpenGl specification released?. I still remember those questions because I was shocked by how utterly stupid it was to have those on a test but, in general, 25 to 50% of the points of any tests was on those completely irrelevant, useless questions that anyone can google for.
Cheating is a problem for universities that never had proper testing standards. But that's a symptom of incompetence on the part of professors and of universities not making the choice to ensure that tests have to be well designed.
I've seen some very hairy multi choice questions in Math/Computer science (think testing order of operations, which function gets called due to complicated polymorphism etc). I think it depends on the subject.
Although I also encountered a few where multiple answers were technically correct. Then you have to guess at which answer the professor thinks is "more correct" which kinda sucks.
Yeah, another decent alternative I've seen was a test with a few multiple choice questions where each question needed quite a bit of reasoning and where you had to give the reasoning to get the points. The multiple choice question was a way for the teacher to quickly know if you had the correct answer and for the student to verify their answer against possible answers and see if they might have made a mistake but the teacher would still need to review the half a page of reasoning for each question.
It can be done decently but it's much easier to write and much more common to see bad multiple choice questions.
In one of my Math/Computer classes in college, part of the class was that along the way you were supposed to be implementing the mathematical concepts in code along the way, so that you would have your code available to run on the test questions when it reached the day of the test! Whether you programmed on a TI-89 calculator or your laptop was up to the student, but the tests were open note and open device and it felt very freeing. The actual methods of input and exact algorithms were of course carefully picked so that there would be no easy off the shelf code solution, but once you understood it the problems could be trivially implemented in any programming language with a BigInt type
It's not just googling
. It's communicating with others. In a no checks environment you can have an a la carte team of paid consultants and helpfull friends solving the test with you.
Bingo! [Surprised I had to scroll so far to find this.] You can discuss the merits of rote learning and test design all day, but cheating exists within any pedagogic paradigm. Back in my day, most cheating was between students doing each other favors or taking a small commission.
The way some teachers solved this was a 1:1 oral complementary exam, where students were asked to explain a random set of their solutions. This is drastically more difficult to cheat in real time, and you will almost certainly look like an idiot if you come unprepared. It's a stupidly simple solution, disincentivizing cheating through shame. Doesn't need any proctoring rootkits and rectal cameras, a simple identity check is enough.
One of the bests I had written in university allowed you to bring every non-digital material you want to have. All slides? The books? The former exercises? No problem! The test was designed to be built on 2 primitives:
1. If you need to lookup stuff in your waste of material you won‘t be able to finish everything. The time constraint was very hard.
2. It was not just repeating former things, you had to have understood them. Everything was a transfer knowledge exercise.
> If you need to lookup stuff in your waste of material you won‘t be able to finish everything. The time constraint was very hard.
Did you enjoy this exam? I am asking as I had a very similar experience in my undergrad:
Our university allowed some basic courses to be studied at different departments, if they were equivalent. This means that there was a CS Major Algebra 1, but also Math Major Algebra 1, and you can take either.
I took the Math Major Algebra 1, being a CS student, and I have passed everything up to the final exam, where I learned that the exam is literally what you describe. All the proofs and arguments needed follow from basic principles (as things in Algebra 1 tend to do), but there is a lot of questions and the time limit is strict.
Thus, you have no time to actually do any deliberate thinking, you have to memorize and understand the basics so well that you can develop the arguments essentially in real time.
(And it was not an open book exam, even, but as you describe, it would not have helped you too much.)
I hated that exam. I never took it out of principle, instead just quitting the course and doing it with a great grade a year later, in CS. I still dislike the idea of forcing knowledge acquisition through strict time limits. What if somebody is smart but has a slow start due to stress? Ugh.
> If you need to lookup stuff in your waste of material you won‘t be able to finish everything. The time constraint was very hard.
Reminds me of my school days. Tests with hundreds of questions and less than a minute to solve each question. They were essentially testing the student's time management skills as well as their physical and mental fortitude in addition to their knowledge. Extremely exhausting and anxiety inducing.
Students typically don't cheat in online tests by looking at their notes. What they do is ask other students or have someone else do the exam for them.
Yes, lots of testing at Pearson VUE. Many State Licensing certificates and IT certification testing is conducted at Pearson Vue. Military and government agencies also do applicant testing at Pearson VUE. Candidates seeking to apply to Secret Service show up for Pearson VUE testing center, along with military recruits and folks needing to update Cisco certificates and college students that need to take a test here or there.
I was so happy when the pandemic hit and the second part of a qualification I was doing could be remotely proctered - everyone from professionals to crack addicts were taking tests of some sort at the same time at the test centre.
My concentration broke when the crack addict picked up the monitor and smashed it on the ground and stormed out, im not sure it they were doing a Cisco qualification or a driving test, but I dont think it was going well.
Of course there are other reasons they might act that way, but "crack addict" is a sometimes just shorthand for a person who's acting and looking a certain way.
I see this as a good thing. If you can self-learn remotely at your own pace and then show up once for a brief proctored (and by extension trusted) exam, it’s still far cheaper and much more accessible than the traditional college or high school alternative.
None at all any more, but it would force everyone to a revolutionary project of re-thinking half of how our societies work - and there are a lot of people with deep pockets and heavy interest in keeping the status quo.
1. You can't look for information if you don't know it exists.
2. Most standardized tests aren't used in the expectation that the testee's knowledge will matter in the future. Instead, the idea is that people who score higher will be generally more competent at whatever it is you want them to do.
>2. Most standardized tests aren't used in the expectation that the testee's knowledge will matter in the future. Instead, the idea is that people who score higher will be generally more competent at whatever it is you want them to do.
I fail to see how information retention maps to skill in this case. I guess it could be a vague indicator of the amount of practice done, but some folks memorize so easily, it makes this system incredibly easy to game.
From my experience, people who score extremely well on standardized tests do amazingly well in professional/knowledge-based fields. Of the top scorers in my high school class, many are not just doctors, lawyers, engineers, and business-people… but they are _top_ doctors, lawyers, engineers, and business-people. When I hear about those students through the vine, it is like “of course they’re successful - I would be surprised if they weren’t” sort of feeling.
> 1. You can't look for information if you don't know it exists.
You absolutely can. I don't know if there are stats on the number of people who have circumnavigated the globe in an amphibious vehicle, but I can Google it.
I don't know whether the thickness of an electrical cable influences the electromagnetic field it creates, but I can Google it.
You can even Google generic stuff like "do charged particles in the atmosphere exert any force on objects at ground level" and get a survey of the forces.
That only holds true if you assume people are making very specific queries in an unfamiliar domain. It's entirely possible to start with very general queries in an unfamiliar domain and hone in on a specific piece.
It's common in software engineering. I don't understand how databases optimize queries, but I can start with a broad query like "what factors does Postgres use to optimize queries" and slowly hone in on whatever is making my query weird/bad.
Ranking and sorting, much like school. Testing on a standardized body of knowledge allows for sorting on intelligence and conscientiousness and subject matter knowledge.
Last time I had to take a certification test, I chose the test center over doing it remotely since I didn't want to install whatever spyware they wanted me to install on my work laptop.
We are definitely in the period where we are seeing the limits of scalability of current technology. Google et al. eliminated customer service, schools are educating on a massive scale, etc.
We always have the option of oral onsite exams as has been custom for many years, and accept that education does not scale excessively.
We can also keep developing surveillance tech and other AI solutions to the problem and handle the ethical issues.
Or we could accept that in this day and age, expecting kids to recite factoids broadcast to them by a "teacher" doesn't make any kind of sense any more, with google in everyone's pocket.
And testing in the typical style that fails when a student can use a textbook or google makes just as little sense.
This is a common rebuttal but despite the fact that everyone does have google in their pockets, clearly a lot of people never use it to learn and only use it for entertainment so testing what they know off by heart does still matter imho.
When we locked down in the UK, millions of people were furloughed for 6 months+. How many of those people retrained, studied etc.? Not as many as we might hope because education is firstly about motivation and secondly about information.
The rebuttal is not aimed at „everyone can learn“, but at „you can look up things that you required to memorize two decades ago.“ I can look up any important constant to hundreds of decimal points, find explanations for almost all publicly known algorithms, way more information than I could ever memorize. Tests often still pretend that knowing this is important. But what‘s more important is knowing what to look for and how to transform the general knowledge you can read into a solution. That’s a skill and it needs training and it‘s the skill that‘s important and should be tested for.
But those tests are harder to design, harder to grade and you cannot reuse them - since someone will just post the solution on the internet. And you need to prevent students from just sharing the solution during the exam.
This is definitely a discussion I will leave to those researching teaching.
That said, for many years I shared the view. Some time ago a convincing argument for "memorization" was presented, and I am slowly changing to the view that it is a reasonable way to learn and to become more creative.
I challenge you to do wny intellectual task without knowing the basics by heart. For example, try writing a large program in APL today - you can easily Google the symbol for any function you want.
Memorization, while not sufficient, is an absolutely necessary part of learning. No amount of googling can help you become skillful at something.
Note that searching for new information or old information that you may have forgotten the specifics of is also perfectly normal, and an important skill in itself. But even there, Google will usually be more of a tool for recall (what was the order of those two parameters? What was the name of that substance that they use for liver issues?) then completely new fact finding.
For example, if I ask you what is the complexity of finding an element in a red-black tree, it would be OK to quickly Google the speicifc answer. But if you have to Google what complexity means, or what a red black tree even is, or if you're not sure whether the complexity is better or worse than searching a linked list, then it's clear you don't have working knowledge of red black trees.
> I challenge you to do wny intellectual task without knowing the basics by heart.
So, in other words, you propose a test. Presumably it's open-book, since you are testing an ability to apply knowledge and not an ability to merely remember it.
This sounds less like a rebuttal and more like a validation.
The GP comment is specifically referring to "reciting basic factoids", but they are the "basics" that you need to know by heart to manipulate them.
While testing for "reciting" them is certainly suboptimal, it's cheaper to prepare a test of the sort than a test where you are effectively putting that knowledge to use.
A balanced test would do a bit of both (so it does not take ages to prepare and grade, yet ensures you know the basics and can apply them in some problems).
The problem with that is that our brains work a lot better when it finds patterns - when it recognises things. And for that, you need to know these things.
We have access to a lot of knowledge at our fingers, it's true. But if the data is all stored there instead of our brains, then we won't be able to recognise that fact A (which we just read) is similar to fact B (which we read a long time but didn't bother to remember because we could just look it up anyway). The implications of that depend on the context, of course, but in a general sense, I think it'd slow our development as a species down.
Exams are generally supposed to test your understanding, not your recollection abilities - with some exceptions, of course, but in general that's supposed to be what they're for.
The irony is, Paul Lockhart could only see the beauty of geometry -- and thus write A Mathematician's Lament -- because he learned his times tables and other fundamental mathematical techniques by heart. Abilities that can be acquired only through drilling.
Drills and memorizaton aren't the only thing in education, but they are key.
You will eventually learn the times tables by heart of you need to multiply enough. Same goes for a lot of fundamentals. Practice is more of a key than rote memorization.
In university, I was notoriously bad at "remembering" things (iow, I was lazy to learn things by heart). I was pretty good at making connections, and I could prove theorems from only knowing a couple of definitions and simpler theorems in advanced math courses. I generally scored average in tests, but I rarely failed (though I occasionally had "original" solutions that teachers commended on even though I failed to reach maximum scores).
But I was never under the illusion that this constitutes working knowledge. I was able to come up with solutions for some problems only because I had some basic knowledge memorized, which allowed me to effectively manipulate ideas and concepts which I did know of. Even then, I used to semi-joke how the hardest thing in theoretical math is all the memorization required.
If you've ever explored any scientific field deep enough, you'll know that it's festered with tricks found by accident and roundabout ways to get to a solution that, if you haven't seen it before, you have no hope of applying in practice.
Or you might think that consulting a book or two (or Internet) will allow you to prove Fermat's theorem while you are being tested for 2h.
Well... I have deeply explored a scientific field (in computer science, not math) the way you are talking about. By reading other's research papers, implementing what they proposed, trying to improve upon it, coming up with novel solutions etc etc.
Very little of what I used on a day-to-day basis came from rote learning, a lot came from the deeper understanding that I got from actually actively engaging with the concepts.
I'm not proposing doing away with schooling completely, but the model should change (much in the way more progressive schools are doing anyway, just a shame that not many parents can afford to send their kids there).
> Very little of what I used on a day-to-day basis came from rote learning
The question is not how much non-rote learning is required to adequately perform any task, but whether rote learning is a necessary component of it.
By engaging with research papers (often requiring multiple readings), you "accidentally" also memorize a lot of the material (even if not for ever and ever), which makes it possible for you to "try to improve it" etc.
So all I am saying is that rote learning is a necessary component of applying any suitably advanced material. Whether that happens organically by engaging with the material deeply enough or on purpose by simply rote-learning is orthogonal to the need for that knowledge to be there.
At issue here in this thread is whether testing for such knowledge is signalling anything of value. I still believe it is because you'll have that knowledge with either approach, even if only one of those approaches will gain you more functional knowledge: basically, such a test is a proxy for you having engaged in the material.
Which is why I suggest a mixed approach in a sibling comment: test for knowledge of "factoids" to simplify test development and application of such knowledge where you are required to manipulate "learned" concepts (so only part of the test is hard to create and keep consistently hard/easy).
it's situations like these that make me think perhaps we shouldn't be scaling quite so large. That is to say, quite so high.
Scaling past a certain size puts you kind of further and further outside of what your environment can optimally sustain. Think if a fish in a bowl never stopped growing.
Add to this the particular challenges that orgs with centralized decision making face when scaling (inertia, homogeneity, single-point-of-failure) and it becomes really hard to see the current big-fish trend as even remotely sustainable.
Perhaps only semi-related, but I like the gore-tex model, myself. Scale out and shard.
As a mid-career engineer, I had to renew a professional certification during the Covid-19 times. That recertification was set up as a proctored online exam.
After 15 minutes of removing random stuff from my desk, under my desk, on my walls, including but not limited to disassembling a wall-mounted TV set, as instructed by the proctor, I gave up. I rented an office at a co-working place that was sterile white. Then they tried to accuse me of using camera-modifying software to add a background, because "no room is that clean".
If we do keep up this proctored test-at-home system, we need to have a workable way to make sure the testing room still works out as a living space after the test.
I don't believe it can be "you can take the test remotely if you let us search the room first" (that judge put an end on this, so it cannot anymore at least in the US). It is impossible to prevent cheating in this context, no matter how hard you try. Their paranoia in your case was foolish.
What they could do is letting people take written test in a lightly controlled environment (just a webcam with audio, no 360/scan before tests), and then complement it with a Q&A part. Yes, that's probably more time consuming and more expensive.
Can someone please describe this "scan" technology? The article didn't.
I'm picturing a swarm of small hovering drones sweeping their blue laser beams over your bedroom office, before one of them finishes with an accusative scan of your retina.
Before the test starts, you are directed to move the camera around the room, behind your chair and things like that. The Pearson proctor made me remove paintings from the wall to prove there was nothing behind them
It's hard to believe that a suitably motivated college student couldn't figure out a way of getting around that kind of scanning - what about papers in books, info on phone screen, even someone holding up posters at the window.
It's probably just a faculty asking the student to make a 360 with the webcam, according to what they did in my country.
That procedure is full of holes, though. There are so many ways to cheat when you don't take a test on-site, that it's not even funny. Those tests have to be performed with a computer on to begin with. I've heard that in some cases, there was testing software that would scan the computer too, similar to games' anti-cheat devices.
Schools first implemented that procedure in a hurry then stuck to it, and they thought they could violate privacy "a little" because states and countries themselves adopted exceptional laws that limited human rights. That felt wrong to me when it happened, and I am glad that at least a judge in the US confirmed it was wrong.
What is the value of standardized tests in todays society? They measure only one thing: how well one can perform on an arbitrary test. They have no meaningful predictive value. Verdicts like this one make them a thing of the past? Good riddance.
This has been eluded to by a few other comments already, but I think it's worth stressing: Maybe the problem is that our definition of "cheating" doesn't make sense anymore.
Our methods of teaching come from a worldview in which the information stored in your head is your only resource. That wasn't true in the past where you could consult colleagues, look things up in books or call experts - and it's certainly not true now that everyone carries the entire knowledge of humankind around in their pocket.
So what is the point in artificially creating a test situation where the student is isolated from resources she'll have at her disposal in real life? How about instead, we create (whatever the plural of curriculum is) that teach you how to use these resources efficiently and test you on how well you perform using all the tools that will be at your disposal in your actual job as well?
> So what is the point in artificially creating a test situation where the student is isolated from resources she'll have at her disposal in real life?
I can envision many scenarios where a person needs to be quick on their feet and does not have time to look things up. Drivers, pilots, doctors, etc. Certification provides a lot of value to society.
The other point of testing is to separate wheat from chaff. To see if someone is disciplined enough to learn the material or see a task through. There are issues of going beyond the ideal amount of testing here, but not totally useless in my opinion.
True - and there's certainly a line to be drawn between core knowledge and wider knowledge. A programmer, looking up what an if-statement is each time won't do well, regardless of the resources at their disposal.
But I'd argue that realistic test conditions correct for this automatically.
If your test is "write a program that does X within two hours" the programmer lacking core knowledge will fail.
If your test is "recite 20 programming paradigms from memory" you will have to check for "cheating"
Those scenarios can be better tested through simulations. E.g. to pass a driving exam you are presented with a road situation, and given only a few seconds to respond — this completely eliminates the possibility of looking things up on the internet. A similar approach can be used for other disciplines.
> separate wheat from chaff
Wheat can demonstrate/use the knowledge, chaff can't. Why does it matter whether the knowledge is internal or external, assuming results are the same?
Because you do not know if the results are the same before making the decision to go with person A or person B. There are time costs in figuring out if someone can do something or is capable of learning how to do something, so test(s) of various kinds are used as a proxy.
Perhaps tests are to make sure that you are a going to be colleague worth consulting then? Yes, pure memorization and recall of facts is less valuable now than before, but knowing how to apply the facts available to you is still a skill worth having, and similarly worth testing. Those skills can also be assessed using written tests, and the potential for cheating is just as on memory tests.
what is the point in artificially creating a test situation where the student is isolated from resources she'll have at her disposal in real life
Let's take language learning as an example: do you consider a foreigner with a dictionary to be just as expressive as a native speaker? Perhaps there is value in testing the immediate knowledge of the student rather than their lookup skills?
I want to test their ability to perform a task - rather than how well they memorized things. Which is not to say that memorization is not important, but it's just one piece of the Puzzle. So when it comes to language tests, rather than having the candidate fill in prepositions into blanks, let's have an actual conversation with them in which they apply their combined skills.
I actually adhere to this philosophy when hiring candidates as well. Rather than having them explain how to reverse binary trees on a whiteboard, I have them sit down and do a mini version of the actual task they will have to do in their job later. And, unsurprisingly, knowing how well they do the actual task is a much better indicator of how well they perform at doing the actual task later.
I understand what the halting problem implies, what complexity is, what Turing machine means and why they are an abstraction for algorithms, and the basic common memes of programming such as using a stack and a heap, IEEE 754 floating point numbers, 2s complement binary numbering, and so on and so forth.
Yet I don't know a lick of Go. If I were to open up a Hello World tutorial for Go during the interview, would I honestly still have a shot?
I think people who are really into "do the job for an hour" interviews don't understand this. Some skills (particularly programming languages) are just really easy to learn. I learned go in a week for a consulting job. If I had to do an interview in go, I would not have gotten the job.
I think this is a good philosophy for hiring employees, but when it comes to education, we wish to go beyond training employees, and what your capabilities are is not the only thing that matters. We should test practical capabilities, but we should test more abstract skills like how many facts you've ingested as well.
When I tell people a story about the history of the french revolution or something, I don't have the opportunity to go research it beforehand. I've already done that research, and I've already thought about those events for months or years, which gave me the opportunity to synthesize an insight or opinion I wish to share.
You don't get that if you don't know things in your head.
> Let's take language learning as an example: do you consider a foreigner with a dictionary to be just as expressive as a native speaker? Perhaps there is value in testing the immediate knowledge of the student rather than their lookup skills?
You may be well aware of this, but you've essentially converged on the classical Chinese Room thought experiment [0]. It's usually applied to the philosophical question as to what does and does not constitute intelligence in the context of AI, but it's interesting how it is also applicable to what does and does not constitute knowledge in the context of learning.
Words are rarely restricted to their denotative (dictionary) meanings. There are often abstract, connotative meanings attached to them by speakers. For example, consider the following list of words.
Hello. Hi. Hey.
These words all have the same basic denotative meaning. They're all a greeting. Yet, there is a subtle connotation of formality which is rarely described in a dictionary.
These types of connotation are not often understood by beginning language learners. It's something they acquire with time and experience in using the language. While a teacher or search could share about the formalities, the language learner is still going to require time in applying those words in different contexts to fully understand these subtleties.
This could be done with tests, by having open book tests.
The point of a test is to check if you are capable on your own, and not piggybacking on someone elses intellect or hard work.
It is true, that in the real world, the resources available to use are infinitely larger than a classroom or a teaching setting. But your idea springs from lack of understanding about the true purpose of education, i.e, to groom children to think.
Alas, modern education has become a cesspool of rote learning, atleast in my country, India.
I believe the issue has more to do with the accreditation of learning institutions. “Untimed power tests” are the gold standard for testing, and in the modern world this includes own internet access. Everyone acknowledges that these are better at assessing knowledge, critical thinking, etc.. The problem is that these tests are very difficult to make and logistical nightmares concerning cheating. A timed, closely proctored, closed book/internet exam is much easy to create and administer, and it halfway (better than nothing) assesses whether or not the student has learned the material.
A big part of the valuable experience comes from having memories of similar situations, and thus a shortcut to the solution. In IT it can be similar to “indexing”, like I may remember reading a relevant stackoverflow question and I know how to find it again, but I don’t remember the exact solution.
But the catch is, to index something you first have to crawl through it. In medicine for example, there may not be much point in holding every single extreme-rare disease in memory, but having memorized them once and mostly forgotten them will lead a good doctor to know about it enough so that he/she can look it up on the odd chance it gets relevant. It will be only possible when he/she memorized it once back then.
My point is, there is a point in memorization (especially that it improves memory all-around - one will remember facts much easily after extensive learning), what should be better distinguished is facts that are to be learnt for life vs those that are only needed for a given test. It’s like unknowns vs unknown unknowns - the latter can be much worse and you don’t want someone like that as a colleague.
I'm going to claim that memory is only one part, maybe 1/3, of what I would like to examine for. Certainly in a job interview, but I also like university exams that include the parts below.
There's another huge part, *problem solving skills*. Working long enough to figure out which pieces of knowledge that you have apply to the situation.
There's another part, *making it work*. Software engineers deal with failures in practice (debugging) as well as theory (don't know how to do it). Sadly but interestingly, sometimes this is the point of failure for a student/examinee when they possess plenty of knowledge but don't know how to use it.
Most if not all of my upper classes in school were open book. Even my beginning CS classes for Java was open book. There was no way you had enough time to finish the tests if you look up every answer. You would have to learn topics on the spot if you didn’t study, it was near impossible to do with the time constraints.
Most of my upper classes were heavily weighted towards group projects anyway.
Being able to look things up, even in seconds, is not equivalent to knowing them. When someone suggests "let's do X", you cannot reply with "but that would cause Y", if Y is buried in some pdf file on your computer, or one Bing search away - how do you know you should search for Y, when there are infinite other things you could search for.
That's one of the main points of The Shallows by Nicholas Carr and I think that I agree.
Long term subconscious memory is where a lot of the connections happen. If your long term memory only knows where to find things without knowing them then that ability is stunted.
The way I used to write/set-up tests when I was teaching is that you could use a limited set of online resources to answer, but if you have to search every question you won't have enough time to do it. So it also tested your ability to quickly locate the information you need should you not know it right away.
Back at university one of the first things we were told about being an a good engineer was: "You don't need to know everything, but you do need to know where to look it up".
It's a statement that I agree with. There is just too much knowledge to know it all. These days there is a very real skill in knowing how to search i.e. a second order version of the above statement "you need to know where to find out how to look this up".
There's a HUGE difference between looking something up to reactivate sleepy memories and learning something new for the first time.
There are some really technical things ive done in the pasy that I couldn't do without a reference right now, but once I've got the reference I don't need to be reminded about low level details or edge cases or gotchas - it's all still there in my head I just need to see the road map and it will all come back to me.
Yep, my dad used to sum this up as "All Engineering school does is teach you where to shit." I guess I may have gone with "where to eat" but he had a flair for the dramatic.
Underestimating the amount of knowledge required or of importance to know (yes, even when google exists) is just another form of the infamous Dunning-Kruger.
Google’s search mechanisms and rankings have changed a lot over my lifetime. I don’t usually trust it to find the right answers to problems in my field unless it’s a very well supported technology. I don’t agree that “knowing how to Google” is a skill in its own right for this reason. Hell they even claimed not long ago they would fix quote searches but they seem worse than ever.
It depends. I haven't yet in my career come across something that I couldn't research online because I hadn't been taught it. Usually the first step is to discover which words to use, reading around the domain. Once I have the words I can usually find out everything I need to do my job.
Knowledge is so easily acquired these days that, while useful, it is not essential. I hire people with zero domain knowledge for that reason.
I see people struggling to find the correct initial keywords to search on all the time; and I have the same problem trying to google domains I don’t know. But this can usually be solved with sufficient effort, though I don’t know many people who are willing to put in that effort.
But more problematic is when people don’t know there’s a problem to google in the first place — it seems to do the job — which is usually resolved by both education and experience, but is ultimately a result if they don’t know what they don’t know.
The knowledge is generally out there, but you need to know you’re looking for it, and it’s only in the common/trivial problems that it’s so obvious that the compiler/tooling/customer just flat-out tells you you’ve missed something
It is a lot faster to have things stored in your head than to look them up. You can just put a time limit on the test and tell people that they can either spend a year before the test starts learning the material and getting it into their heads, of they can try to learn biochemistry from scratch using Google after the 40-minute timer for the test starts.
The thing is, if you search for something enough you will end up learning it. It's a natural way to learn things.
This is how people become software engineers these days. They _constantly_ look stuff up, watch videos, and learn by doing that. Yes, you can still go to college and take tests that require you to "know" things, but honestly the vast majority of what I "knew" in college turned out to be short-term knowledge stored for tests.
A person can certainly become a software engineer without a college education, but I worry that constantly learning things just in time for using them will give them disconnected snippets of knowledge in a lot of areas rather than a good concept of how the underlying system functions. I guess it depends on if they are studying computer architecture, complexity theory, and whatnot in those videos or if they are just copying snippets of code from SO.
I learned by doing, initially, because I started learning programming with minimal resources before going to college and had to unlearn or restructure a lot of the knowledge I had gained up to that point once I did. I would probably have gotten there in the end, but it would have taken me a lot longer. The world was different then, though. Instead of easy access to info before going to college I was lucky to find a book somewhere that someone threw out that talked about things in a peripheral way that I needed to know, or ran into someone I could discuss things with that were as info-deprived as I was.
That's where my skepticism of this approach comes from, I guess. But I didn't live in a world where knowledge was at my fingertips from a very early age, so who knows?
When someone suggests "let's do X", you cannot reply with "but that would cause Y"
This is reasonable. E.g. sure a person can google how to build a house, but also sure that at planning phase they’ll mess and conflict with a half of good rules of thumb. Or forget them right after reading, because there is no habit nor knowledge in their head, and all that info is pretty transient. Pro is not the one who has read an article, but the one who can tell a couple of their own notes about it.
Otoh, gp is also correct that tests should check how you are doing in a real life situation. So maybe use both ways of testing?
Right but you shouldn't be tested on things that can just be "looked up". You should be tested on your ability to apply what you know, and teachers should be taking the time to write globally unique problems for every single test (gasp)
This works for concrete knowledge (declarative, procedural) where simply knowing a set of facts or a pre-existing process is sufficient to perform the job. But when you most into more abstract, creative work, this idea falls apart quickly.
You can easily find an answer to "how do I enable file sharing on Windows." You cannot easily find an answer to "how do I design a website for an engineering firm which wants to increase it's inbound leads by 25%."
The former is a simple search away. The latter requires the integration of numerous abstract concepts which the shallows of a search engine cannot return.
Further, such an approach also limits our ability to create new knowledge.
In online tests, students cheat by asking the answers to other people online (eg. chatrooms with other students in the same class) or by having someone else do the test.
Teachers don't care about students looking into their books/pdf, what they want to prevent is them getting the answers from a snapchat group.
As a physics student in college, almost all of my (physics) exams were open book to some degree. Some were even take home tests where I had a few days to complete it, and was allowed to use software like Mathematica and Matlab.
That sounds nice in theory, but how would you write a test for, say, basic physics, when every type of question can be Googled? Even if you make brand new questions, it's still just generally available categories of problems, like inclined planes and stuff. It's even worse with algebra, because there's software that can both solve it for you and show the steps (which you can then copy onto a test).
Unfortunately, in order to progress to more advanced topics, you need to understand the basics yourself. It's not sufficient to just be able to research it online. This is true in every field I can think of. That's why you can't just give people online access for every kind of test.
I had a basic physics teacher that allowed notes and gave us a chest sheet of his own which literally listed equations, described variables and explained the situation you might want the equation. People still failed. In a basic thermodynamics class the teacher said it was open everything but the person next to you: laptops, internet, the book etc. For fun I took the test without either book or laptop and did just fine. Others found it hard. In both cases the problems required thinking about the situation posed and which basic principles should be used to evaluate the problem.
In my opinion, cheating from internet access or outside resources is not the problem, cheating by gogling exact problems is.
I agree. We had open notes / open internet (no communication) exams at engineering school and they wouldn't help you pass the test if you didn't know the material. Maybe you could spend all exam period solving one of the questions if you weren't prepared but you'd be lucky to get a D.
I agree. I dropped out of school in kindergarten. Why do I have to know my multiplication tables by heart? Why do I even have to know how to add two numbers together? I don't even know the alphabet and how to read and write - I'm just talking into a microphone and this is being written. I don't even know what country I live in or who the polical leader is, why should I? I can look it up online, I just don't have any desire to.
As you can see, your argument is silly.
It's important to know things, in your head.
Of course I realize that you don't mean people shouldn't learn the alphabet or learn math, but where does one draw the line, where they don't have to learn stuff in their head?
But there are always holes in what one is even able to look up. Maybe not for everyone, but for most.
I just and a conversation the other day about this with someone and I was flabbergasted. The topic we discussed is that anyone can create a SQL database - anyone can. But do they normalize the database to 3rd or 4th normal form? He worked with a lot of corporations, and he said most companies don't have normalized databases. In university, that was a basic thing that they taught a semester course on that. An entire semester. Not to memorize anything, but just the general concepts.
Someone can learn how to create an incorrectly structured database easily, but they don't even know about normalization, so how can you even look it up in the first place? And even if you do, somehow, learn what it is and look it up, I took the class for an entire semester. How is someone going to learn it in 15 minutes by looking it up? I looked up stuff in the book, too, and got info from the professor, and it took me an entire semester, and everyone else in the class, too. Unless you're calling us all dummies.
There's all kinds of stuff like that. You don't know how to look up something if you don't even know it exists to look up, and it is not obvious.
I had to hastily evaluate a bunch of online proctoring systems during lockdown. They went from dismal(1) to borderline scammy(2). I advised against using any, and got a lot of pushback from the department heads. Someone in the regional government must have seen the same as me, and after a couple weeks the regional education authorities had banned the damn things.
(1) one of them systematically failed to do face recognition on females with particular haircut styles
(2) Another claimed to be "AI powered" when it was pretty obvious after a couple meetings that they were just handing out the proctoring to real people quickly poring after the fact over a slideshow of the whole thing.
There is often a lot of criticism of "learning by rote" and often the blanket rebuttal is "we all have access to Google" etc. however there is a very big difference between someone who knows something and someone who has to look it up, which is why I will never accept a job candidate who will "learn what they need to learn".
How long would an experienced dev honestly take to learn a mid-level amount about e.g. Kubernetes or Go or Rust? at least months and months of common usage, not something that is comparable with someone who already knows stuff and can get cracking on day one and already avoid lots of the footguns that exist in our industry that you normally only learn the hard way.
Let's stop pretending that the internet is a replacement for education, in most cases it is far too unstructured, which is why people still take courses and why education and school still exists.
> an experienced dev honestly take to learn a mid-level amount about e.g. Kubernetes or Go or Rust? at least months and months of common usage, not something that is comparable with someone who already knows stuff and can get cracking on day one
I'm not sure how this is relevant? You seem to be trying to make a point about hiring people with specific skills vs candidates with proven ability in the field, broadly speaking, before committing an apparent non-sequitur in which someone (it's not clear who) is "pretending that the internet is a replacement for education". Do you often interview candidates who learned K8s (or Rust!) as part of their education? I assume (given your apparent strong feelings on the issue) that you have some familiarity with CS curricula, but perhaps not.
As a related issue, you're certainly sending a strong signal to the candidates you're hoping to attract, judging by your profile. Whether it's a positive one is left as an exercise for the reader.
> I will never accept a job candidate who will "learn what they need to learn".
That's funny, because I would say the opposite, if they do NOT, "learn what they need to learn", I would find it problematic.
I'm not a Java programmer, but it happens regularly that when java programmer truly get stuck, I go through their code and find the bugs, learning what I need to learn on the fly. Let's say they're having problem with network connectivity, I can learn how that should be done, reason about fault tolerance and teach myself how to look at the traffic in wireshark.
I'm not a infrastructure operations person, but if there are strange behavior in the trace, I can sit down with those who are, understand the network topology, and understand where and why the tcp connection may have issues, teaching myself what I need as I go.
Then I bring the network person and app developer together, go through the findings, explain to the developer what this looks at from the network side and explain to the network infra person what that means for the programmer. If there are some kinds of network instability that is irreducible, I can show the developer design patters that ensure that the code he's writing is resilient to that kind of problem.
I'm also not a Data Scientist, but when the Data Scientists are having problems, I can sit down and look at their problem, learn about the methods they're using, find patterns they did not see, prerequisites that were not met, or find errors in their math.
The key is to have both deep and broad knowledge of the basics of the technical
and mathematical fields. THOSE I do mostly know pretty well, from half a lifetime of being curious and taking the time to really understand things, instead of just memorize them. If you know the foundations well enough, learning a given application can be learned very quickly.
If a student has easy to hand lookups and cheat sheets, then unsupervised they can appear more fluent. But someone sitting watching would obviously noticed
and judge them more harshly than someone reading a completed paper.
The obverse is something often complained about on HN - being forced to perform
at interviews that do not measure fluency but are tests or other shortcuts to actually sitting with and judging someone
Tape your notes to the lid of the laptop and it's out of sight from even the most complete "scan" of a room. If people can cheat during an in person exam it's not possible to enforce cheating restrictions off-site, it's a completely useless loss of privacy to enforce an unenforceable idea.
I'm not pro-cheating, but this is just another example of tech trying to show that it can solve any social problem through even more technology and we already know this doesn't work. You must face the social issues head on, you can't just throw money at them in the form of more technology.
> two hours before a General Chemistry II test [...] Mr. Ogletree replied that there were confidential documents, including 1099 forms, in the bedroom where he was taking his test and that he would not be able to secure them before the exam.
This policy is stupid for a variety of reasons, including that it seems completely incapable of preventing cheating, but using "confidential documents I can't move" or "medications" as an excuse just seems silly to me.
170 comments
[ 2.8 ms ] story [ 159 ms ] threadMost tests are essentially nothing more than tests for the ability of rote memorization - they don't (really) check if the students actually understood the material and are able to apply it to new situations ("transfer of learning"), especially because it is way more difficult to create such tests and to fairly grade them. However, rote memorization tests are extremely easy to cheat on, which requires intrusive countermeasures.
In the end, this is a "common thread" across most of modern academia - the focus has long since shifted from taking in the most gifted of society for actual academic research to providing a fresh herd of fungible graduates for large corporations to prey upon each year, and it shows everywhere from how admissions are handled over the sad and sorry state of frontal teaching to, as seen, dehumanizing exams.
ETA: I've ranted about the causes in the past here, with the reply to my post being really worth the read IMO: https://news.ycombinator.com/item?id=32383965
Who do these tests protect? Do they bolster the student? If so, why would they want to cheat? Is it because the information isn't relevant to their future and only the letter grade is? If a student cheats and then never encounters the natural consequences (like getting the same subject matter wrong on the job) - what was the point of the test?
The pessimist in me thinks that we're protecting industries. I would guess that an industry wants a plethora of candidates that meet a specific bar to choose from, and the information the students must memorize is inconsequential for the most part.
But that's the entirety of the higher education experience. It's all about presenting opportunities to learn and grow. People who treat it as mere credentialism are missing about 99% of the point. And yes, that includes a lot of faculty, students, and people pressuring students (to go either direction, really.)
I don't see a problem with tests. I see a problem with people treating degrees as some sort of magic totem.
This a hundred times. If I had to hire someone for a job, I would rather hire someone who knows how to search online the answer to a problem than the one who repeats the book word for word without knowing what he/she is talking about.
Exams are crap like that because better exams would require the use of one professor asking non trivial questions for each student, which costs money.
Do you have hiring authority or have you ever had it?
How do you get creativity, how are candidates better able to combine idea together? By forming each idea into a memory 'chunk' that can be handled as a single concept, and then working to combine these chunks.
If you hire people who can search google, then you get people who can search what is easily searchable, and who can create what has been previously posted online. If they somehow have 'internal' knowledge and capability, that's memory.
frontal teaching to large audiences isn't dehumanizing, it's simply effective reflected empirically by the large amounts of excellent graduates coming out of countries that focus heavily on that style of teaching.
I've never met a world class mathematician coming from a hippie waldorf school, I've met plenty coming out of the Soviet education system.
They practice applying the knowledge.
> frontal teaching to large audiences isn't dehumanizing, it's simply effective reflected empirically by the large amounts of excellent graduates coming out of countries that focus heavily on that style of teaching.
LOL have you ever had the fortune of dealing with these "excellent graduates" in offshoring projects? As soon as you go one tiny step beyond what they were taught in "university" shit gets nasty. Applying abstract knowledge to a specific use case is what is important in IT, and those who never had to actually understand the topic fail miserably.
The offshoring thing is literally just about that. The nastiness and cultural mismatch of that industry. The huge Chinese and Indian domestic tech industries show that they can build successful things, just not necessarily for foreign markets.
Without memorisation ("Just google", as many comments seem to suggest), you are severely limited in what you can come up with, and how you can contribute.
If you don't have foundational knowledge (memory): 1) you don't know what to google; 2) within a discussion concepts will go over your head so you can't contribute; 3) you don't have the components to play around with and mix and match in your head to generate new ideas.
If your work is dreadfully formulaic, sure, you can probably get by with limited foundational knowledge and just searching. If your work in any way is knowledge work, then you need _knowledge_, not just access to a vast reference.
You got the premise wrong. The main way that students cheat in online test is by communicating with other students/tutors and by having other people take the tests for them.
Doesn't matter how perfect your exam is if the student has a friend coaching them during the exam and telling them all the answers.
And it is far more common than you think. I have been tutoring for a for 5 years and there was a massive amount of students who tried to hire me to do online exams for them during the 2 years of covid. I probably would have done it if I was still a student full of debt at that time.
I've never had one, but from what I've heard they're the kind of thing that if done properly ends up being a better way to test people, but they're harder for instructors to implement.
Typically you just use the questions from ~5 years ago and rotate them through.
Moreover, if people want to cheat they can find some way to cheat regardless of any countermeasures you deploy.
Because most of teachers didn't teach online class before. They are not used to design exams that would not be way too easy it suddenly becomes open book.
I won't even say they're "lazy", coming up good take-home exams is super hard, especially when you spend your entire career not need to do that.
> if people want to cheat they can find some way to cheat regardless of any countermeasures you deploy.
I agree if it's online exam, it's kinda doomed either way. But it's more to act as a deterrent than total solution. And it does work to some "less-adventurous" students.
Note: above is just directly replying to your questions. I don't support scanning student's room.
Though maybe an extra step of storing a copy of the ID with the exam entry and having a professor who knows them verify that it matches the real student would catch most of that...
I was a Academic highschool teacher for 8 years and I agree with your statement but also don't think that we should make it too easy for them. At minimum a screen recorder so the students know even if they were not caught during the exam, they could later be caught
Bonus anecdote When the first bluetooth phones hit the market we had students use their (pseudo) smart phones (that looked like expensive calculators with greyscale displays back then) to send each other the solutions during the physics exam. No teacher at the time had this as a threat model
However, if lots of students cheat in the same way semester after semester, the school has a credibility problem.
Solve the fucking problem without violating privacy. If it doesn't get solved, fine, let it be. Accept those consequences and live with it.
I also think that the "extra time for technical difficulties" is going to be very hard to prove or disprove and would lead to very arbitrary decisions.
I see the same when I interview people online and I hear them frantically click their keyboard - but no help, I know before what Google returns.
[1] I don't ask API knowledge, if someone doesn't know some API I tell them just to invent the call they need.
(Then I spent an hour-ish talking to swardley about 3D printing and its future impact on IP law, etc. -- this was back in early 2007, before it was generally available. That was the weirdest non-technical interview I've had.)
It's also why I believe that in most cases multiple choice questions have no place in any well designed tests because it's extremely hard to create multiple choice questions that actually test the application of knowledge versus the regurgitation of rote learning.
When I was in university (grande école) in France, luckily all tests followed that format, they were open book, very few questions that needed deep detailed answers and were about interesting problems. Even finding tests from previous years and doing them was actually fun (tests were never reused).
I did an exchange in the US at RIT and most tests had very boring rote learning questions. For example in a computer graphics class: Which company were voting members of the OpenGL Architecture Review Board? Which year was the first OpenGl specification released?. I still remember those questions because I was shocked by how utterly stupid it was to have those on a test but, in general, 25 to 50% of the points of any tests was on those completely irrelevant, useless questions that anyone can google for.
Cheating is a problem for universities that never had proper testing standards. But that's a symptom of incompetence on the part of professors and of universities not making the choice to ensure that tests have to be well designed.
Although I also encountered a few where multiple answers were technically correct. Then you have to guess at which answer the professor thinks is "more correct" which kinda sucks.
It can be done decently but it's much easier to write and much more common to see bad multiple choice questions.
The way some teachers solved this was a 1:1 oral complementary exam, where students were asked to explain a random set of their solutions. This is drastically more difficult to cheat in real time, and you will almost certainly look like an idiot if you come unprepared. It's a stupidly simple solution, disincentivizing cheating through shame. Doesn't need any proctoring rootkits and rectal cameras, a simple identity check is enough.
1. If you need to lookup stuff in your waste of material you won‘t be able to finish everything. The time constraint was very hard. 2. It was not just repeating former things, you had to have understood them. Everything was a transfer knowledge exercise.
Did you enjoy this exam? I am asking as I had a very similar experience in my undergrad:
Our university allowed some basic courses to be studied at different departments, if they were equivalent. This means that there was a CS Major Algebra 1, but also Math Major Algebra 1, and you can take either.
I took the Math Major Algebra 1, being a CS student, and I have passed everything up to the final exam, where I learned that the exam is literally what you describe. All the proofs and arguments needed follow from basic principles (as things in Algebra 1 tend to do), but there is a lot of questions and the time limit is strict.
Thus, you have no time to actually do any deliberate thinking, you have to memorize and understand the basics so well that you can develop the arguments essentially in real time.
(And it was not an open book exam, even, but as you describe, it would not have helped you too much.)
I hated that exam. I never took it out of principle, instead just quitting the course and doing it with a great grade a year later, in CS. I still dislike the idea of forcing knowledge acquisition through strict time limits. What if somebody is smart but has a slow start due to stress? Ugh.
Reminds me of my school days. Tests with hundreds of questions and less than a minute to solve each question. They were essentially testing the student's time management skills as well as their physical and mental fortitude in addition to their knowledge. Extremely exhausting and anxiety inducing.
Many students and professionals taking tests cheat casually and without embarrassment: It is seen as simply an efficient way to maximize grades.
I was so happy when the pandemic hit and the second part of a qualification I was doing could be remotely proctered - everyone from professionals to crack addicts were taking tests of some sort at the same time at the test centre.
My concentration broke when the crack addict picked up the monitor and smashed it on the ground and stormed out, im not sure it they were doing a Cisco qualification or a driving test, but I dont think it was going well.
Of course there are other reasons they might act that way, but "crack addict" is a sometimes just shorthand for a person who's acting and looking a certain way.
1. You can't look for information if you don't know it exists.
2. Most standardized tests aren't used in the expectation that the testee's knowledge will matter in the future. Instead, the idea is that people who score higher will be generally more competent at whatever it is you want them to do.
I fail to see how information retention maps to skill in this case. I guess it could be a vague indicator of the amount of practice done, but some folks memorize so easily, it makes this system incredibly easy to game.
It doesn't really matter if you understand why it works; the facts are what they are.
You absolutely can. I don't know if there are stats on the number of people who have circumnavigated the globe in an amphibious vehicle, but I can Google it.
I don't know whether the thickness of an electrical cable influences the electromagnetic field it creates, but I can Google it.
You can even Google generic stuff like "do charged particles in the atmosphere exert any force on objects at ground level" and get a survey of the forces.
That only holds true if you assume people are making very specific queries in an unfamiliar domain. It's entirely possible to start with very general queries in an unfamiliar domain and hone in on a specific piece.
It's common in software engineering. I don't understand how databases optimize queries, but I can start with a broad query like "what factors does Postgres use to optimize queries" and slowly hone in on whatever is making my query weird/bad.
We always have the option of oral onsite exams as has been custom for many years, and accept that education does not scale excessively.
We can also keep developing surveillance tech and other AI solutions to the problem and handle the ethical issues.
In the end it will probably become a balance.
And testing in the typical style that fails when a student can use a textbook or google makes just as little sense.
When we locked down in the UK, millions of people were furloughed for 6 months+. How many of those people retrained, studied etc.? Not as many as we might hope because education is firstly about motivation and secondly about information.
But those tests are harder to design, harder to grade and you cannot reuse them - since someone will just post the solution on the internet. And you need to prevent students from just sharing the solution during the exam.
That said, for many years I shared the view. Some time ago a convincing argument for "memorization" was presented, and I am slowly changing to the view that it is a reasonable way to learn and to become more creative.
Memorization, while not sufficient, is an absolutely necessary part of learning. No amount of googling can help you become skillful at something.
Note that searching for new information or old information that you may have forgotten the specifics of is also perfectly normal, and an important skill in itself. But even there, Google will usually be more of a tool for recall (what was the order of those two parameters? What was the name of that substance that they use for liver issues?) then completely new fact finding.
For example, if I ask you what is the complexity of finding an element in a red-black tree, it would be OK to quickly Google the speicifc answer. But if you have to Google what complexity means, or what a red black tree even is, or if you're not sure whether the complexity is better or worse than searching a linked list, then it's clear you don't have working knowledge of red black trees.
So, in other words, you propose a test. Presumably it's open-book, since you are testing an ability to apply knowledge and not an ability to merely remember it.
This sounds less like a rebuttal and more like a validation.
While testing for "reciting" them is certainly suboptimal, it's cheaper to prepare a test of the sort than a test where you are effectively putting that knowledge to use.
A balanced test would do a bit of both (so it does not take ages to prepare and grade, yet ensures you know the basics and can apply them in some problems).
We have access to a lot of knowledge at our fingers, it's true. But if the data is all stored there instead of our brains, then we won't be able to recognise that fact A (which we just read) is similar to fact B (which we read a long time but didn't bother to remember because we could just look it up anyway). The implications of that depend on the context, of course, but in a general sense, I think it'd slow our development as a species down.
Exams are generally supposed to test your understanding, not your recollection abilities - with some exceptions, of course, but in general that's supposed to be what they're for.
Drills and memorizaton aren't the only thing in education, but they are key.
But I was never under the illusion that this constitutes working knowledge. I was able to come up with solutions for some problems only because I had some basic knowledge memorized, which allowed me to effectively manipulate ideas and concepts which I did know of. Even then, I used to semi-joke how the hardest thing in theoretical math is all the memorization required.
If you've ever explored any scientific field deep enough, you'll know that it's festered with tricks found by accident and roundabout ways to get to a solution that, if you haven't seen it before, you have no hope of applying in practice.
Or you might think that consulting a book or two (or Internet) will allow you to prove Fermat's theorem while you are being tested for 2h.
Very little of what I used on a day-to-day basis came from rote learning, a lot came from the deeper understanding that I got from actually actively engaging with the concepts.
I'm not proposing doing away with schooling completely, but the model should change (much in the way more progressive schools are doing anyway, just a shame that not many parents can afford to send their kids there).
The question is not how much non-rote learning is required to adequately perform any task, but whether rote learning is a necessary component of it.
By engaging with research papers (often requiring multiple readings), you "accidentally" also memorize a lot of the material (even if not for ever and ever), which makes it possible for you to "try to improve it" etc.
So all I am saying is that rote learning is a necessary component of applying any suitably advanced material. Whether that happens organically by engaging with the material deeply enough or on purpose by simply rote-learning is orthogonal to the need for that knowledge to be there.
At issue here in this thread is whether testing for such knowledge is signalling anything of value. I still believe it is because you'll have that knowledge with either approach, even if only one of those approaches will gain you more functional knowledge: basically, such a test is a proxy for you having engaged in the material.
Which is why I suggest a mixed approach in a sibling comment: test for knowledge of "factoids" to simplify test development and application of such knowledge where you are required to manipulate "learned" concepts (so only part of the test is hard to create and keep consistently hard/easy).
Scaling past a certain size puts you kind of further and further outside of what your environment can optimally sustain. Think if a fish in a bowl never stopped growing.
Add to this the particular challenges that orgs with centralized decision making face when scaling (inertia, homogeneity, single-point-of-failure) and it becomes really hard to see the current big-fish trend as even remotely sustainable.
Perhaps only semi-related, but I like the gore-tex model, myself. Scale out and shard.
https://www.researchgate.net/publication/303803687_Working_W...
Google not choosing to provide customer service is not something either businesses or consumers should look forward to.
After 15 minutes of removing random stuff from my desk, under my desk, on my walls, including but not limited to disassembling a wall-mounted TV set, as instructed by the proctor, I gave up. I rented an office at a co-working place that was sterile white. Then they tried to accuse me of using camera-modifying software to add a background, because "no room is that clean".
If we do keep up this proctored test-at-home system, we need to have a workable way to make sure the testing room still works out as a living space after the test.
What they could do is letting people take written test in a lightly controlled environment (just a webcam with audio, no 360/scan before tests), and then complement it with a Q&A part. Yes, that's probably more time consuming and more expensive.
That was a public school, so the impact is probably scoped.
I'm picturing a swarm of small hovering drones sweeping their blue laser beams over your bedroom office, before one of them finishes with an accusative scan of your retina.
'I spy with my little eye and it is 'white'' -> the white of your eyeballs.
That procedure is full of holes, though. There are so many ways to cheat when you don't take a test on-site, that it's not even funny. Those tests have to be performed with a computer on to begin with. I've heard that in some cases, there was testing software that would scan the computer too, similar to games' anti-cheat devices.
Schools first implemented that procedure in a hurry then stuck to it, and they thought they could violate privacy "a little" because states and countries themselves adopted exceptional laws that limited human rights. That felt wrong to me when it happened, and I am glad that at least a judge in the US confirmed it was wrong.
So what ? The only punishment is a slap on the wrist. "If we have to, we will try to comply"
Our methods of teaching come from a worldview in which the information stored in your head is your only resource. That wasn't true in the past where you could consult colleagues, look things up in books or call experts - and it's certainly not true now that everyone carries the entire knowledge of humankind around in their pocket.
So what is the point in artificially creating a test situation where the student is isolated from resources she'll have at her disposal in real life? How about instead, we create (whatever the plural of curriculum is) that teach you how to use these resources efficiently and test you on how well you perform using all the tools that will be at your disposal in your actual job as well?
I can envision many scenarios where a person needs to be quick on their feet and does not have time to look things up. Drivers, pilots, doctors, etc. Certification provides a lot of value to society.
The other point of testing is to separate wheat from chaff. To see if someone is disciplined enough to learn the material or see a task through. There are issues of going beyond the ideal amount of testing here, but not totally useless in my opinion.
But I'd argue that realistic test conditions correct for this automatically.
If your test is "write a program that does X within two hours" the programmer lacking core knowledge will fail.
If your test is "recite 20 programming paradigms from memory" you will have to check for "cheating"
> separate wheat from chaff
Wheat can demonstrate/use the knowledge, chaff can't. Why does it matter whether the knowledge is internal or external, assuming results are the same?
Perhaps tests are to make sure that you are a going to be colleague worth consulting then? Yes, pure memorization and recall of facts is less valuable now than before, but knowing how to apply the facts available to you is still a skill worth having, and similarly worth testing. Those skills can also be assessed using written tests, and the potential for cheating is just as on memory tests.
what is the point in artificially creating a test situation where the student is isolated from resources she'll have at her disposal in real life
Let's take language learning as an example: do you consider a foreigner with a dictionary to be just as expressive as a native speaker? Perhaps there is value in testing the immediate knowledge of the student rather than their lookup skills?
Yet I don't know a lick of Go. If I were to open up a Hello World tutorial for Go during the interview, would I honestly still have a shot?
When I tell people a story about the history of the french revolution or something, I don't have the opportunity to go research it beforehand. I've already done that research, and I've already thought about those events for months or years, which gave me the opportunity to synthesize an insight or opinion I wish to share.
You don't get that if you don't know things in your head.
You may be well aware of this, but you've essentially converged on the classical Chinese Room thought experiment [0]. It's usually applied to the philosophical question as to what does and does not constitute intelligence in the context of AI, but it's interesting how it is also applicable to what does and does not constitute knowledge in the context of learning.
[0] https://en.wikipedia.org/wiki/Chinese_room
Words are rarely restricted to their denotative (dictionary) meanings. There are often abstract, connotative meanings attached to them by speakers. For example, consider the following list of words.
Hello. Hi. Hey.
These words all have the same basic denotative meaning. They're all a greeting. Yet, there is a subtle connotation of formality which is rarely described in a dictionary.
These types of connotation are not often understood by beginning language learners. It's something they acquire with time and experience in using the language. While a teacher or search could share about the formalities, the language learner is still going to require time in applying those words in different contexts to fully understand these subtleties.
The point of a test is to check if you are capable on your own, and not piggybacking on someone elses intellect or hard work.
It is true, that in the real world, the resources available to use are infinitely larger than a classroom or a teaching setting. But your idea springs from lack of understanding about the true purpose of education, i.e, to groom children to think.
Alas, modern education has become a cesspool of rote learning, atleast in my country, India.
A big part of the valuable experience comes from having memories of similar situations, and thus a shortcut to the solution. In IT it can be similar to “indexing”, like I may remember reading a relevant stackoverflow question and I know how to find it again, but I don’t remember the exact solution.
But the catch is, to index something you first have to crawl through it. In medicine for example, there may not be much point in holding every single extreme-rare disease in memory, but having memorized them once and mostly forgotten them will lead a good doctor to know about it enough so that he/she can look it up on the odd chance it gets relevant. It will be only possible when he/she memorized it once back then.
My point is, there is a point in memorization (especially that it improves memory all-around - one will remember facts much easily after extensive learning), what should be better distinguished is facts that are to be learnt for life vs those that are only needed for a given test. It’s like unknowns vs unknown unknowns - the latter can be much worse and you don’t want someone like that as a colleague.
I'm going to claim that memory is only one part, maybe 1/3, of what I would like to examine for. Certainly in a job interview, but I also like university exams that include the parts below.
There's another huge part, *problem solving skills*. Working long enough to figure out which pieces of knowledge that you have apply to the situation.
There's another part, *making it work*. Software engineers deal with failures in practice (debugging) as well as theory (don't know how to do it). Sadly but interestingly, sometimes this is the point of failure for a student/examinee when they possess plenty of knowledge but don't know how to use it.
Most of my upper classes were heavily weighted towards group projects anyway.
Long term subconscious memory is where a lot of the connections happen. If your long term memory only knows where to find things without knowing them then that ability is stunted.
It's a statement that I agree with. There is just too much knowledge to know it all. These days there is a very real skill in knowing how to search i.e. a second order version of the above statement "you need to know where to find out how to look this up".
There are some really technical things ive done in the pasy that I couldn't do without a reference right now, but once I've got the reference I don't need to be reminded about low level details or edge cases or gotchas - it's all still there in my head I just need to see the road map and it will all come back to me.
I wouldn't want to go to doc that has to Google basic stuff.
Knowledge is so easily acquired these days that, while useful, it is not essential. I hire people with zero domain knowledge for that reason.
But more problematic is when people don’t know there’s a problem to google in the first place — it seems to do the job — which is usually resolved by both education and experience, but is ultimately a result if they don’t know what they don’t know.
The knowledge is generally out there, but you need to know you’re looking for it, and it’s only in the common/trivial problems that it’s so obvious that the compiler/tooling/customer just flat-out tells you you’ve missed something
Explain this to me by looking it up. https://imgur.com/Bj24AAR
This is how people become software engineers these days. They _constantly_ look stuff up, watch videos, and learn by doing that. Yes, you can still go to college and take tests that require you to "know" things, but honestly the vast majority of what I "knew" in college turned out to be short-term knowledge stored for tests.
By searching and learning, you
I learned by doing, initially, because I started learning programming with minimal resources before going to college and had to unlearn or restructure a lot of the knowledge I had gained up to that point once I did. I would probably have gotten there in the end, but it would have taken me a lot longer. The world was different then, though. Instead of easy access to info before going to college I was lucky to find a book somewhere that someone threw out that talked about things in a peripheral way that I needed to know, or ran into someone I could discuss things with that were as info-deprived as I was.
That's where my skepticism of this approach comes from, I guess. But I didn't live in a world where knowledge was at my fingertips from a very early age, so who knows?
This is reasonable. E.g. sure a person can google how to build a house, but also sure that at planning phase they’ll mess and conflict with a half of good rules of thumb. Or forget them right after reading, because there is no habit nor knowledge in their head, and all that info is pretty transient. Pro is not the one who has read an article, but the one who can tell a couple of their own notes about it.
Otoh, gp is also correct that tests should check how you are doing in a real life situation. So maybe use both ways of testing?
You can easily find an answer to "how do I enable file sharing on Windows." You cannot easily find an answer to "how do I design a website for an engineering firm which wants to increase it's inbound leads by 25%."
The former is a simple search away. The latter requires the integration of numerous abstract concepts which the shallows of a search engine cannot return.
Further, such an approach also limits our ability to create new knowledge.
Teachers don't care about students looking into their books/pdf, what they want to prevent is them getting the answers from a snapchat group.
I definitely think that is a better way to test.
Unfortunately, in order to progress to more advanced topics, you need to understand the basics yourself. It's not sufficient to just be able to research it online. This is true in every field I can think of. That's why you can't just give people online access for every kind of test.
In my opinion, cheating from internet access or outside resources is not the problem, cheating by gogling exact problems is.
As you can see, your argument is silly.
It's important to know things, in your head.
Of course I realize that you don't mean people shouldn't learn the alphabet or learn math, but where does one draw the line, where they don't have to learn stuff in their head?
But there are always holes in what one is even able to look up. Maybe not for everyone, but for most.
I just and a conversation the other day about this with someone and I was flabbergasted. The topic we discussed is that anyone can create a SQL database - anyone can. But do they normalize the database to 3rd or 4th normal form? He worked with a lot of corporations, and he said most companies don't have normalized databases. In university, that was a basic thing that they taught a semester course on that. An entire semester. Not to memorize anything, but just the general concepts.
Someone can learn how to create an incorrectly structured database easily, but they don't even know about normalization, so how can you even look it up in the first place? And even if you do, somehow, learn what it is and look it up, I took the class for an entire semester. How is someone going to learn it in 15 minutes by looking it up? I looked up stuff in the book, too, and got info from the professor, and it took me an entire semester, and everyone else in the class, too. Unless you're calling us all dummies.
There's all kinds of stuff like that. You don't know how to look up something if you don't even know it exists to look up, and it is not obvious.
(1) one of them systematically failed to do face recognition on females with particular haircut styles
(2) Another claimed to be "AI powered" when it was pretty obvious after a couple meetings that they were just handing out the proctoring to real people quickly poring after the fact over a slideshow of the whole thing.
How long would an experienced dev honestly take to learn a mid-level amount about e.g. Kubernetes or Go or Rust? at least months and months of common usage, not something that is comparable with someone who already knows stuff and can get cracking on day one and already avoid lots of the footguns that exist in our industry that you normally only learn the hard way.
Let's stop pretending that the internet is a replacement for education, in most cases it is far too unstructured, which is why people still take courses and why education and school still exists.
I'm not sure how this is relevant? You seem to be trying to make a point about hiring people with specific skills vs candidates with proven ability in the field, broadly speaking, before committing an apparent non-sequitur in which someone (it's not clear who) is "pretending that the internet is a replacement for education". Do you often interview candidates who learned K8s (or Rust!) as part of their education? I assume (given your apparent strong feelings on the issue) that you have some familiarity with CS curricula, but perhaps not.
As a related issue, you're certainly sending a strong signal to the candidates you're hoping to attract, judging by your profile. Whether it's a positive one is left as an exercise for the reader.
That's funny, because I would say the opposite, if they do NOT, "learn what they need to learn", I would find it problematic.
I'm not a Java programmer, but it happens regularly that when java programmer truly get stuck, I go through their code and find the bugs, learning what I need to learn on the fly. Let's say they're having problem with network connectivity, I can learn how that should be done, reason about fault tolerance and teach myself how to look at the traffic in wireshark.
I'm not a infrastructure operations person, but if there are strange behavior in the trace, I can sit down with those who are, understand the network topology, and understand where and why the tcp connection may have issues, teaching myself what I need as I go.
Then I bring the network person and app developer together, go through the findings, explain to the developer what this looks at from the network side and explain to the network infra person what that means for the programmer. If there are some kinds of network instability that is irreducible, I can show the developer design patters that ensure that the code he's writing is resilient to that kind of problem.
I'm also not a Data Scientist, but when the Data Scientists are having problems, I can sit down and look at their problem, learn about the methods they're using, find patterns they did not see, prerequisites that were not met, or find errors in their math.
The key is to have both deep and broad knowledge of the basics of the technical and mathematical fields. THOSE I do mostly know pretty well, from half a lifetime of being curious and taking the time to really understand things, instead of just memorize them. If you know the foundations well enough, learning a given application can be learned very quickly.
If a student has easy to hand lookups and cheat sheets, then unsupervised they can appear more fluent. But someone sitting watching would obviously noticed and judge them more harshly than someone reading a completed paper.
The obverse is something often complained about on HN - being forced to perform at interviews that do not measure fluency but are tests or other shortcuts to actually sitting with and judging someone
I'm not pro-cheating, but this is just another example of tech trying to show that it can solve any social problem through even more technology and we already know this doesn't work. You must face the social issues head on, you can't just throw money at them in the form of more technology.
Remote learning is bollocks Matrix pod shit. Just don't.
This policy is stupid for a variety of reasons, including that it seems completely incapable of preventing cheating, but using "confidential documents I can't move" or "medications" as an excuse just seems silly to me.