The Respondus Lockdown browser, which I had to use last year, does something similar. I'm very pale and have white walls and literally had to hang a dark background because it couldn't see me, along with shining a bright light on my face.
>The target audience for such things probably only has one computer
...or they could borrow their friend's computer? Also, it doesn't have to be a computer. It could be a tablet/phone with usb-otg/bluetooth keyboard/mouse.
Bar exams involve a significant element of speed and time management; being able to consult reference material on a secondary device doesn’t give you a huge advantage and may not be worth the risk of getting caught. Some bar exams are open book anyway. But being able to send a copy of the whole exam to someone else who can work on it in parallel, and send you completed answers to paste in, would give you a huge advantage. Locking down the only computer that can read the exam or submit answers creates a significant barrier to this.
Ok, that can also be solved by readily available technology. All that's necessary is a HDMI capture device + streaming to however many people one wants. Gamers do that all the time.
I've been closely watching this situation. It's a disaster. Basically, a bunch of places have been trying to do a remote bar exam since July, with various delays (Illinois, for example, is finally about to happen next week... theoretically). It's... just not working. Applicants are turning up security problem after security problem after security problem.
Which I think is predictable. The bar examiners (and many applicants) are incredibly paranoid about cheating, far more paranoid than normal university exams. So they're asking for, essentially, very strong managed device levels of security to prevent cheating, with stuff like facial recognition layered on top... and on an incredibly variety of applicants' own devices. It's like if the CIA had a BYOD policy.
I find the requirement of using a laptop without being allowed to use an external monitor to view potentially very long documents (half screen!) for hours on a tiny screen to be ridiculous.
The moral fitness test isn't designed to ensure that all lawyers are ethical. It only eliminates from consideration people whose past actions demonstrate that they have questionable ethics.
But many law students are young and have no paper trail/track record, so this isn't a very strong filter.
As an attorney who handles[] confidential client data running malware like this proctoring software is a breach of professional ethics, though not one yet acknowledged.
[] And not just will handle, because of limited cross state privileged many people who take the bar are already licensed and practising under another state.
It's not just a character and fitness check for past conduct. Almost all states require that students pass the Multistate Professional Responsibility Exam (MPRE) for admission to the bar. In some ways it's like the bar exam, but it tests knowledge of the rules of professional conduct (~lawyer ethics).
It’s also very, very easy (CA has the highest pass threshold, which is around 60%). Considering this is a multiple-choice test, that is a very low threshold.
More importantly, the test does not in any way measure whether an applicant will act ethically. It only tests whether you know the rules.
Not just IL, also NY, CA, MA, FL, TX, PA, OH and a dozen more. ~35,000 nationwide, simultaneously!, next Monday and Tuesday, two sessions per day for two days. It all has to work perfectly 150,000 times!
How much do people want to bet that this is using the haarcascade_frontface_default.xml classifier that's used in opencv face detection tutorials? I've seen similarly bad results on dark skin color.
My god, facial recognition that doesn’t accurately detect PoC? Who would have thought?
More seriously at this point why does any company shipped any kind of facial recognition software without verifying such catastrophic accuracy problems don’t occur?
In the past you could maybe argue that it was new enough to be a poorly understood problem, but for years now the most basic validation checks should include racial bias testing.
The state of remote proctoring software is really, really bad.
Students are forced to install software that records video+audio of them, performs facial recognition, records their every keystroke for analysis, records their browsing history and open applications, aks them to show their entire room to the camera, keeps this data indefinitely with no privacy policy, and sometimes even gets compromised. Of course, sometimes it also doesn't even work, which makes it even worse!
There is generally no recourse for students who do not want to do this. Linux students are told to use Windows or fail the class. VM users are told they may be failed if they try to run the software in a VM (which is not allowed by the software and must be REed to attempt to begin with), and students that value their privacy or security (as this software is malware) have no recourse whatsoever often.
Regardless, droves of schools continue to spend absurd amounts of their endowment on expensive contracts with these remote proctoring malware companies, desperately attempting to hold up the facade that their degrees have a value that is somewhere near how much they charge in tuition.
I feel like I only escaped school recently myself, but it looks like it would be an even more miserable experience if I had to start the process again in 2020.
The solution is to not rely on remote proctoring at all. You ultimately cannot fully monitor an environment you don’t physically control, and in trying to do so, you’ll only end up pissing a lot of people off without doing anything to solve the problem. If potential cheating is such a critically important issue that proctors are ABSOLUTELY necessary, then don’t administer the tests online in the first place. Just augment the testing facility with COVID-19 related precautions, like required masks, hand sanitizer and spaced out desks.
Remote proctoring does not have to be automated. Plenty of tools do it with proctors monitoring the candidate.
You don't need it to be at location , you need human proctors .
Human proctors are expensive the bar exams are to blame here, if their question papers banks also too narrow so they have to do it the same time. Good exams will give you slots you can choose from , spreads the load.
No provider can come up with 30,000 proctors for just a day , even if bar exam boards were ready to pay, which they don't seem to be
I'm not talking about automated proctoring, I'm talking about remote proctoring. Although automated proctoring is its own can of worms, especially when it runs client-side and essentially demands that you cede control of your computer setup to it.
The sole purpose of having proctors is to alleviate cheating. Virtual proctoring is completely ineffective at doing so, and on top of that, effectively invites strangers to watch you in your own house and dictate what you can or cannot run on your own computer.
Given the current global pandemic, the risk of not doing these exams for folks who need them for their jobs of risking covid-19 is a far bigger problem than privacy.
An exam body is certifying you are qualified in a skill or a subject.
Yes these systems are intrusive, they are trying hard to make sure integrity of their assessment is there.
I don't see how else in today's world both objectives can be met. Many industries regulate your professional interactions. Every call can be recorded in customer support or fin tech many of them are doing it from their homes today.
Make questions that you cannot easily cheat on. If I can simply google the answer and plug it in without changing anything, then it's not a good question. I think open ended problems are the way to go where only knowing can't cut it.
In the case of the bar exam, they just need to get over it. Bar applicants have to pass an incredibly intrusive background check, then we trust them with the lives and livelihoods of their clients, to hold thousands and thousands of other people's funds, etc. So let's trust them not to cheat.
Disclaimer : one of my products is used for remote proctoring.
It depends on the vendor of the exam too, our proctoring sdk works on chrome and Firefox which means all major OSes too, we try hard not go outside browser context, we support android and iOS natively.
However many exam apps are very poorly written and have limited support for different OS, and usually never for mobile. I have seen full flash apps used today with 3months to deprecation.
we do face detection but not recognition and recommend using human live proctors for high stakes requirements.
The OS vendors are doing their part as well, we partner with both Apple and MS and give constant feedback, Apple announced AAC mode this WWDC and MS had number of APIs
Linux support is a factor in the education assements space due to chromebooks, most of my devs and half of my ops use linux , so I can say atleast one vendor cares.
Today there is profussion of vendors and overnight shops attempting to build in this space post covid-19, sales here is like any other enterprise product , tech merit is only one component, and nuances are lost on the customer in the hurry post covid 19, exam/certificate bodies have little scrutiny and rarely choose the better product for the user.
However in few years things will improve, buyers will understand the tech better, vendors will consolidate.
Hearing that one of the "vendors that cares" still does facial analysis gives me bad vibes.
Can you explain a bit about how the face detection works and what sort of detection data is sent outside the local client? If so, do you store or send it anywhere?
Your feed usually is captured , it is no different than a hangout or zoom recording. Many providers actually use zoom .
Face detection is detection of a human face in a feed, not a specific face recognition. Usually such detection is used in negative sense, i.e. lack of face in the feed would prompt for human review.
Local context is not feasible, you need to store the feed remotely for compliance in case of dispute etc, it same as video call or a support call that is recorded.
I'm actually quite curious as to where you guys end up taking the product, because at least pertaining to the legal profession, as of a few years ago, the entire system from your first proctored exam as a 1L to the bar exam didn't actually require the browser context at all. The UI we used had a heavy Windows 98 aesthetic going on, and mostly just consisted of native text-boxes with pretty minimal formatting tools, timer, and not much of anything else. But at the same time it's also understood on some level everybody knows that the rigamarole is done for its own sake entirely. It's always understood on some level that it's not the substantive law that these exams test for but instead how ably is one able to jump through arbitrary hoops in a prescribed time and format, which is one way of looking at the legal profession as a whole. It's entirely possible that as the vendor you may end up caring more about the exam than the actual test-takers themselves.
Seems like it's pretty easy to work around proctoring software. Some ideas:
* put the webcam in a separate room, pointing towards a body double and a clean working area. meanwhile, you're in the other room completes the test with full access to cheat materials by your side. you can also hire someone to do the test for you, and all you have to do is sit in the other room.
* plug a y-splitter into the microphone jack. in one end plug in a microphone, in the other plug in a phone playing a looped background noise track. the looped background noise track prevents the proctor from knowing that the microphone was unplugged. have someone that's out of the field of view of the webcam look at the monitor (with magnifying optics if necessary) and determine what materials need to be read out loud. When ready, disconnect the microphone from the y-jack and read the answers out loud. When done plug the microphone back in.
* have cheating materials on a secondary display (eg. monitor's PiP feature, or a tablet placed next to the monitor). get your accomplice to navigate to the materials as necessary, or use your feet to operate a mouse under the desk.
I could essentially do it in my system without any modifications.
I use desktop + laptop hooked up to the same screen and sharing mouse and keyboard through a usb switch. I can toggle between systems with a button on my desk without giving any clues of anything happening.
Completely different machine, so no hopes of ever seing that.
I interviewed at an online proctor company, and I discovered that the build of chrome that they used tracked my eyes, tracked my mouse movements, and tracked my searches as I sat in their conference room and took some bullshit personality test.
I had a decent offer, but, still turned them down. Just wasn't interested in helping something like that succeed
44 comments
[ 5.4 ms ] story [ 92.6 ms ] thread...or they could borrow their friend's computer? Also, it doesn't have to be a computer. It could be a tablet/phone with usb-otg/bluetooth keyboard/mouse.
Can't be blocked and is not detectable.
Automated and data-scraping like this is downright unethical.
Which I think is predictable. The bar examiners (and many applicants) are incredibly paranoid about cheating, far more paranoid than normal university exams. So they're asking for, essentially, very strong managed device levels of security to prevent cheating, with stuff like facial recognition layered on top... and on an incredibly variety of applicants' own devices. It's like if the CIA had a BYOD policy.
And yet supposedly the Bar only admits members who have passed an ethics vetting.
But many law students are young and have no paper trail/track record, so this isn't a very strong filter.
[] And not just will handle, because of limited cross state privileged many people who take the bar are already licensed and practising under another state.
I'm taking it this month, presumably in person.
More importantly, the test does not in any way measure whether an applicant will act ethically. It only tests whether you know the rules.
More seriously at this point why does any company shipped any kind of facial recognition software without verifying such catastrophic accuracy problems don’t occur?
In the past you could maybe argue that it was new enough to be a poorly understood problem, but for years now the most basic validation checks should include racial bias testing.
Students are forced to install software that records video+audio of them, performs facial recognition, records their every keystroke for analysis, records their browsing history and open applications, aks them to show their entire room to the camera, keeps this data indefinitely with no privacy policy, and sometimes even gets compromised. Of course, sometimes it also doesn't even work, which makes it even worse!
There is generally no recourse for students who do not want to do this. Linux students are told to use Windows or fail the class. VM users are told they may be failed if they try to run the software in a VM (which is not allowed by the software and must be REed to attempt to begin with), and students that value their privacy or security (as this software is malware) have no recourse whatsoever often.
Regardless, droves of schools continue to spend absurd amounts of their endowment on expensive contracts with these remote proctoring malware companies, desperately attempting to hold up the facade that their degrees have a value that is somewhere near how much they charge in tuition.
I feel like I only escaped school recently myself, but it looks like it would be an even more miserable experience if I had to start the process again in 2020.
You don't need it to be at location , you need human proctors .
Human proctors are expensive the bar exams are to blame here, if their question papers banks also too narrow so they have to do it the same time. Good exams will give you slots you can choose from , spreads the load.
No provider can come up with 30,000 proctors for just a day , even if bar exam boards were ready to pay, which they don't seem to be
The sole purpose of having proctors is to alleviate cheating. Virtual proctoring is completely ineffective at doing so, and on top of that, effectively invites strangers to watch you in your own house and dictate what you can or cannot run on your own computer.
An exam body is certifying you are qualified in a skill or a subject.
Yes these systems are intrusive, they are trying hard to make sure integrity of their assessment is there.
I don't see how else in today's world both objectives can be met. Many industries regulate your professional interactions. Every call can be recorded in customer support or fin tech many of them are doing it from their homes today.
It depends on the vendor of the exam too, our proctoring sdk works on chrome and Firefox which means all major OSes too, we try hard not go outside browser context, we support android and iOS natively.
However many exam apps are very poorly written and have limited support for different OS, and usually never for mobile. I have seen full flash apps used today with 3months to deprecation.
we do face detection but not recognition and recommend using human live proctors for high stakes requirements.
The OS vendors are doing their part as well, we partner with both Apple and MS and give constant feedback, Apple announced AAC mode this WWDC and MS had number of APIs
Linux support is a factor in the education assements space due to chromebooks, most of my devs and half of my ops use linux , so I can say atleast one vendor cares.
Today there is profussion of vendors and overnight shops attempting to build in this space post covid-19, sales here is like any other enterprise product , tech merit is only one component, and nuances are lost on the customer in the hurry post covid 19, exam/certificate bodies have little scrutiny and rarely choose the better product for the user.
However in few years things will improve, buyers will understand the tech better, vendors will consolidate.
Can you explain a bit about how the face detection works and what sort of detection data is sent outside the local client? If so, do you store or send it anywhere?
Face detection is detection of a human face in a feed, not a specific face recognition. Usually such detection is used in negative sense, i.e. lack of face in the feed would prompt for human review.
Local context is not feasible, you need to store the feed remotely for compliance in case of dispute etc, it same as video call or a support call that is recorded.
* put the webcam in a separate room, pointing towards a body double and a clean working area. meanwhile, you're in the other room completes the test with full access to cheat materials by your side. you can also hire someone to do the test for you, and all you have to do is sit in the other room.
* plug a y-splitter into the microphone jack. in one end plug in a microphone, in the other plug in a phone playing a looped background noise track. the looped background noise track prevents the proctor from knowing that the microphone was unplugged. have someone that's out of the field of view of the webcam look at the monitor (with magnifying optics if necessary) and determine what materials need to be read out loud. When ready, disconnect the microphone from the y-jack and read the answers out loud. When done plug the microphone back in.
* have cheating materials on a secondary display (eg. monitor's PiP feature, or a tablet placed next to the monitor). get your accomplice to navigate to the materials as necessary, or use your feet to operate a mouse under the desk.
User impersonation is a real problem in physical exam too, atleast in remote proctoring you have a video to review.
I use desktop + laptop hooked up to the same screen and sharing mouse and keyboard through a usb switch. I can toggle between systems with a button on my desk without giving any clues of anything happening.
Completely different machine, so no hopes of ever seing that.
I had a decent offer, but, still turned them down. Just wasn't interested in helping something like that succeed