-there's already an app for getting feedback if you don't understand... it's called RAISING YOUR HAND!!!!!11!
-are you going to add a "distractometer" to log how many times people are distracted by using and checking the app?
Ok, now that that's done. I would actually lose the "o-meter" phrasing and have the app simply say "Confused" and "Understand". In fact, you could even lose the "understand" interaction. I doubt people will be hitting a button all the time every time they "understand" something during a lecture. Just have one button for confusion.
Is there a way to post feedback (anonymously) at the end of a lecture?
Radical piviot: you know what this model of instant feedback would be good for? Occupy Wall Street and general assemblies.
Thanks for the feedback! In large classes students tend to be reluctant to raise their hands (especially in classes with 200+ students). ESL students are also unlikely to raise their hand and say they don't get it. I actually tested the app in 3 first year classes and it didn't seem to be distracting.
I'm hoping to add a feature for leaving feedback at the end of a lecture, great idea!
I went to a small college too... In a class of 10-20, it was pretty easy for the professors to know when we were confused since she was looking at each of us.
Exactly right. I teach at a uni in South Korea. Students who never lived/traveled outside the country will never, ever raise their hands if they don't understand. Not only that, but if you ask them outright, they will nod and say clearly, "yes, I understand", and then turn to their best friends and say in Korean, "I actually don't understand any of it".
This app would be a tremendous help not just to my classes (I've been here a long while, and know how to read the signs of confusion & check understanding), but also to the classes of newbies in Korea.
New teachers here sometimes go for months naively asking students "do you understand this stuff" and believing it to be true.
If Koreans could vote on this anonymously, I think they would actually do that at every opportunity.
I like cutting the input down to only "confused". You can just assume the remaining people logged into that "room" are not confused - either they understand it or are comfortably making sense of it.
This can also be a problem in my opinion. Imagine you are only able to enter "confused". Then as soon you are using this app in a class room everyone who sees that you are using the app knows that you will enter "confused". This is a problem for many people I guess. You don't wanna lose your face sometimes. Hm - one solution to this could be to make the UI of the app absolutely generic or exchangeable. I would also build in a random delay between the "tap" on the button and the actual transmission of the data to the "lecturer". The delay can be minimal but this makes it hard to track it back to the source.
Raising your hand only tells the professor that you, an individual, are confused. Interruptions by a single student that doesn't understand the material, when the rest of the class does, might actually cause the professor to focus too much attention on a subject that doesn't need it.
This is far better. There will always be a couple people that are not following. If the professor is getting 90% "I understood" responses, he can simply mention that anyone with questions on that topic is free to meet them after class, while the rest of the time can be spent on topics where there is a lower rate of understanding.
Also, having the "Understand" button might have the same negative effect as the excessively intrusive UAC checks in Windows Vista: people pressing it without thinking twice about it.
I can definitely see the appeal of this - like you mentioned in your blog post, there's always worry that others will think you're stupid or something for not understanding something.
That having been said, I had a few questions/concerns:
- How would it work when there were specific gaps? It might be helpful if comments could be included to the instructor when the rating is provided. That would allow the teacher to reinforce those particular topics.
- How would you guarantee participation? If, for example, only people that don't understand the topic speak up, wouldn't that skew the data presented to the teacher?
Great points. I'm definitely interested in allowing students to provide more in-depth feedback.
Participation is a much trickier issue. In the classes that I tested in about 40 - 60% of the students actually used it. I was thinking of weighting the confusion 'votes' based on 'active' users.
I'm hoping that through feedback form professors I'll be able to add features that make it more worth while for students.
That's a lot more participation than I would have guessed, that's pretty awesome! Thanks for the info, and I look forward to seeing this getting more widespread use :)
I think it would be awesome if you can generalize the idea for online lectures also (coursera, khan academy, udacity, etc). Since videos are not real time, letting students hit "confused" for certain segment of an online video will let the developers splice in a better explanation for bad parts of a video or even let them a/b test two different versions of a segment of the same video.
And also, instead of a confused button, maybe allow them to optionally elaborate on the source of their confusion? A potential solution for online videos is via timed comments (similar to how soundcloud does it, except maybe with interval of confusions instead http://37signals.com/svn/posts/1143-soundcloud-expands-the-a...)
I like the idea of the app. Your website looks horrific on ie. I know, no one uses ie any more, but I am forced to use it at work and there are probably several million other people in the same boat. You should fix it if possible.
I like it. I think you've got one too many lines, though: you don't need both a confuse-o-meter and an understand-o-meter - they're reporting the same metric. I'd consider averaging them into one line for overall sentiment.
Great concept - quality is definitely an issue in the classroom setting that traditional methods of control do little to address - i.e.: end of term surveys etc.
Thinking freely here:
Could you produce a report for the school's administration of each professor's confused/understood ratio?
As gee-totes suggested, a messaging system would close the feedback loop nicely.
Is there a means of authenticating students? If this tool became a means of evaluation, it would be open to gaming by cagey professors.
Great suggestion, producing a report is a great idea. I could even use the confusion data to predict exam test scores.
If students are going to be evaluating their teachers (providing richer feedback than confused/understood), then you're correct there will have to be authentication for students.
Be careful with cumulative evaluations - how confused students are may not correlate with how good a teacher is. And, if professors are going to be evaluated with the results, they are less likely to use the tool well...
For authentication, I'd make it so that when the professor each session, the class is given a random 4-5 digit code. That way the students would have to be in class to get the code and participate.
You may also want to think about adding the ability for the instructor to add different buttons, not just "understood / confused". You could then have class polls, or group questions to see how many people are paying attention.
This should all be able to be controlled from the instructor dashboard.
Good thoughts! The 4-5 digit code would be a good way of differentiating between sessions/lecture. When adding class polls etc, it my be advantageous to give each student their own account. That could conflict with the current anonymous aspect of understoodit.com though.
I agree about anonymity. I'm a high school teacher and there are a bunch of smartphone apps that let students send in answers to multiple choice questions that the teacher puts on the board (I think it's called SmartResponse). The problem is, all responses have the students' phone numbers attached. So privacy concerns make it so that an otherwise useful product is basically unusable. Obviously the level of concern would be different in College/University, but I think students might be less inclined to use the app if it is not anonymous.
That said, this is a great idea. This is the kind of real-time feedback that I would love to have access to. Many of my students would rather fail than put their hand up and ask for help. Something like this would really help them out.
I like the idea of the code to authenticate students.
I guess you'd have to watch out for selection effects in all of this too... an interesting challenge!
> Could you produce a report for the school's administration of each professor's confused/understood ratio?
That would probably make professors (especially non-tenured professors) either terrified of making use of the app, or terrified of tackling complicated subject matter. Even if administrator's announce they aren't going to use the data to decide salary or tenure issues, good luck getting the professors to believe it.
Feedback that helps professors improve is great. Yet another administrative metric that warps people's behaviors, not so much.
Having a group scared of your app is the mark of disruption!
I hear what you're saying about avoiding complicated subject matter - that's an interesting effect I hadn't considered.
That being said: I was the victim of several "bad" profs in university. Thing is I don't think they were "bad" - I just don't think they knew that they weren't "good" and even if they had, wouldn't have had any idea how move from bad to good.
I see this app as one part of a larger equation around designing systems that produce quality education. A complete suite would include tools for professors, administration, and students.
Nice concept! Although I can imagine this only works in large classrooms, as where I'm used to have pretty small sized classes (~30).
I have had the opportunity to experiment with something like a voting system. The teacher mixed his lecture with some (multiple choice) questions the students can answer via the voting system. This created a far more dynamic lecture with instant feedback on the answers provided. Maybe a feature you want to consider? Because with the smartphones and tablet these days you eliminate the need for an extra device (like we were experimenting with).
Besides this all, the site looks broken here (Fx 12.0, Vista) because the CSS is not loaded. IE8 in Vista is also messy (with a loaded CSS file).
Our CompSci Prof. uses something built in-house by students to evaluate who understands what & whatnot in real-time too.
How long is this around? Two years. Works.
He uses his iPad to present his slides wirelessly to our beamer and a student voluntarily manages the polls and when enough people haven't understood a topic, the student hits a button and the slide updates the graph, then the Prof focuses on that topic a little more.
That sounds more complicated than it is, it's dead easy and flawless, despite the fact that our Prof is an iDiot or iNerd, hehe :) He's just a huuge fan of everything Apple.
It challenges everyone to cooperate, thankfully it's anonymous, so we can fool around too, but the Prof notices that :D
In my eyes, the skill to built this is really low compared to the other stuff we've done, but it's a nice tool. Unfortunately most Prof just don't fucking care and go-on with their one-man-show.
PS: Our Prof uses the stats and data generated to create the exam also ;)
Is everyone expected to have a smartphone now a days? I really don't want a class to require a smartphone to participate. This would be much better as a web app.
How did you try to communicate that? I cannot find any indication one way or the other. Just showing it on a laptop and a phone is not good enough - maybe you have native apps on multiple platforms. Plus the phrase "there's an app for that" is almost always associated with native apps, not web apps.
I would specifically say something like "nothing to download or install!".
I mentioned "Understoodit runs on devices that you and your students already own: smartphones, tablets, netbooks and notebooks." But I agree I think saying 'nothing to download or install' is a great idea, thanks!
37 comments
[ 3.4 ms ] story [ 78.2 ms ] thread-there's already an app for getting feedback if you don't understand... it's called RAISING YOUR HAND!!!!!11!
-are you going to add a "distractometer" to log how many times people are distracted by using and checking the app?
Ok, now that that's done. I would actually lose the "o-meter" phrasing and have the app simply say "Confused" and "Understand". In fact, you could even lose the "understand" interaction. I doubt people will be hitting a button all the time every time they "understand" something during a lecture. Just have one button for confusion.
Is there a way to post feedback (anonymously) at the end of a lecture?
Radical piviot: you know what this model of instant feedback would be good for? Occupy Wall Street and general assemblies.
I'm hoping to add a feature for leaving feedback at the end of a lecture, great idea!
Also great idea on the Occupy Wall Street idea!
Overwhelming majority of students are deathly afraid of looking like an idiot by raising their hand. It happens everywhere, not just large classrooms.
I think this is a great idea.
This app would be a tremendous help not just to my classes (I've been here a long while, and know how to read the signs of confusion & check understanding), but also to the classes of newbies in Korea.
New teachers here sometimes go for months naively asking students "do you understand this stuff" and believing it to be true.
If Koreans could vote on this anonymously, I think they would actually do that at every opportunity.
Wish I had that for my calc lectures...
This is far better. There will always be a couple people that are not following. If the professor is getting 90% "I understood" responses, he can simply mention that anyone with questions on that topic is free to meet them after class, while the rest of the time can be spent on topics where there is a lower rate of understanding.
That having been said, I had a few questions/concerns:
- How would it work when there were specific gaps? It might be helpful if comments could be included to the instructor when the rating is provided. That would allow the teacher to reinforce those particular topics.
- How would you guarantee participation? If, for example, only people that don't understand the topic speak up, wouldn't that skew the data presented to the teacher?
Participation is a much trickier issue. In the classes that I tested in about 40 - 60% of the students actually used it. I was thinking of weighting the confusion 'votes' based on 'active' users.
I'm hoping that through feedback form professors I'll be able to add features that make it more worth while for students.
I think it would be awesome if you can generalize the idea for online lectures also (coursera, khan academy, udacity, etc). Since videos are not real time, letting students hit "confused" for certain segment of an online video will let the developers splice in a better explanation for bad parts of a video or even let them a/b test two different versions of a segment of the same video.
And also, instead of a confused button, maybe allow them to optionally elaborate on the source of their confusion? A potential solution for online videos is via timed comments (similar to how soundcloud does it, except maybe with interval of confusions instead http://37signals.com/svn/posts/1143-soundcloud-expands-the-a...)
Thinking freely here:
Could you produce a report for the school's administration of each professor's confused/understood ratio?
As gee-totes suggested, a messaging system would close the feedback loop nicely.
Is there a means of authenticating students? If this tool became a means of evaluation, it would be open to gaming by cagey professors.
Anyway, nice work, good video, great idea!
Great suggestion, producing a report is a great idea. I could even use the confusion data to predict exam test scores.
If students are going to be evaluating their teachers (providing richer feedback than confused/understood), then you're correct there will have to be authentication for students.
For authentication, I'd make it so that when the professor each session, the class is given a random 4-5 digit code. That way the students would have to be in class to get the code and participate.
You may also want to think about adding the ability for the instructor to add different buttons, not just "understood / confused". You could then have class polls, or group questions to see how many people are paying attention.
This should all be able to be controlled from the instructor dashboard.
okay class, we just went over big-O. here is an algorithm... what do you think the big-O of this is: O(n), O(n^2), etc...
Nothing that you'd have to keep track of the answers for longer than the length of the class.
Plus, if you added logins, that would just make it that much more of a setup / hassle to get classes going.
That said, this is a great idea. This is the kind of real-time feedback that I would love to have access to. Many of my students would rather fail than put their hand up and ask for help. Something like this would really help them out.
I like the idea of the code to authenticate students. I guess you'd have to watch out for selection effects in all of this too... an interesting challenge!
That would probably make professors (especially non-tenured professors) either terrified of making use of the app, or terrified of tackling complicated subject matter. Even if administrator's announce they aren't going to use the data to decide salary or tenure issues, good luck getting the professors to believe it.
Feedback that helps professors improve is great. Yet another administrative metric that warps people's behaviors, not so much.
I hear what you're saying about avoiding complicated subject matter - that's an interesting effect I hadn't considered.
That being said: I was the victim of several "bad" profs in university. Thing is I don't think they were "bad" - I just don't think they knew that they weren't "good" and even if they had, wouldn't have had any idea how move from bad to good.
I see this app as one part of a larger equation around designing systems that produce quality education. A complete suite would include tools for professors, administration, and students.
I have had the opportunity to experiment with something like a voting system. The teacher mixed his lecture with some (multiple choice) questions the students can answer via the voting system. This created a far more dynamic lecture with instant feedback on the answers provided. Maybe a feature you want to consider? Because with the smartphones and tablet these days you eliminate the need for an extra device (like we were experimenting with).
Besides this all, the site looks broken here (Fx 12.0, Vista) because the CSS is not loaded. IE8 in Vista is also messy (with a loaded CSS file).
I'm using a web font from font deck and it seems to be slow loading which blocks the rest of the page from loading...
This was being used before the smartphone, so imagine similar products exist. However your metrics look meaningful.
It isn't web-based, and IIRC they offer/sell a voting widget to TV producers that can be given to audience members to vote with.
Worthy of a "web 2.0/iOS" disruption me thinks...
[1]http://www.quicktally.com/
P.S. Well done OP, that's a great concept and nicely (MVP?) executed.
How long is this around? Two years. Works.
He uses his iPad to present his slides wirelessly to our beamer and a student voluntarily manages the polls and when enough people haven't understood a topic, the student hits a button and the slide updates the graph, then the Prof focuses on that topic a little more.
That sounds more complicated than it is, it's dead easy and flawless, despite the fact that our Prof is an iDiot or iNerd, hehe :) He's just a huuge fan of everything Apple.
It challenges everyone to cooperate, thankfully it's anonymous, so we can fool around too, but the Prof notices that :D
In my eyes, the skill to built this is really low compared to the other stuff we've done, but it's a nice tool. Unfortunately most Prof just don't fucking care and go-on with their one-man-show.
PS: Our Prof uses the stats and data generated to create the exam also ;)
I would specifically say something like "nothing to download or install!".