The concept of calibration - actually being right at the rate that matches your estimated percent confidence - is important for machine learning. It's important to know how well you can trust the prediction of a ML model (ironically to that you can do something like send low confidence predictions for manual review, which apparently might not be that calibrated either)... Deep learning models in particular are not good at accurately estimating confidence, so there is a whole body of work on getting better confidence estimates, e.g. [0]. It's nice to see that people make the same kind of mistakes.
Given that people had a bimodal distribution of confidence, I would assume that means a lot of the perceived overconfidence come from people answering a question wrong that they thought was easy. I wonder if the test takers would say “that was a trick question”.
Yeah, I had the thought that the math might work out in such a way that this is the expected result.
That is, if you say you are "50%" confident, it really means you have no confidence - simply guessing would get you 50%, so you can't go less than that. But on the other end, for people who say they're 100% or 95% confident, it's pretty much all downside since you can't go over 100%.
Yeah I'm going through https://www.congap.com/ and I see quite a few questions where the correct answer is often the one that goes against intuition. Thus, low information responders might make a reasoned guess and assign somewhat greater than 50% confidence when in reality they have a less than 50% chance of getting it right.
Incidentally, I pretty quickly learned to guess the most unusual answer for questions I have no direct information about.
Google doesn't tell me word one about "Black Magin Woman" and "Feetwood Mac"; or about "stop sides". If it tells you, I feel 100% confident that you are hallucinating.
The quiz reliably informed me that I am consistently underconfident.
Well, then, you could have communicated more clearly without being deliberately obtuse.
Did you also communicate to the author that there were many typos in the survey? If not, you probably just deliberately skewed the survey for your own amusement.
The book 'How to measure anything' talks about how to improve your own calibration.
From what I recall (it's been a while since I read it), the author recommended testing yourself to estimate a bunch of questions with numerical answers. For example: what's the height of the empire state building?
You write down a range for each answer, trying to make the range narrow enough that you're just about 80% confident that the actual answer is within the range.
By doing this repeatedly, and periodically reviewing your cumulative correct rate, you can calibrate appropriately (e.g. widening or narrowing your ranges for future questions).
Job applicants for junior trading (market making) roles are often given similar questions. An example question might be:
"Make me a market on the height of the empire state building."
By this the interviewer means: what price would you buy at and what price would you sell at?
So for example I might respond 1000/2000. which means I'm buying at 1000ft and selling at 2000ft. Presumably, I believe the true height is somewhere between these two numbers (ideally 1500ft), at a certain confidence level. The interviewer might ask you to quote a spread given a confidence interval.
The interviewer might then lift your bid or ask and then ask you to make a new market. There's a couple factors at play here:
1. The interviewer has more information than you (presumably), so whether he bought from or sold to you is valuable information. You might want to raise your quotes if he buys in order to capitalize on the new information.
2. You want to decrease your risk. So for example if he just lifted your bid, you might want to lower your quotes: offer a lower price to buy and a lower price to sell. A good market maker capture the difference between the bid-ask spread: the price for which you buy at and the price that you sell at (like any other middle man). By lowering your quotes, you are increasing the probability that you will flatten out your risk (you are not trying to speculate on the height of the empire state building).
These two things are in contention, and the interviewer wants to see how you think about it/deal with it.
Also the interviewer wants to ascertain whether your confidence levels are correct over multiple questions. ie if the confidence level is 80%, then you should be wrong (true value lies outside your range) about 20% of time. It is not good to be either under or over confident. Accurately dealing with uncertainty is what they are looking for.
Anyways I thought this was a very similar process that you were referring to, except you are dealing with money. Which always helps to clarify things!
Your factor #1 above (about adverse selection through trading with a well-informed counterparty) reminded me of something I read in 'Trading and Exchanges'.
The section 'The Adverse Selection Spread Component' says that dealers will set their bid/ask spreads to account for the information they will get when someone hits their bid or whatever. So they need to estimate the probability that their next trade is with an informed counterparty.
If that probability is high, then they'll quote a higher spread.
Conversely, if they are quoting for retail order flow, they may assume they're never dealing with informed counterparties, and can offer a tighter spread.
Exactly! Being a market maker means constantly interacting with counterparties that have more information than you. In order to be successful you must utilize the core information they do not have: you are in the center of the order flow. Essentially, you must try to infer the knowledge that others possess by utilizing information about how they trade with you.
Curious, does this confidence level factor into any sort of Kelly Criterion-type calculation for what percentage of a portfolio to risk during a given trade? What quantitative factors enable someone to calibrate their confidence level for a market making event?
They still ask these questions...they were asking them in the 2000s, I actually assumed they stopped asking this stuff.
They demonstrate absolutely nothing. All they test for is whether you know enough about the recruiting process to know they are going to ask you some daft question about the number of windows in a city (this is called second-level thinking, it is useful in finance, it is always interesting that ppl who work in finance don't understand that...they usually ask these questions because someone asked them at their interview).
If you want to know whether someone can trade: ask them what they are trading already, and just sit them down at the desk and see what happens.
Investment banks are the grade-A example of terrible/illogical hiring (although one that largely achieves the goal of hiring more people like the people who already work at investment banks).
Also, thinking that you can master uncertainty isn't really the point. You can't. The whole discussion is pointless. It took me a very long time to realise that the neuralgia that people who work in finance have around uncertainty (largely a function of hiring practices at banks/educational background and including myself) was predictably profitable (the second-level thinking here is realising that someone lifting your bid/offers in a trading game doesn't represent reality because in reality, the person you are trading with under conditions of uncertainty doesn't have perfect information or even more information than you...so when uncertainty increases, the potential profit increases). Thinking that you can harness the vagaries of life and your own irrationality or poor decision-making skills is, however, common amongst people with a lot of education...and totally wrong (it is an irrational fear of uncertainty borne of a heavily structured life with prescribed questions and answers).
I agree. I would say intelligence is what you have naturally, education is what you acquire in school, and wisdom is what you acquire through experience. These are not interchangeable.
I'd say smart is what you have naturally, education is what you acquire in school, intelligence is a snapshot of what you actively remember from education plus what you've learned via recent experience, and wisdom is a scratchpad where your brain makes a note whenever it comes across something that's relevant to all of the above.
Nice summary, but I dislike the fact that education has made the default understanding of these categories to be about retaining information/knowledge rather that your ability to compute/reason/intuit.
Yes, familiarity with a field should improve skill, but I like to think that one can be smart, intelligent or wise in a situation that does not involve previous information about it.
Someone exposed to a brand new game or puzzle can be smart, intelligent or wise in solving it separately (to some degree) from having prior information.
By three methods we may learn wisdom: First, by reflection, which is noblest; Second, by imitation, which is easiest; and third by experience, which is the bitterest.
It's also important to distinguish between 'knowing' and 'understanding'.
If someone tells me how to says Good morning in, let's say, Mongolian, I 'know' how to greet people, but dont 'understand' how it works (can't tell which part of the sound/symbol represents what, etc.)
I think anyone who has inhabited planet earth for more than 25-30 years or so starts to realize that most of people are either:
1. “Geniuses” of speculation and every topic under the sun (overconfident braggarts that make sweeping claims that are difficult to criticize because they are so general; such dreamers talk big but really have little expertise)
2. People that actually are “deep-knowledge” experts, but only in one particular, very specialized domain and can otherwise barely tie their shoelaces
And that the 1. Kind of people make visionary, ridiculous claims that are then tempered and shored up by the 2. Kind of people behind the scenes that actually do the work so as not to cause horrific catastrophes and that true genius is rare and hardly ever recognizable as such within the genius’s own lifetime.
If you have no knowledge but boundless confidence and a good sense for what kind of statements you can make without actually resulting in direct (i.e. traceable to your statements) physical or financial injury to other people you can get pretty far in life.
I’m aware of the irony that the claims in this post put me in the 1. category.
There's a subset of 1 who are good at making wild guesses while leaving open the possibility they're wrong so 2 has an opportunity to share what they know.
I don't have actual expert knowledge in anything but a bit of knowledge of a broad range of things. So there's certainly a spectrum of "1" where you have all levels of actual knowledge.
It reads like someone is just grumpy dividing it up like that.
There are also a lot of people who know nothing and struggle to understand anything new and don't either brag about it or have any deeper field on understand. They can also usually tie their shoelace.
Of course, and of all the categories, it is those of this bunch that actually keep the world running in terms of what ultimately matters for us humans.
You do have a point though (why I recall decade old comics as conversation pieces and not people's names... sigh) There are a lot of people on "mount stupid", in fact the chance that someone talking about a subject is trapped in the overconfidence of mount stupid is very high. In other words, most of the people you'll hear talking about something are entirely full of shit. (this is a large source of problems in our time)
There are people who don't know anything and realize it, then there are a whole lot of people who know enough to be aware that their knowledge is limited and don't want to try to talk over the overconfident folks (refuting someone sure of themselves is a lot more difficult than jabbering about your own feelings)
The people who really do know a whole lot and have the confidence to match are rare.
One thing that sticks out on HN are long discussions about things that folks have not much of an idea about. Non-computer-related engineering and farming threads are usually a shitshow of overconfident nonsense. Something to notice about computer people is they have the confidence that they can figure anything out (generally good) but frequently lack the wisdom to be aware of their knowledge (generally bad).
> I'm skeptical of any sorting of things into two bins, people or otherwise.
I get what you are saying, but let me elaborate it a little bit for fun.
If you project anything into one dimension and define a threshold, you have 2 bins, and you describe it with 1 bit. The fact that you are projecting does not make it an suspicious oversimplification per se.
You can then add more dimensions into study and further classify the part of reality under study, and get more fancy with your bins, but from several points of view, there will be one master bit - that is, you have a classification tree, and a first level classification. So you can for sure talk about 2 bins, irrespective that there are other ways to look at the data, or funny cases near the threshold. So for example it makes sense to talk about people from Fidji vs. people not from Fidji in some contexts, and that should not be suspicious.
So really what we should be suspect is from people who constantly jump to oversimplified models of reality to make grandiose takes on reality without first spending some time in explaining why that dimension dominates over others, why that threshold in particular, and why alternatives are not so useful as this simple model.
I have doubts about this study. It could be that participants are thinking about the questions differently than the author anticipates.
For example, one could argue that a stop sign has ten sides and not eight because it is a physical object with a front and back. And if it has circular bolt holes for mounting to a post then you could also say it has many more sides than ten.
This is fair to be part of the study although could be varied a bit more. The error that participants make in assumptions is surprise that the market does price.
The math behind this is wrong. It’s taking a naive average of your confidence scores so that if your average confidence level is 95% then you must think you got 9.5 of 10 correct.
No, if I answered 95% confidence on all 10 then my estimate of the probability I got at least one wrong is 1 - .95^10 = 40%. Whereas if I answered 100% confidence on 9 and 50% confidence on one, then i am predicting 50% probability that I got 9, and 50% probability I got all 10.
I assume the paper is riddled with similar mathematical errors. You can’t just average probabilities together like that.
58 comments
[ 4.9 ms ] story [ 116 ms ] thread[0]Guo et al., https://arxiv.org/abs/1706.04599
That is, if you say you are "50%" confident, it really means you have no confidence - simply guessing would get you 50%, so you can't go less than that. But on the other end, for people who say they're 100% or 95% confident, it's pretty much all downside since you can't go over 100%.
Incidentally, I pretty quickly learned to guess the most unusual answer for questions I have no direct information about.
I suspect that's a strategy many people who are both educated and intelligent would pick up on.
And I confess I have no idea what a stop side is. But apparently I was correct to guess that it does not have six sides.
And I remain dubious that the Mona Lisa was painted by someone other than Leonardo.
The quiz reliably informed me that I am consistently underconfident.
Did you also communicate to the author that there were many typos in the survey? If not, you probably just deliberately skewed the survey for your own amusement.
On those questions I put the slider at 50%.
From what I recall (it's been a while since I read it), the author recommended testing yourself to estimate a bunch of questions with numerical answers. For example: what's the height of the empire state building?
You write down a range for each answer, trying to make the range narrow enough that you're just about 80% confident that the actual answer is within the range.
By doing this repeatedly, and periodically reviewing your cumulative correct rate, you can calibrate appropriately (e.g. widening or narrowing your ranges for future questions).
"Make me a market on the height of the empire state building."
By this the interviewer means: what price would you buy at and what price would you sell at?
So for example I might respond 1000/2000. which means I'm buying at 1000ft and selling at 2000ft. Presumably, I believe the true height is somewhere between these two numbers (ideally 1500ft), at a certain confidence level. The interviewer might ask you to quote a spread given a confidence interval.
The interviewer might then lift your bid or ask and then ask you to make a new market. There's a couple factors at play here:
1. The interviewer has more information than you (presumably), so whether he bought from or sold to you is valuable information. You might want to raise your quotes if he buys in order to capitalize on the new information.
2. You want to decrease your risk. So for example if he just lifted your bid, you might want to lower your quotes: offer a lower price to buy and a lower price to sell. A good market maker capture the difference between the bid-ask spread: the price for which you buy at and the price that you sell at (like any other middle man). By lowering your quotes, you are increasing the probability that you will flatten out your risk (you are not trying to speculate on the height of the empire state building).
These two things are in contention, and the interviewer wants to see how you think about it/deal with it.
Also the interviewer wants to ascertain whether your confidence levels are correct over multiple questions. ie if the confidence level is 80%, then you should be wrong (true value lies outside your range) about 20% of time. It is not good to be either under or over confident. Accurately dealing with uncertainty is what they are looking for.
Anyways I thought this was a very similar process that you were referring to, except you are dealing with money. Which always helps to clarify things!
The section 'The Adverse Selection Spread Component' says that dealers will set their bid/ask spreads to account for the information they will get when someone hits their bid or whatever. So they need to estimate the probability that their next trade is with an informed counterparty.
If that probability is high, then they'll quote a higher spread.
Conversely, if they are quoting for retail order flow, they may assume they're never dealing with informed counterparties, and can offer a tighter spread.
They demonstrate absolutely nothing. All they test for is whether you know enough about the recruiting process to know they are going to ask you some daft question about the number of windows in a city (this is called second-level thinking, it is useful in finance, it is always interesting that ppl who work in finance don't understand that...they usually ask these questions because someone asked them at their interview).
If you want to know whether someone can trade: ask them what they are trading already, and just sit them down at the desk and see what happens.
Investment banks are the grade-A example of terrible/illogical hiring (although one that largely achieves the goal of hiring more people like the people who already work at investment banks).
Also, thinking that you can master uncertainty isn't really the point. You can't. The whole discussion is pointless. It took me a very long time to realise that the neuralgia that people who work in finance have around uncertainty (largely a function of hiring practices at banks/educational background and including myself) was predictably profitable (the second-level thinking here is realising that someone lifting your bid/offers in a trading game doesn't represent reality because in reality, the person you are trading with under conditions of uncertainty doesn't have perfect information or even more information than you...so when uncertainty increases, the potential profit increases). Thinking that you can harness the vagaries of life and your own irrationality or poor decision-making skills is, however, common amongst people with a lot of education...and totally wrong (it is an irrational fear of uncertainty borne of a heavily structured life with prescribed questions and answers).
Smart does not imply having knowledge.
Yes, familiarity with a field should improve skill, but I like to think that one can be smart, intelligent or wise in a situation that does not involve previous information about it.
Someone exposed to a brand new game or puzzle can be smart, intelligent or wise in solving it separately (to some degree) from having prior information.
By three methods we may learn wisdom: First, by reflection, which is noblest; Second, by imitation, which is easiest; and third by experience, which is the bitterest.
The Wise Mind is the combination of the Logical Mind and the Emotional Mind.
If someone tells me how to says Good morning in, let's say, Mongolian, I 'know' how to greet people, but dont 'understand' how it works (can't tell which part of the sound/symbol represents what, etc.)
1. “Geniuses” of speculation and every topic under the sun (overconfident braggarts that make sweeping claims that are difficult to criticize because they are so general; such dreamers talk big but really have little expertise) 2. People that actually are “deep-knowledge” experts, but only in one particular, very specialized domain and can otherwise barely tie their shoelaces
And that the 1. Kind of people make visionary, ridiculous claims that are then tempered and shored up by the 2. Kind of people behind the scenes that actually do the work so as not to cause horrific catastrophes and that true genius is rare and hardly ever recognizable as such within the genius’s own lifetime.
If you have no knowledge but boundless confidence and a good sense for what kind of statements you can make without actually resulting in direct (i.e. traceable to your statements) physical or financial injury to other people you can get pretty far in life.
I’m aware of the irony that the claims in this post put me in the 1. category.
It reads like someone is just grumpy dividing it up like that.
https://www.smbc-comics.com/comic/2011-12-28
You do have a point though (why I recall decade old comics as conversation pieces and not people's names... sigh) There are a lot of people on "mount stupid", in fact the chance that someone talking about a subject is trapped in the overconfidence of mount stupid is very high. In other words, most of the people you'll hear talking about something are entirely full of shit. (this is a large source of problems in our time)
There are people who don't know anything and realize it, then there are a whole lot of people who know enough to be aware that their knowledge is limited and don't want to try to talk over the overconfident folks (refuting someone sure of themselves is a lot more difficult than jabbering about your own feelings)
The people who really do know a whole lot and have the confidence to match are rare.
One thing that sticks out on HN are long discussions about things that folks have not much of an idea about. Non-computer-related engineering and farming threads are usually a shitshow of overconfident nonsense. Something to notice about computer people is they have the confidence that they can figure anything out (generally good) but frequently lack the wisdom to be aware of their knowledge (generally bad).
Reminds me of the saying that there are 10 kinds of people in the world: those who understand binary and those who don’t.
Those who understand binary,
Those who don't,
Those who understand ternary...
I get what you are saying, but let me elaborate it a little bit for fun.
If you project anything into one dimension and define a threshold, you have 2 bins, and you describe it with 1 bit. The fact that you are projecting does not make it an suspicious oversimplification per se.
You can then add more dimensions into study and further classify the part of reality under study, and get more fancy with your bins, but from several points of view, there will be one master bit - that is, you have a classification tree, and a first level classification. So you can for sure talk about 2 bins, irrespective that there are other ways to look at the data, or funny cases near the threshold. So for example it makes sense to talk about people from Fidji vs. people not from Fidji in some contexts, and that should not be suspicious.
So really what we should be suspect is from people who constantly jump to oversimplified models of reality to make grandiose takes on reality without first spending some time in explaining why that dimension dominates over others, why that threshold in particular, and why alternatives are not so useful as this simple model.
For example, one could argue that a stop sign has ten sides and not eight because it is a physical object with a front and back. And if it has circular bolt holes for mounting to a post then you could also say it has many more sides than ten.
"sides" usually is a term describing 2d objects,
3d objects are described by faces, edges, and vertices
No, if I answered 95% confidence on all 10 then my estimate of the probability I got at least one wrong is 1 - .95^10 = 40%. Whereas if I answered 100% confidence on 9 and 50% confidence on one, then i am predicting 50% probability that I got 9, and 50% probability I got all 10.
I assume the paper is riddled with similar mathematical errors. You can’t just average probabilities together like that.