Almost all polls are worthless, because the decision to respond to the poll is itself a major portion of the signal.
It takes extensive work to get around that problem alone, let alone "viral" and other worthless push-polls - work that even the very best in the industry, at the highest level, can make major mistakes on.
>Almost all polls are worthless [...] because the decision to respond to the poll is itself a major portion of the signal.
But most polls I see are from professional pollsters like Pew or YouGov? For them, correcting for this is opinion polls 101. That's why you see "representative sample". They're not just randomly polling 1000 people and taking the average. They're adjusting for a bunch of other demographic factors as well.
No matter how they adjust demographic factors they cannot control for the willingness to respond to polls at all. They are always subsetting this weird population set that shares that strange variable of being willing to spend ten plus uncompensated minutes with a spam call pollster.
This is appealing to authority, assuming the work ought to be perfect just because people are professionals. What if there is no good way to control for these latent factors? What if the latent factors change by the year and their magnitude of effect on the overall result changes? All this happens. No matter what you model with statistics, you are only ever presenting your model and its assumptions based on the data that were used with it. You can never say "now I understand reality" if you actually believe in statistics as they are and not just the popular religion of 'statistics'.
The entire world works around the legitimacy of authority.
Yes - the quality varies, and pollsters don't have a high degree of institutional authority.
But some do.
If you want to learn about 'institutional legitimacy' - go visilt Pew.
Pew is an extremely well regarded institution. They are extremely good at polling. Go look at their methods.
They are a long running institution with serious professionals, big budgets, people who really know waht they are talkinga bout.
It's the difference between getting diagnosis from 'a neighbour' vs a 'doctor' but in this case there is no specific qualification.
Some 'respected pollsters' are basically insiders who have been doing it for a long time, with relationships and credibility - others, more institutional.
Check out Pew, they are probably he closest thing to institutional professional as possible.
Political polling is inherently hard - they will be the first to admit.
This is totally wrong. Accounting for partisan non-response bias is a standard part of a pollsters job, not some insurmountable gotcha. Some are better than others at this. Polls aren't oracular, nor are they claiming to be. It's almost always better to look at an average of polls rather than a single one.
There is a lot of public data on past elections that anyone can analyze and see exactly how accurate polls have been in the past. If you're expecting polls to be a perfectly accurate prediction then you are misunderstanding their purpose. What they are great at is giving a probabilistic estimate of the outcome, within a certain range.
It's likely that any key race you care about next week will end up within 4 points of the current polling average. It's extremely likely that it will end up within 8 points. That's useful information!
This filter isn't specific to polls. Anything that reaches people through reshares gets selected on how surprising or pleasing it is, so the visible sample skews toward the tails. Marketing has the same problem with screenshots of campaign results: the ones that travel are the outliers, and anyone calibrating on what they've seen shared ends up with expectations nobody can hit.
Curious whether the effect survives controlling for pollster. Some pollsters have consistent house effects and some have much bigger followings, so part of the slope could be "polls from well-known outfits get reposted more" rather than "favorable margins get reposted more".
All polls are flawed. Remember last US pres election polls by CNN? Completely wrong. Why? Because they surveyed mostly left-"learing" participants.
Right and supposedly middle (this doesn't exist) media does the same.
That means we gotta find the average as you suggested.
---
Side note
> Given that Musk has absolutely destroyed the Twitter API for academic use
Why bring up Musk, when you can simply say, "Twitter lowered Academic API rate limit because ...". You sound like you need someone to blame not professional and politically motivated (yes the post is about the political poll but this shows the "bias" and his supposed poll will be biased depending on how he gathered data).
Twitter would have the same way of generating metrics for the justification as you are trying to justify the flaw in the poll outcome.
Most likely the decision is based on their internal metrics/dashboard report
In 2021 Twitter was encouraging free academic use of the API [0].
This was in line with long standing behaviors in this space [1].
Musk bought Twitter in 2022. In 2023, they began substantially restricting API access versus how it had been for years.
It is not unfair at all to look at that timeline, look at Musk's acquisition as the biggest change that had happened to the company in that timeframe, and draw a relationship.
- All reputable firms try get valid results, they have professionals, they know who they are talking to and the limits. What we see in the last 2-3 weeks of the elections are the polls they want to release however, those that err in one direction or another.
- They definitely accommodate for misrepresentation samples - if more of their respondents indicate 'registered republicans' than in the general public they will adjust.
- It's very hard to measure 1) the people that are hard to reach and statistically more likely to not trust pollsters and 2) likelihood of actual voting aka enthusiasm.
- there are plenty of centrist institutions and pollsters
- It's fair to call out Elon because he's making the policies
Twitter is just a bad way to measure anything except for the Zeitgeist of Twitter readers.
I am upvoting your comment for correcting my knowledge on CNN not doing their own polling.
Some I do not really agree, as reputable firms != trustworthy polls.
e.g. Check out Times magainzes "Best Inventions of the Year, John Termus" (https://time.com/collection/best-inventions/2026/apple-iphon...) wtf is that. Time magazine might be "reputable" but whatever they discuss is not trustworthy given Appl duo being crap.
TIME magazine is a tabloid-ish reader publication. It's popular editorial, not 'highly objective news'. Who TIME puts on their cover has nothing to do with the quality of the polls they might cite.
> Remember last US pres election polls by CNN? Completely wrong.
I do. I don't remember that at all. I think you might have some odd definition of completely wrong though.
But please, by all means, share these 100% wrong polls that showed, for example, Trump having 0% chance of winning. Most of the polls I saw were fairly accurate within their margin of error and made it clear the race was tight (which, in the end, it was). And nothing I've seen has changed that, but if you have polls from CNN specifically that you can point to, by all means.
All polls are not representative. Stated preferences are not revealed preferences. People are terrible at answering honestly. You have to surveil their behavior to understand them.
I think we should do our best to ruin polling data.
Polls are the first step in a pipeline that effectively turns money into votes. The better that pipeline functions, the worse our democracy functions. Let the only accurate poll be the election itself.
I actually really disagree with this. If we want our representatives to reflect the will of the people, they need to have an accurate understanding of their electorate. When issues come up we need a way for the people to be heard, and polling is an important part of that.
This conditional is true, but there is a related one: If we want our representatives to outmaneuver the electorate on behalf of donors, they must have an accurate understanding of what their electorate is paying attention to, and what they are not.
Surfacing understanding of the electorates beliefs and desires is good but double edged. I think we can balance it out with yet other strategies and don't agree with GP's cynicism here but it's worth taking on directly.
I believe you're saying "Information is power and it can be used by an adversary to the people to outmaneuver them."
True, but that's not the only party which can use this. The people themselves can use it to build better coalitions and to hold representatives accountable for acting against their will.
I'm all for better coalition building, but is it truly the same data that enables both?
Expecting people to share their political leanings with any stranger who asks without encouraging them to make a friend/foe determination beforehand seems like it opens the playing field to interference just as well as it does to good faith feedback. Surely we can find a way to bias this interaction in the right direction somehow.
It's a good question. I prefer longer term questions about more fundamental beliefs, as that's what coalitions can form around. It also opens the door to solutions which take longer (more likely to work). Specifically, as an American, I advocate for a robust polling regime well before the primaries to encourage more popular candidates and slates to emerge.
I like where your headed with that. What would it look like to me, as the target of this good faith polling effort?
When I try to imagine the ideal case, it comes to me from somebody I know personally and see face-to-face regularly. Somebody who knows and trusts the candidate (perhaps transitively, for a hop or two). Somebody who wouldn't do the same at the behest of some billionaire looking to protect some shady project of theirs that's unfolding in my region.
Like, the information should flow back to leadership along a social graph that resembles the structure of the coalition we're building together. If not exactly peer to peer, from a transport perspective, then at least in a way that is authenticated by the presence of those peers.
If it was up to me, I'd implement a soul bound identity layer, upon which its trivial to make an actually private communication of a) your preferences and b) your status as a citizen who has not already broadcast your preferences - such that there is a legible commons of aggregate positions known to be held by real people by not known to be held by specific people. I haven't thought about this as deeply as it requires though, I know others have pointed out you need an economic aspect (e.g. quadratic voting) to understand not just the direction of everyones preferences but the weight and thus relative importance of them.
I like where you're trying to go, but I'm afraid that you might have a chicken/egg problem on your hands. You need an oracle for the soulbound identities, which requires a widely supported coalition, but to get such a coalition you need the tool to already work.
I figure the tools of the successful revolutionary become the tools of the new government, so whatever it is it's gotta be useful for a small minority that wants to coordinate in the face of a sophisticated adversary, and then once it grows up it's gotta be useful for carrying out the bureaucratic business of a government. Unless it survives that transition with its trustworthiness intact, it won't be seen by the masses as trustworthy enough for the job. People just aren't that keen on trusting computers on the say-so of nerds like us.
So for starters I imagine an ocean of Sybil identities controlled by an enemy. Some of these will be trying to get info about your intentions so they can interfere (polling of one kind). Also you'll have need to coordinate with comrades without making your entire trust graph known to all parties (polling of the other kind).
It'll never get to the point where coalitions are large if it doesn't function well when they're small, and at that phase you gotta be choosy about who you share info with.
34 comments
[ 0.22 ms ] story [ 3.6 ms ] threadIt takes extensive work to get around that problem alone, let alone "viral" and other worthless push-polls - work that even the very best in the industry, at the highest level, can make major mistakes on.
But most polls I see are from professional pollsters like Pew or YouGov? For them, correcting for this is opinion polls 101. That's why you see "representative sample". They're not just randomly polling 1000 people and taking the average. They're adjusting for a bunch of other demographic factors as well.
Yes - they are aware of that.
My point is --> this problem is the most basic and fundamental problem that is the focus of every pollsters existence <--
Asking questions is extremely easy.
Getting a 'truly random sample' is extremely hard.
Trying to factor in those things is 100% of the work, that's what they do, it's their job.
Some are better, some are lazier, some have better data, some are individuals, some are teams so it varies hugely.
When the conditions are controlled they'll have a better idea than otherwise.
Their job is not 'asking questions' it's 'figuring out how to get random samples'.
The entire world works around the legitimacy of authority.
Yes - the quality varies, and pollsters don't have a high degree of institutional authority.
But some do.
If you want to learn about 'institutional legitimacy' - go visilt Pew.
Pew is an extremely well regarded institution. They are extremely good at polling. Go look at their methods.
They are a long running institution with serious professionals, big budgets, people who really know waht they are talkinga bout.
It's the difference between getting diagnosis from 'a neighbour' vs a 'doctor' but in this case there is no specific qualification.
Some 'respected pollsters' are basically insiders who have been doing it for a long time, with relationships and credibility - others, more institutional.
Check out Pew, they are probably he closest thing to institutional professional as possible.
Political polling is inherently hard - they will be the first to admit.
There is a lot of public data on past elections that anyone can analyze and see exactly how accurate polls have been in the past. If you're expecting polls to be a perfectly accurate prediction then you are misunderstanding their purpose. What they are great at is giving a probabilistic estimate of the outcome, within a certain range.
It's likely that any key race you care about next week will end up within 4 points of the current polling average. It's extremely likely that it will end up within 8 points. That's useful information!
Curious whether the effect survives controlling for pollster. Some pollsters have consistent house effects and some have much bigger followings, so part of the slope could be "polls from well-known outfits get reposted more" rather than "favorable margins get reposted more".
Right and supposedly middle (this doesn't exist) media does the same.
That means we gotta find the average as you suggested.
---
Side note
> Given that Musk has absolutely destroyed the Twitter API for academic use
Why bring up Musk, when you can simply say, "Twitter lowered Academic API rate limit because ...". You sound like you need someone to blame not professional and politically motivated (yes the post is about the political poll but this shows the "bias" and his supposed poll will be biased depending on how he gathered data).
Twitter would have the same way of generating metrics for the justification as you are trying to justify the flaw in the poll outcome.
Most likely the decision is based on their internal metrics/dashboard report
This was in line with long standing behaviors in this space [1].
Musk bought Twitter in 2022. In 2023, they began substantially restricting API access versus how it had been for years.
It is not unfair at all to look at that timeline, look at Musk's acquisition as the biggest change that had happened to the company in that timeframe, and draw a relationship.
[0] https://newsonline.library.vanderbilt.edu/2021/03/use-twitte...
[1] https://arxiv.org/abs/2404.07340
- CNN doesn't do polling, they have firms do that
- All reputable firms try get valid results, they have professionals, they know who they are talking to and the limits. What we see in the last 2-3 weeks of the elections are the polls they want to release however, those that err in one direction or another.
- They definitely accommodate for misrepresentation samples - if more of their respondents indicate 'registered republicans' than in the general public they will adjust.
- It's very hard to measure 1) the people that are hard to reach and statistically more likely to not trust pollsters and 2) likelihood of actual voting aka enthusiasm.
- there are plenty of centrist institutions and pollsters
- It's fair to call out Elon because he's making the policies
Twitter is just a bad way to measure anything except for the Zeitgeist of Twitter readers.
Some I do not really agree, as reputable firms != trustworthy polls.
e.g. Check out Times magainzes "Best Inventions of the Year, John Termus" (https://time.com/collection/best-inventions/2026/apple-iphon...) wtf is that. Time magazine might be "reputable" but whatever they discuss is not trustworthy given Appl duo being crap.
I do. I don't remember that at all. I think you might have some odd definition of completely wrong though.
But please, by all means, share these 100% wrong polls that showed, for example, Trump having 0% chance of winning. Most of the polls I saw were fairly accurate within their margin of error and made it clear the race was tight (which, in the end, it was). And nothing I've seen has changed that, but if you have polls from CNN specifically that you can point to, by all means.
The entire objective of a polls is to get a probabilistic sample. That's the whole thing. Anything that deviates that ... creates problematic outcome.
Polls are the first step in a pipeline that effectively turns money into votes. The better that pipeline functions, the worse our democracy functions. Let the only accurate poll be the election itself.
What? You’re stating this like it’s self-evident, but I don’t follow.
> I'll give you a gift card if you tell me how you're going to vote tomorrow
and:
> I want to represent you well, lets talk about what's important to you
I wouldn't propose we interfere with the latter. Or is the former truly useful for improving the quality of representation?
Surfacing understanding of the electorates beliefs and desires is good but double edged. I think we can balance it out with yet other strategies and don't agree with GP's cynicism here but it's worth taking on directly.
It's just clearer communication if you state your position plainly rather than bedazzling it with a bunch of disclaimers.
True, but that's not the only party which can use this. The people themselves can use it to build better coalitions and to hold representatives accountable for acting against their will.
Expecting people to share their political leanings with any stranger who asks without encouraging them to make a friend/foe determination beforehand seems like it opens the playing field to interference just as well as it does to good faith feedback. Surely we can find a way to bias this interaction in the right direction somehow.
When I try to imagine the ideal case, it comes to me from somebody I know personally and see face-to-face regularly. Somebody who knows and trusts the candidate (perhaps transitively, for a hop or two). Somebody who wouldn't do the same at the behest of some billionaire looking to protect some shady project of theirs that's unfolding in my region.
Like, the information should flow back to leadership along a social graph that resembles the structure of the coalition we're building together. If not exactly peer to peer, from a transport perspective, then at least in a way that is authenticated by the presence of those peers.
I figure the tools of the successful revolutionary become the tools of the new government, so whatever it is it's gotta be useful for a small minority that wants to coordinate in the face of a sophisticated adversary, and then once it grows up it's gotta be useful for carrying out the bureaucratic business of a government. Unless it survives that transition with its trustworthiness intact, it won't be seen by the masses as trustworthy enough for the job. People just aren't that keen on trusting computers on the say-so of nerds like us.
So for starters I imagine an ocean of Sybil identities controlled by an enemy. Some of these will be trying to get info about your intentions so they can interfere (polling of one kind). Also you'll have need to coordinate with comrades without making your entire trust graph known to all parties (polling of the other kind).
It'll never get to the point where coalitions are large if it doesn't function well when they're small, and at that phase you gotta be choosy about who you share info with.