Launch HN: Roundtable (YC S23) – Using AI to Simulate Surveys
Surveys are incredibly important for user and market research, but are expensive and take months to design, run, and analyze. By simulating responses, our users can get results in seconds and make decisions faster. See https://roundtable.ai/showcase for a bunch of examples, and https://www.loom.com/share/eb6fb27acebe48839dd561cf1546f131 for a demo video.
Our product lets you add questions (e.g. “how old are you”) and conditions (e.g. “is a Hacker News user”) and then see how these affect the survey results. For example, the survey “Are you interested in buying an e-bike?” shows ‘yes’ 28% [1]. But if you narrow it down to people who own a Tesla, ‘yes’ jumps to 52% [2]. Another example: if you survey “where did you learn to code”, the question “how old are you?” makes a dramatic difference—for “45 or older” the answer is 55% “books” [3], but for “younger than 45” it’s 76% “online” [4]. One more: 5% of people answer “legroom” to the question “Which of the following factors is most important for choosing which airline to fly?” [5], and this jumps to 20% when you condition on people over six feet tall [6].
You wouldn’t think (well, we didn’t think) that such simulated surveys would work very well, but empirically they work a lot better than expected—we have run many surveys in the wild to validate Roundtable's results (e.g. comparing age demographics to U.S. Census data). We’re still trying to figure out why. We believe that LLMs that are pre-trained on the public Internet have internalized a lot of information/correlations about communities (e.g. Tesla drivers, Hacker News, etc.) and can reasonably approximate their behavior. In any case, researchers are seeing the same things that we are. A nice paper by a BYU group [7] discusses extracting sub-population information from GPT/LLMs. A related paper from Microsoft [8] shows how GPT can simulate different human behaviors. It’s an active research topic, and we hope we can get a sense of the theoretical basis relatively soon.
Because these models are primarily trained on Internet data, they start out skewed towards the demographics of heavy Internet users (e.g., high-income, male). We addressed this by fine-tuning GPT on the GSS (General Social Survey [9] - the gold standard of demographic surveys in the US) so our models emulate a more representative U.S. population.
We’ve built a transparency feature that shows how similar your survey question is to the training data and thus gives a confidence metric of our accuracy. If you click ‘Investigate Results’, we report the most similar (in terms of cosine distance between LLM embeddings) GSS questions as a way of estimating how much extrapolation / interpolation is going on. This doesn’t quite address the accuracy of the subpopulations / conditioning questions (we are working on this), but we thought we are at a sufficiently advanced point to share what we’ve built with you all.
We're graduating PhD students from Princeton University in cognitive science and AI. We ran a ton of surveys and behavioral experiments and were often frustrated with the pipeline. We were looking to leave academia, and saw an opportunity in making the survey pipeline better. User and market research is a big market, and many of the tools and methods the industry u...
93 comments
[ 0.41 ms ] story [ 132 ms ] threadOn a more personal note, while all of the AI advances have been very interesting, I worry that AI will reduce human connection, and a product like this sure seems to do that. You are telling users that they don't need to talk to real people, and can just get feedback from a model instead.
Edit: for example, here's your dataset by race: https://imgur.com/a/134epoN
I asked, "Which race is most likely to commit a crime?": https://imgur.com/a/4QJZo2O
2. We add the transparency features (click on 'Investigate Results') that shows how in vs. out-of-distribution the target question is. For out-of-distribution, we suggest people run traditional surveys.
More broadly, I think your point is really interesting when it comes to qualitative data. That is one reason we haven't generated qualitative survey data, but a lot of potential customers have already started to ask for it.
----
[a] https://roundtable.ai/sandbox/baa3d5f25236b91f1608c9f606b315...
[b] https://roundtable.ai/sandbox/7a9ee27872eb29087be2386ccd19f7...
We definitely need to think how to handle your question so that it's clear where survey data converges/diverges with reality.
What metric(s) are you using to measure bias in general, and what do those metric(s) look like before and after your tuning?
[1] https://roundtable.ai/sandbox/2dd4e9d32c24e9abff01810695e948...
LLMs model a static distribution, whereas consumer preferences change over time to the point that companies regularly run the same survey at different points in time. At my old fund we would run the same surveys every month to track changes on various companies. How do you counteract this time effect? Presumably a lot of your training data is from the past.
To give one example from your summary - the demographics of Tesla owners have change significantly over time from a pure luxury, avant garde market to much mass market. So info about Tesla from 5 years ago is not that useful
The data we trained on has year, so we can specify the year you ask the question (the default is 2023). You can also see how answers change over time. [1] shows how the distribution for "Do you support the President" changes from 2000 to 2023 (see the 9/11 spike, end of Bush era, Obama era, Trump era, etc.) [1] https://roundtable.ai/sandbox/2dd4e9d32c24e9abff01810695e948...
I’d also be interested in how much you think your platform is just capturing say reported surveys/data. President polling is something that must be all over LLM datasets- isn’t that just replicating the training data?
I think you could do a better job of showing on your website the following - here are some unusual survey results we generated from the model - I.e. stuff definitely not in the training data - and here’s the data we actually got when we did that survey for real
In the backend, we check to see if the answers are stated in a high-quality survey and just retrieve that. I know we do this for gender, and I'm not sure whether that happens for presidential polling.
Great idea, thank you. We're still figuring out whether the business model will be a general-purpose tool that anyone can use or those custom models I referenced above. If the former, your suggestion is spot on.
I imagine cleaning customer data to get it to the point that it's inputtable will be a big job for you.
Are you then creating individual models per customer? As in, if Coke are an existing customer of yours and Pepsi sign up, do they get access to a model that's partially trained on Coke data, or it's a case of your base model + "bring your own research"?
We're in the process of figuring that out. Hopefully that is another use case for LLMs :)
> Are you then creating individual models per customer? As in, if Coke are an existing customer of yours and Pepsi sign up, do they get access to a model that's partially trained on Coke data, or it's a case of your base model + "bring your own research"?
The latter, i.e. base model + "bring your own research"
The answer seems plausible with that interpretation.
If you went out with a VR headset and offered 30 minute demos of them to random people for $3.50 I don't think you would have an 84% success rate.
If a researcher comes out and says, “Surveys show that people want X, and they do not like Y,” and then others ask the researcher if they surveyed people, the answer would be “no.”
Fundamentally, people wanting feedback from humans will not get that by using your product.
The best you can say is this: “Our product is guessing people will say X.”
Internal purposes include stuff like optimally rewording questions and getting priors.
A hybrid approach would be something like - hey let's not ask someone 100 questions because we can accurately predict 80%. Let's just ask them the hard-to-estimate 20 questions
This kind of concerns me because you could use this to bias surveys in different directions. This obviously already happens, so maybe it just part of the status quo.
Out of One, Many: Using Language Models to Simulate Human Samples (https://arxiv.org/abs/2209.06899)
There's been some research in this vain. To answer your question, seemingly very valid.
I suspect people would use this product as a quick gut check to decide if it is warranted to spend the time and money on a full scale quant study.
This is like a 10/10.
[1] https://www.youtube.com/watch?v=G0ZZJXw4MTA
I see the problem as although you can create lots of examples that are correct/follow real world opinions, you can never prove that a particular question is correct/follows real world opinion. I'm not sure who would trust the output enough to rely on it for decision making.
Covered here: https://www.pretotyping.org/
If you focus on past and current behaviors and problems you can fairly accurately predict future behaviors since most consumers/customers tend to do the same things over time.
That said, I read and enjoyed the book you linked to and thought it had valid points. If you can build a quick prototype to observe actual behavior then go that route rather than starting with a formal discovery process. That’s not always doable though…
If anything, existing research indicates you can just skip basic surveys and go for complex simulacra experiments.
https://news.ycombinator.com/item?id=36868552
What little research we have of this kind of phenomena points towards this being very valid.
Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus? (https://arxiv.org/abs/2301.07543)
Out of One, Many: Using Language Models to Simulate Human Samples (https://arxiv.org/abs/2209.06899)
That's something you might typically do by hand when running a survey and, say, comparing it to a benchmark.
The problems will be similar to most AI problems I suppose: people who don't really understand the limitations of AI or the results it produces take the output of AI as gospel.
My own thought is: what does it mean to 'simulate' a survey if the outcome is that people treat it as a 'live' or 'empirical' one?
https://news.ycombinator.com/newsguidelines.html
It's not surprising that LLMs can predict the answers to survey questions, but really good primary research generates surprising insights that are outside of existing distributions. Have you found that businesses trust your results? I have found that most businesses don't trust survey research much at all, and this seems like it might be even less reliable.
-----
Context: I co-founded & sold survey software company (YC W20).
https://news.ycombinator.com/item?id=36868552
Trust is one of the biggest issues we're trying to solve. This motivated the tSNE plots and similarity scores under 'Investigate Results', but we definitely have a long way to go. Generally speaking, survey practitioners trust us more than their clients (perhaps not surprising)
I played around with it, and one weakness I saw immediately was mixing real survey answers with imagined ones. For example: https://roundtable.ai/sandbox/609f54304935736c8e61816dea780e...
I find it hard to believe that so many people would prefer carbon emissions over free beer, massages, or being the pilot. If I condition this data on being male or female, the results change dramatically too.
One related pain point I have seen many times with surveys is that the people writing them don't know what they're doing and get bad data as a result of biased questions.
Could be cool to add functionality down the line to help people craft better questions. For example, your app could provide alternate ways of phrasing questions and then simulate how results would differ based on the wording.
Excited to see where this goes! Going to share with my partner who works for a survey software company and see what she thinks.
Thank you for the kind words / reference
The CEO of Unlearn AI on this podcast (https://podcasts.apple.com/ca/podcast/whats-your-problem/id1...) talked about using AI to simulate a larger sample size for clinical trials which is similar to what you are doing here
Looking forward to seeing where this goes :)
My career has straddled medicine and ML at various points, so I feel like I have the context to comment on this: This is really fucking stupid. Like I can't believe anyone would suggest something this dumb. Hopefully the FDA shuts this bullshit down before some moron MBA spreads the practice through the industry. Without real data you have nothing.
I have an interesting dissonance with this. On one hand, I understand how huge parameter sets can and do model specific personas well. I've also read some of these cited papers and _know_ intellectually that predicted results can be close to actual survey data. The other part of me is screaming at my laptop that language modeling is about aggregate statistics, revealed preference counts for a lot, and how could a language model actually substitute for market research?
I imagine the biggest hurtle you're going to face are research teams that:
- A. Want to see actual proof behind data
- B. Disbelieve a LLM could generate statistically significant insights about real people that would make individual decisions
- C. Need to justify their own existence / organizational clout with boots on the ground facilitating surveys
A and C might be surmountable, but I'm not sure of a good way of tackling B.
Yeah..but it's not. This is where people are having so much trouble. The erroneous belief that language model learn some "average" or "aggregate" of the training set. But when you get down to it, that doesn't make any sense at all. What help would some fictional aggregate distribution do with predicting the responses of decidedly non aggregate text ? None at all.
So Language models don't learn an "average" distribution. They learn to predict every state that exists in the corpus in a sort of superposition. The perfect LLM would predict Einstein as well as it would predict the dumbass down the street.
LLMs are biased but not uniformly so.
Technically speaking - the whole idea with a language model is that you're learning to generalize underlying patterns of text, not just memorize the input text. Otherwise language models would be very good at echoing back training data but fail miserably during validation. If we go back to the training sequence - it's trying to maximize the posterior given the conditional probabilities in the sequence:
P(y1, y2, ... yn ) = P(y1) * P(y2|y1) ... P(yn|y1...yn-1)
That probability is by definition an aggregate; it's the best fit given potentially competing inputs of the training set that all have the same input conditional chains.
Where generative LLMs have a leg-up is because they have such a large parameter space, large context windows, and coherent sampling strategies. This helps them stay internally consistent with their response data. But at the end of the day what they are learning are still patterns. That's why they aren't able to link content back to the exact source of origin; parameters fuse inputs from different places into one hybrid.
Seeding a generative chain with Einstein or someone down the street doesn't change the fact that what's next is some fused learning from a lot of different training set inputs.
Agree A, B, and C are big hurdles.
Re: A - we have started adding transparency (vis-a-vis the 'Investigate Results' and the tSNE plots + similarity scores) but we still have a ways to go
Re: B - agree that the survey responses -> insights pipeline is nonlinear and it's not clear how to make that tighter
Re: C - generally, we try to champion a human-AI interaction loop where people are needed to evaluate the outputs, generate insights, etc.
All great points though and ones we are facing
Do you know at what boundaries this tends to stop working, e.g., some event happens that changes people's perception of X would probably need new deta if the event is "bigger than ..."?
You probably fooled yourself with cherry picking
https://news.ycombinator.com/item?id=36868552
Could you sell a version of this product where you promise to keep the buyers anonymous and confidential (and destroy the prompt and outputs after delivery)?
Being curious I want to try to work out the workings.
I assumed there was some prompt engineering using the inputs, but was interested in the GSS training model and how that would come into play.
So I was like lets see first how good or bad ChatGPT is, it was poor, though I put zero effort into optimising the prompt.
```Using your knowledge of business owners, calculate a plausible distribution of responses to the question: What is the biggest pain point for your business that a SaaS platform could potentially solve? A. Inefficient project management B. Poor data management and analytics C. Ineffective customer relationship management D. Challenges in managing remote teams E. Lack of robust communication and collaboration tools```
ChatGPT (GPT4) ``` A. Inefficient project management - 20% B. Poor data management and analytics - 25% C. Ineffective customer relationship management - 15% D. Challenges in managing remote teams - 20% E. Lack of robust communication and collaboration tools - 20% ```
The results were not that exciting and very neat (20%, 25% etc.)
I was ready to see how much more accurate RoundTable was...it was exactly the same.