I’m not fond of the provocative title because prompting is easy and only getting easier; the advice seems to be predicated on the use of relatively deficient LLMs. I don’t doubt there will still be operator skill involved, but I anticipate the state of the art for LLMs ability to adapt to “bad” prompts will outpace our ability to learn to prompt them effectively.
Disclaimer: I watched the video but didn’t read the paper.
I think there are a lot of instances where writing prompts can be hard just because it’s hard to express your needs in words sometimes. Bad prompts are often ambiguous and there is only so much even a perfect LLM can correct for. That is, until we have direct connections to our brains.
I think you're right about prompts getting "easier" but I don't think it's a good thing. I expect it will evolve like google search. Where initially there are ways to increase specificity, or at least introduce enough randomness to get some different results, it will converge to something that ignores most of what you prompt and gives you what OpenAI wants you to see. That's really the only way adapting to "bad" prompts even could work
LLMs will definitely get better -- and people will adapt too. But natural language contains inherent ambiguity, and that's not going to change.
A large part of the paper talks about how the challenges of programming and developing ML systems don't go away just because you're using natural language and an LLM.
For example, users incorrectly extrapolating from a single failed/successful prompt to other contexts: that's not something that will be solved by better LLMs, really. It's solved by getting users to recognize when they do/don't have enough data to believe what they're seeing.
Paper is great, too. What's it like trying to write a paper while the field is shifting so rapidly? Is there anything you would have liked to include that had to be cut so you could get it out the door?
Oh, that's not a full TL;DR -- but sounds like you've read the paper. :)
Shifting field: yes, we spend a lot of time trying to figure out what is going to change and what isn't, at least in HCI (my field). LLM capabilities are changing by the...day? So anything that's very focused on LLM capabilities won't be relevant for long.
Humans don't change quite as fast!
Keep in mind this paper was first written in September 2022, reflecting work done prior to then...practically an eternity in which we've since seen ChatGPT take the world by storm, and now GPT-4. But I think the lessons there are still very relevant, and will be for quite some time.
Note that CHI is not a programming language or software engineering conference, but a conference in human-computer interaction: it's the ACM Conference on Human Factors in Computing Systems.
Well CHI is being held in Europe this year, so apparently you don't!
But more seriously, the field is "HCI" everywhere, including in America, and has been for at least thirty years. I have vague memories of hearing the story about why the initial ACM SIGCHI folks didn't go with HCI at the time but I can't recall. Anyway, it wasn't long after CHI was founded that basically everyone was using "HCI" on both sides of the Atlantic.
Inertia mostly. It's a question every grad student asks at some point.
If anyone is actually curious: The Association for Computing Machinery (ACM) started a Special Interest Group on Computer-Human Interaction (SIGCHI.org). Later, the Association for Information Systems (AIS) started a Special Interest Group on Human-Computer Interaction (SIGHCI.org) which has remained about an order of magnitude smaller in impact and comes from a different originating field (Information Systems). My impression is that nearly all the papers in one journal would also make sense in the other, but CHI sometimes has a little more design-y or consumer product stuff while IS-HCI has some stuff that's more from the perspective of an IT department.
The CHI folks can't unilaterally rename to SIGHCI without stomping over the smaller org, which means merging two different organizations, journals, and conferences run by different professional associations. Such a thing is occasionally proposed (e.g. http://www.bridginghci.org/wp-content/uploads/2018/03/chi-20...) which would be nice but in reality I think that's unlikely to happen because the cost exceeds the benefits.
That's a point Ben Shneiderman insisted on making in 1983, when he named his "Human Computer Interaction Lab" at the University of Maryland HCIL instead of CHIL, where I worked with him from 1986-1990. Go Terps! ;)
Do you have any citations of European or other labs using that human-first convention before 1983?
University of Maryland Human–Computer Interaction Lab:
Ben also coined the term "Direct Manipulation", and came up with the design of making links blue for the early HyperTIES hypermedia system we developed at HCIL.
30 YEARS AT THE UNIVERSITY OF MARYLAND’S HUMAN-COMPUTER INTERACTION LAB (HCIL). (2013) By Ben Shneiderman, Kent Norman, Catherine Plaisant, Benjamin Bederson, Allison Druin, Jennifer Golbeck:
>One attraction of the University of Maryland was its strong psychology department. My computing colleagues were intrigued by my early attempts to use empirical techniques to study programmers as they wrote, modified, or debugged programs. These crossover ideas caught the attention of Azriel Rosenfeld (1931–2004), a world leader in computer vision, who was forming an interdisciplinary Center for Automation Research (CfAR). He led the Computer Vision Lab and invited me to form a Human-Computer Interaction Lab when he launched CfAR in 1983. In a campus reorganization, HCIL became a unit in the Institute for Advanced Computer Studies and eventually became jointly managed with the iSchool.
>Rosenfeld's invitation advanced my efforts by at least five years, giving credibility to the "marriage of computer science and psychology," which I described in my 1980 book, Software Psychology. Gaining credibility was important, as this was still a time when many computer scientists were unsure about the value of psychological studies of programmers and database systems' users, and even the growing field of interactive computer systems. The term human-computer interaction (HCI) was still novel, but I insisted on putting the human first, as opposed to the ACM's choice of computer-human interaction to make a more pronounceable name, "CHI."
Sadly I think this is a significant part of it. It is so very hard to convince anyone in CS to fund unsexy projects. I think the majority of innovation on the unsexy things happens internally at the large tech companies.
Or maybe their chasing it because it's a highly relevant topic that might impact lots of people around the world, you know, a kind of human-computer interaction.
As someone in the field, trust me when I say there's going to be a big wave of HCI research that will pretty much just be "What can we glue ChatGPT to?"
Oh man, wild to see an article about the biggest conference in my field pop up on HN.
It's surprising how quickly HCI people managed to pivot to AI stuff - the paper deadline for this conference was Sept. 15, 2022, which was about a month before ChatGPT was even released. So... expect to see even more AI at next year's conference in Honolulu!
36 comments
[ 2.0 ms ] story [ 75.1 ms ] threadDisclaimer: I watched the video but didn’t read the paper.
A large part of the paper talks about how the challenges of programming and developing ML systems don't go away just because you're using natural language and an LLM.
For example, users incorrectly extrapolating from a single failed/successful prompt to other contexts: that's not something that will be solved by better LLMs, really. It's solved by getting users to recognize when they do/don't have enough data to believe what they're seeing.
Disclaimer: I wrote the paper.
... And the laughs.
Paper is great, too. What's it like trying to write a paper while the field is shifting so rapidly? Is there anything you would have liked to include that had to be cut so you could get it out the door?
Shifting field: yes, we spend a lot of time trying to figure out what is going to change and what isn't, at least in HCI (my field). LLM capabilities are changing by the...day? So anything that's very focused on LLM capabilities won't be relevant for long.
Humans don't change quite as fast!
Keep in mind this paper was first written in September 2022, reflecting work done prior to then...practically an eternity in which we've since seen ChatGPT take the world by storm, and now GPT-4. But I think the lessons there are still very relevant, and will be for quite some time.
Well CHI is being held in Europe this year, so apparently you don't!
But more seriously, the field is "HCI" everywhere, including in America, and has been for at least thirty years. I have vague memories of hearing the story about why the initial ACM SIGCHI folks didn't go with HCI at the time but I can't recall. Anyway, it wasn't long after CHI was founded that basically everyone was using "HCI" on both sides of the Atlantic.
Also it isn't "computer/human interaction." It is "conference for human and computer interaction."
If anyone is actually curious: The Association for Computing Machinery (ACM) started a Special Interest Group on Computer-Human Interaction (SIGCHI.org). Later, the Association for Information Systems (AIS) started a Special Interest Group on Human-Computer Interaction (SIGHCI.org) which has remained about an order of magnitude smaller in impact and comes from a different originating field (Information Systems). My impression is that nearly all the papers in one journal would also make sense in the other, but CHI sometimes has a little more design-y or consumer product stuff while IS-HCI has some stuff that's more from the perspective of an IT department.
The CHI folks can't unilaterally rename to SIGHCI without stomping over the smaller org, which means merging two different organizations, journals, and conferences run by different professional associations. Such a thing is occasionally proposed (e.g. http://www.bridginghci.org/wp-content/uploads/2018/03/chi-20...) which would be nice but in reality I think that's unlikely to happen because the cost exceeds the benefits.
Do you have any citations of European or other labs using that human-first convention before 1983?
University of Maryland Human–Computer Interaction Lab:
https://en.wikipedia.org/wiki/University_of_Maryland_Human%E...
Ben Shneiderman:
https://en.wikipedia.org/wiki/Ben_Shneiderman
Ben also coined the term "Direct Manipulation", and came up with the design of making links blue for the early HyperTIES hypermedia system we developed at HCIL.
https://en.wikipedia.org/wiki/Direct_manipulation_interface
Revisiting why hyperlinks are blue (blog.mozilla.org):
https://blog.mozilla.org/en/internet-culture/why-are-hyperli...
https://news.ycombinator.com/item?id=29897811
https://news.ycombinator.com/item?id=29921532
>Ben Shneiderman recalled that "Tim told me at the time that he was influenced by our design as he saw it in the Hypertext on Hypertext project".
Hypertext on Hypertext CACM1988:
https://www.youtube.com/watch?v=29b4O2xxeqg
30 YEARS AT THE UNIVERSITY OF MARYLAND’S HUMAN-COMPUTER INTERACTION LAB (HCIL). (2013) By Ben Shneiderman, Kent Norman, Catherine Plaisant, Benjamin Bederson, Allison Druin, Jennifer Golbeck:
http://interactionsdev.acm.org/archive/view/september-octobe...
>One attraction of the University of Maryland was its strong psychology department. My computing colleagues were intrigued by my early attempts to use empirical techniques to study programmers as they wrote, modified, or debugged programs. These crossover ideas caught the attention of Azriel Rosenfeld (1931–2004), a world leader in computer vision, who was forming an interdisciplinary Center for Automation Research (CfAR). He led the Computer Vision Lab and invited me to form a Human-Computer Interaction Lab when he launched CfAR in 1983. In a campus reorganization, HCIL became a unit in the Institute for Advanced Computer Studies and eventually became jointly managed with the iSchool.
>Rosenfeld's invitation advanced my efforts by at least five years, giving credibility to the "marriage of computer science and psychology," which I described in my 1980 book, Software Psychology. Gaining credibility was important, as this was still a time when many computer scientists were unsure about the value of psychological studies of programmers and database systems' users, and even the growing field of interactive computer systems. The term human-computer interaction (HCI) was still novel, but I insisted on putting the human first, as opposed to the ACM's choice of computer-human interaction to make a more pronounceable name, "CHI."
Ken Perlin:
https://en.wiki...
It's surprising how quickly HCI people managed to pivot to AI stuff - the paper deadline for this conference was Sept. 15, 2022, which was about a month before ChatGPT was even released. So... expect to see even more AI at next year's conference in Honolulu!
Do any of these studies give a map of programming tasks that are enabled by LLM's vs tasks that don't benefit?