Show HN: Verify LLM Generated Code with a Spreadsheet
Mito is a spreadsheet that generates Python code as you edit it. We've spent the past three years trying to lower the startup cost to use Python for data work. In doing so, we’ve been thrust into the middle of many Python transition processes at larger enterprises, and we’ve seen up-close how non-technical folks interact with generated code.
The Mito AI chatbot lives inside of the Mito spreadsheet (https://www.trymito.io/>. The obvious benefit of this is that you can use the chatbot to transform your data and write a repeatable Pythons script. The less obvious (but equally important) benefit is that by connecting a spreadsheet and chatbot, Mito helps you understand the impact of your edits and verify LLM generated code. Every time you use the chatbot, Mito highlights the changed data in the spreadsheet. You can see a quick demo here (https://www.tella.tv/video/clibtwssv00000fl65oky13nu/view).
Three main insights shaped our approach to LLM code generation:
# Consumers of generated code don't know enough Python to verify and correct the code
Mito users span the range of Python experience. For new programmers, generating code using LLMs is an easy step one. Ensuring the generated code is correct is the forgotten step two.
In practice, LLMs often generate incorrect code, or code with unexpected side effects. A user will prompt an LLM to calculate a total_revenue column from price and quantity columns. The LLM correctly calculates total_revenue = price * quantity but then mistakenly deletes price and quantity.
New programmers find it almost impossible to verify generated code by reading it alone. They need tooling designed for their skillsets.
# Not everyone knows how to use a chat interface for transformations
We were surprised to learn that many Mito users a) had no experience with ChatGPT, and b) didn’t understand the chat interface at all! Mito AI presents users a few example prompts and an input field. A surprising number of users thought the example prompts were all they could use Mito AI for.
AI chatbots are new. Us builders might be using them for natural language interactions, but users are still learning how to use them in new contexts. This stands in stark contrast to spreadsheets, where pretty much ever business user has experience. Shout out 40 years of Excel dominance!
# The more context a prompt has about the user’s data + edits, the better the LLM results
For the LLM to generate code that can execute correctly, the prompt should include the names of the dataframes, the column headers, (some) dataframe values, and a few previous edits as examples. Duh.
But there’s no reason users should be responsible for writing this prompt. No one loves writing long chats, and in practice Mito AI users expect to be able to write ~12 words. Spreadsheets are well-suited to building the rest of the prompt for you - they have all of your data context, and know your recent edits.
With these three insights, it became very clear to us what role a spreadsheet could play in LLM based code-gen: a spreadsheet is the prompt builder, and a spreadsheet is the code verifier.
Mito AI builds an effective prompt by supplementing your input with the context of your data and recent edits.
Mito AI then helps you to verify the LLM generated code by highlighting the added, modified, and removed data within the chat interface - and within the spreadsheet. This way, you can ensure your LLM generated code is correct.
Give it a spin. Let us know what you think of the recon and how we can make it more helpful!
Also, if you like what we’re doing, we’re hiring – come help us build! ( I've personally never seen a Mito users with a detailed enough spec of their report that an LLM would be able to use it to check compliance -- but maybe if we built the functionality users would create them... It seems to me less of an issue that LLMs produce code that aren't compliant with language specs, but rather that LLMs produce code that do not meet the stated requirements, or that they break from earlier stated requirements as new requirements are requested. And of course, code not meeting requirements is not a problem native to LLMs, but to all code production agents everywhere (including humans). This nails one of the biggest issues that we're trying to help Mito users with -- code that meets __changing__ requirements. ie: the input data column headers change. The best approach we have right now is giving users flexibility to parametrize their script (ie: tell the script the right column header or reference columns by position instead of name) It turns out that teams that have moved their data to Snowflake tend to avoid these issues primarily because the schema of their data changes much less frequently. I wonder if Clojure can do it... I think the spec can check if function-calls and data given have valid types and reasonable unit/quantity each... Hmm. In general, it feels it's halting problem hard to determine if any general purpose programming language program is executable or safe... but maybe if we reduce the size of the spec under consideration? LLM/ChatGPT is based on stochastic approach to NLP while the alternative e.g. Typed Feature Grammar (TFG) is based on deterministic approach [1]. Apparently Cuelang is based on the latter and since it's a constraint based language you can use it against the prompt (input) and also for checking the output as well [2],[3]. What it means is that LLM/ChatGPT and TFG/Cuelang can be used in concert to produce or generate codes by well crafted prompt based on the language spec and similarly also to debug based on the spec. This will probably change and enhance the discipline of software engineering as we know it for the better. [1]Feature (linguistics): https://en.wikipedia.org/wiki/Feature_(linguistics) [2]The CUE Data Constraint Language: https://github.com/cue-lang/cue [3]Show HN: GPT-JSON – Structured and typehinted GPT responses in Python: 1) “What It Wants Me To Say”: Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Models (https://arxiv.org/abs/2304.06597) -- they try to tackle a similar problem as what you described above 2) On the Design of AI-powered Code Assistants for Notebooks (https://arxiv.org/abs/2301.11178) - uses Mito as part of their case study > Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Models I love the idea of giving users feedback on how to get better at prompting the LLM. I think the key to using this approach within Mito is giving users guidance at the right time -- sometimes shorter prompts get the job done, and they're always easier to write :) A really sweet integration of this approach could be: when the LLM generated code errors or when we notice that the user undoes their previous prompt, we offer the user help in converting non-working prompts into ones that follow best practices of breaking complex tasks down into small steps. > On the Design of AI-powered Code Assistants for Notebooks - uses Mito as part of their case study Andrew McNutt, one of the authors presented this paper here: https://www.youtube.com/watch?v=g0prh8mE3bI Their different classifications of notebook code-gen tools has actually been super helpful in my own thinking. Thanks for the help, Andrew if you're a HNer14 comments
[ 3.7 ms ] story [ 43.3 ms ] thread