Ask HN: Why is ChatGPT so bad at electronics?
I feel LLMs like GPT-4 could be a perfect tool for tedious, time consuming tasks like searching through datasheets for components with specific characteristics, but in its current form that's not the case: ChatGPT for instance can list components of quite obscure families and seem to have a quite a accurate general idea of their purposes, but when it comes to important details it just fails spectacularly every time.
Is there any company trying to improve this application of language models?
45 comments
[ 4.5 ms ] story [ 120 ms ] threadTalk to it about something you don't know about, and you'll think it's really good technology ;)
https://www.epsilontheory.com/gell-mann-amnesia/
[1] It's widely attested to him, including on Wikipedia and in an aside in an old NYTimes article [https://www.nytimes.com/1982/02/27/us/required-reading-smith...], but I couldn't track down the original source to verify it. It's possible it's falsely attributed, but on the other hand he wasn't super prominent -- editor of a major magazine for progressivism, called The Progressive, yes, but no Mark Twain -- and so there'd be little incentive to say it was his when you could call it Twain's.
I saw a spectacularly bad example of open ai trying to reason about electronics yesterday. Something like how do I use the gpio pins of my jetson and it failed so hard it was funny. That one seems simple to me. Identify that you need to look up the pinouts, find the image. Label the pins… I suspect there’s something wrong in this generation of gpts when it comes to reasoning about electronics.
We are now rolling out my tooling in the company so everyone can forget about the boring stuff and just focus on the business logic. There is resistance as this is going to cost jobs; we don't have infinite work to do and this is much (much) faster.
Been programming since I was in elementary school, and current Copilot, OpenAI and even Gemini models generate code at a very very junior level. It might solve a practical problem, but it can’t write a decent abstraction to save its life unless you repeatedly prompt it to. It also massively struggles to retain coherence when it has more moving parts; if you have different things being mutated, it often just forgets it and will write code that crashes/panics/generates UB/etc.
When you are lucky and you get something that vaguely works, the test cases it writes are of negative value. Test cases are either useless cases that don’t cover edge cases, are incorrect entirely and fail, or worse yet — look correct and pass, but are semantically wrong. LLM models have been absolutely hilariously bad at this, where it will generate passing cases for the code as written, but not for the semantics of the code being written. Writing it by hand would catch it quickly, but a junior dev using these tools can easily miss this.
Then there is Rust; most models don’t do rust well. In isolation they are kind of okay, but overall it frequently generates borrowing issues that fail to compile.
It might help but you have to be the backstop when it comes to the final call. Measuring the false positive/false negative rate could be tedious, but it's important to have a good estimate of, in order to use it wisely.
> True. 1234 microamperes (uA) is equal to 1.234 milliamperes (mA), which is sub-milliampere.
Unfortunately, that is only true until it isn't. Most datasheets are not as complete as you would like, and many are just incorrect. It's up to you to take the incomplete information and make a judgement call as an engineer. Undocumented features are not uncommon.
You are ChatGPT, a large language model trained by OpenAI, based on the GPT-4 architecture. You are tasked with answering questions and providing assistance in the domain of Electronics and Circuit Analysis. You will apply structured and concise explanations while incorporating relevant academic and technical references from uploaded materials. Your goal is to ensure clarity, accuracy, and technical correctness in your responses, especially when dealing with advanced concepts. Always follow the structured format requested by the user.
Is typing your requirements that much easier than going through traditional search filters at Digikey?
You should check it out, www.zenode.ai!
This specific failure might be s kind of averaging problem, where common answers around the general theme are preferred over more specific (and correct). LLMs can also fail completely at trivial concepts such as negation, or separating between "Y above X" and "Y below X".
It does a pretty good job at getting you to the "draw the rest of the owl" stage
I've seen it with statistics as well, asking it to implement some things in code. You'll get working but mathematically wrong code.
For this kind of task, you probably want a model that has specifically been trained on every product datasheet ever, and not ten million reddit threads and forum posts about how a 555 or 328p can solve any problem.
I doubt that chatgpt has been fed every datasheet for every part made in the last decade or two. Even if it had, that's likely far outweighed by the amout of noise coming from people talking about the most common parts.
But fundamentally I'm not sure that LLMs are great for this type of work. No two datasheets are the same and I've never seen one that wasn't missing some kind of information. What you very much do not want is an LLM hallucinating a value that does not actually exist in the datasheet. Or have it conflate two parts and mix up their values. These models just don't seem to be up to the task of returning real information from abstract queries. They're just meant to generate probabilistic text sequences.
But yeah, you hit it on the head. LLMs alone were not enough to be able to read the datasheet. It took a mix of different algorithms, including OCR, Computer Vision, Neural Nets, and yes, LLMs to be able to consistently read all the differences between different manufactures, categories, etc.
Give us a whirl at www.zenode.ai and let me know what you think, we just launched our MVP and while there's a ton to improve we think it's pretty helpful!
linguistic models are based on images of objects and phenomena, not on the physical properties of those objects and phenomena
there's practical knowledge and there's theoretical knowledge
practical knowledge can be used as technology but theoretical knowledge cannot
by and large, chat gpt is a generator of 'idle chatter' that people are used to in their communication and often in their thinking
We had to do a ton of work to get the tool to consistently return the same value (and still sometimes it shits the bed).
So yeah, even light technical jargon can confuse the heck out of an LLM, which is why that wasn't the only tool that's necessary to build this.
Try us out, would be curious to get your feedback! www.zenode.ai
can your search template be used to search for any strictly catologized things or is the template strictly tied to searching for electronic components?
__
i ask this because it occurred to me that it might be possible to make a universal cataloger of physical things and phenomena for linguistic ai, which would make it possible to... you know
__
in other words, you can create a 'real ai' from this publicly available ai linguistic model ;)
AI is the perfect tool for this task, but LLMs alone aren't capable of doing this. It took us multiple algorithms working together to read all the tables, equations, footnotes, etc. Now we've extracted the information from over 10M datasheets, and created a parts database that you can search across using natural language.
We're actually planning to hit Show Hacker News this week or next, just ironing out a few details first!
Thanks for the try though, we'll definitely be improving this dramatically over the next few weeks!