Ask HN: What do embedded engineers do?
I would very much like to transition to doing more embedded type software that is closer to interfacing directly with hardware, but my only experience in this has been writing code for Arduino, different Raspberry Pi flavors, and sometimes the ESP8266.
I really enjoy doing these kinds of projects but through all the effort of making it as easy as possible for people to write, compile, and use code on these embedded chips I've never gotten the sense that people would be willing to pay me to do similar type of work.
I'm not under the impression that writing firmware or doing lower level programming is all like this, so I would like to hear what kinds of projects and challenges people who may have actually been employed doing work like this have come across. Was it interesting work? Is this type of software engineering in demand?
36 comments
[ 2.9 ms ] story [ 130 ms ] threadIt was interesting work, and sometimes I miss it. There were immovable resource constraints which required careful thinking, and that felt like doing real engineering in a way that everyday software development usually doesn't. Debugging could be a significant challenge. Sometimes you can plug in a JTAG header and use a source-level debugger, or you can print to a serial port, but sometimes you have to get creative with spare I/O pins and and LEDs or an oscilloscope. I had opportunities to feel proud of my cleverness. One has to think carefully about time and power use and sleep modes and interrupts; your world is a state machine. It can be delightful to have complete control; often there's no operating system at all, except to whatever degree you end up writing one in the course of your project.
On the down side, the work came to feel repetitive. Here's a project, here's a dev board; bring up the devices, write the application, test it, ship it, repeat. Also, the firmware is kind of an afterthought compared to the mechanical & electrical engineering. You live on the hardware's timetable, which is slow and frustrating when you are used to just recompiling & running tests at the drop of a hat. Also slow and frustrating were the weird proprietary IDEs and generally old-fashioned engineering practices (though this was over a decade ago, and perhaps things have improved since).
If by "in demand" you mean "high-paying", I'm afraid not; there's a significant pay gap. You can make a comfortable living, but it's not a path to wealth like the tech bigcos offer.
These comments leave me confused. My Michigan based understanding would have you both poorly paid for embedded work and obscenely paid for backend work. I know some folks in CA make 200K and up, I'm just not sure how common that is. And anyone doing embedded should be at least 100K, so I don't see an easy x4.
I am a pure embedded sw. guy (3 YOE) and have already started to learn DB management and higher-level languages (Java and C#). How was your transition?
Pulling together working hardware, packaging, and logistics is non-trivial, often involves multiple people, many moving pieces, extended chunks of time, and heroic efforts to resolve the inevitable problems. When the hardware comes together, it is only after a valiant team effort and at some cost. Everyone knows how hard it was. By comparison, firmware is invisible, usually written by fewer (or one) people, and usually carries lower risk. If your firmware has a bug two weeks before shipping, it's a recompile and we're back on track. If your hardware has a bug two weeks before shipping, it might be the end of the organization.
With hardware, it's the physical 'thing' that is the product. With applications, it's the software itself that is the valuable thing.
It's gotten better over the years, but there are still huge misunderstandings and misvaluations of the different work. Obviously a hardware system needs hardware engineers, but nearly every hardware system these days involves software systems that do the actual work of coordination, communication, and control. It can't be treated as an afterthought but it often is.
Prima donnas? What a twit.
I wrote nothing about having software "calling the shots" (I assume that's who you mean as "prima donnas"), but when software is essential to your system, you should have that team at the table in your planning and budgeting efforts. If not, you'll get poor quality because it's low priority. And you'll get poor quality if you reliably underpay people, the good ones leave.
I like to tell them that software should be on the BoM even though the marginal cost in $0.00. Having it noted as a component helps get people on board with proper versioning, and also raises visibility for something otherwise unseen.
From my perspective a lot of that is because the tooling and mindset for hardware is stuck in 1980s/90s. Compared to modern software development, there is very little to no automation in hardware design.
Things like CI for hardware designs and even automated version control are just starting to become a thing in the embedded space. Automation like building Bill of Materials, build pdfs, gerbers are not in any way optimized.
Scaling embedded stuff (design, manufacture, etc.) is very much a linear thing today, which is why salaries are low.
On the electronics side, better modules like RPi's compute module, open source footprint libraries, and reference designs have significantly sped up development since I started twenty years ago. What used to take weeks or months in Altium can now be measured in days, especially if companies publish their reference designs in the application format instead of PDF. The turn around time and cost on PCBs have also dropped precipitously, to speak nothing of the turn key assembly that the fabs offer. Pretty much the only thing that has stayed the same is that MetCal induction soldering irons are still the best.
On the mechanics side, it's hard to appreciate just how much better vendors' CAD libraries have become, to the point where you can drag and drop Misumi/Mcmaster/etc parts from a Solidworks extension onto your assembly like it was Gary's Mod. 3D printing alone has made everything more efficient during development and has enabled many interesting production designs from rocket nozzles to turbojet engines.
Almost all of my ex-colleagues went to web dev/fin tech :(
I disagree. At my workplace(regulated industry), Embedded software engineers are paid 2x-5x of other engineering groups. Infact, some of our embedded engineers have so much leverage that they have negotiated 3-4 day work-weeks. Second highest paid group are DevOps(though they don't actually do much for us, as there is separate IT department). And our generic software engineers for web and custom-devices also make decent salary. What you mention could be a US centric thing.
Just to make some sense, our embedded software engineers not only write good code, they also maintain their own CI/CD, extensive test coverage, harsh code reviews, very well maintained codebase(unlike on other places I have worked where arcane header files are thrown around and used, no questions asked), automated deployment and update system, A/B testing, rollback and other interesting things typically practiced by higher level(web/desktop/mobile etc.) engineers.
I have also worked at a big name AG where they make specialized hardware, the engineers there also made very decent pay but quality of software and development practices were more like script kiddies copy pasting things and calling it a day if the emulator showed correct operation.
on the good side, it's less competitive comparing to apps or web etc, it's more specialized, so you can still find jobs in your 60s if you want to.
and yes you're supposed to know both software and hardware well.
You can pick up work like that on upwork or fiverr, but you have to have a bit of business sense to know which projects are feasible and practical.
Combined with another specialty, it can be challenging and lucrative. High-speed/FPGA specialists for example earn more, as do instrumentation electronics engineers (physics/signal processing) and optical electronics engineers.
People used to mention medical devices as another specialization but I think nowadays the embedded design side of things is not the “hard part” and so it’s less lucrative.
As a generalist I find the work extremely satisfying. You end up covering a wide array of subjects from assembly-level optimization up to designing communication protocols. You also have to be quite flexible as a programmer. While all embedded processors speak C, each has its own subtly different quirks. You have to be aware of where your framework's abstractions end and where you need to start manually poking bits into magic registers.
As others have said, the pay is not great. Despite being at a startup, I'm still being paid at about average market rate. It's good enough for my lifestyle.
But practically I'm not sure if one could be a great embedded programmer without also being an EE. The code is very tightly coupled to the physical hardware in ways that you don't expect, and there's really nothing in normal software development that prepares you for that. I mean, I'm sure some people do it, but I can't imagine how they could be at all efficient.
Do it if you think it's satisfying work, but do understand that your code is quite often butting up against physics. You sometimes have to code around the physical and electromagnetic properties of your widget, and it is really not at all straightforward if you haven't experienced it before.
Alternatively, you could just specialize in "Linux, but on a single board computer". Which is technically embedded, but not in the same class as what I'm describing here.
The other thing to note is if trying to spin out, finding VCs to fund a hardware startup is also tricky since the time horizon is much longer than a software startup. That being said, I've worked at a hardware startup before and if you have a compelling enough story, you can raise money without needing an initial prototype. In fact, I would recommend this since I also know of a lot of engineers who've spent a lot of time building something first and then there is no market/funding for it.
Now getting back to me not recommending this route, especially on the PhD level. Stable jobs for this line of work are hard to come by and are often underpaid for reasons I don't entirely understand besides that pure software deployments can reach larger audiences in a shorter amount of time which equals more ROI. The space is a bit fragmented and the level of skill varies. Many undergrads aren't really taught embedded-c either and often gravitate to higher level languages which are more domain flexible. Unless you land a job at a tech company building rapid internal prototypes which pays fairly well, the more common route I see in this space is for highly skilled people working contract jobs which can be too unstable for some. If you really wanted to go this route, I would recommend coupling embedded engineering with another domain (e.g., edge machine learning, technology applied for conservation/ecology). I did this about 2 years ago in my PhD and I can say that it has brought me significantly more attention and interest from funders and potential employers.
I've been in and out of embedded for over 25 years now. Mostly in. Some things I've worked on over the years: Battery monitoring for a Ford EV back in 1997, control software for the Ibot (Dean Kamen's stair climbing wheelchair), a few small prototype gizmos, electric power steering systems (due to my early motor control experience on the Ibot), engine controller test code, airbag controllers (very boring, it's mostly endless diagnostics), ac/dc converters mostly for EV charging but also for other uses (up to 900KW), Electric water pumps, and something really fun that I can't tell you about yet. I've focused on motors because I accidentally became really good at Field Oriented Control and having that specialization pays well while being applicable to a bunch of different things - different products, but also the power conversion stuff was very closely related.
That's just the embedded side. I've bounced back to PC apps a few times in completely unrelated areas. It probably helps to be an engineer first, and software person second. This avoids the notion of being a "coder" and gets me involved more at product level design and development.
I know a guy who has done a lot of embedded audio DSP work. He's an older guy too and just had a few months out of work. It was fairly easy for him to find a job though and he's happy doing something new. Audio is an area where embedded and AI are actually coming together. If that's your thing, try diving into speech recognition on the Pi or something and then scale DOWN to smaller hardware. IMHO on-device voice control is going to become a mildly big deal in certain areas.
Be flexible and always try to work on interesting things!
As others say, many companies in the embedded space have had a very hard time realizing they are software companies and their practices are very old school and frustrating.
Talking salaries (Sweden), yeah it’s a bit higher in the cloud but not wildly so.
My recommendation is to start working in a not tiny company and on an existing product. Then it’s more about adding logic rather than knowing everything about RTOS and bootloaders. Them you will pick these things up as you go.
Battery Management Systems: Turn on and off 324 single cells in a 200 kWh array, measure their voltages and temperatures and ensure they're cells are not engaged above threshold temperatures (asymmetric for charging and discharging), or below threshold voltages.
Electric Vehicle Chargers: Communicate with car, communicate with charge point operator.
LoRa transceivers: Configure transceiver, send and receive encrypted messages over LoRa.
It was all Rust, and all very interesting.
But as other commenters have pointed out, there is a pay loft compared to other industries.