It's not the best story - I'm sure there are some greats here - but I tricked GitLab into running scripts that looked like https://gitlab.com/jyavorska/c64exec/-/blob/master/.gitlab-c... by modifying a runner to pass everything through the VICE Commodore BASIC emulator. It would even attach the output file as an artifact to the build.
I used SQLite for coordination between processes. It was a huge Python application that originally used the multiprocessing library and had to be migrated to Rust.
In hindsight, it would have been better to use a local HTTP server. Seemed like overkill at the time.
What I meant is that doesn't make sense for the Python version because multiprocessing has its own mechanisms for doing that. Same for the post-migration Rust binary.
I had an old boiler that would sometimes trip and lock out the heat until someone went down and power cycled it. (It was its own monstrous hack of a gas burner fitted to a 1950s oil boiler and I think a flame proving sensor was bad.)
Every time it happened, it made for a long heat up cycle to warm the water and rads and eventually the house.
So I built an Arduino controlled NC relay that removed power for 1 minute out of every 120. That was often enough to eliminate the effect of the fault, but not so often that I had concerns about filling too much gas if the boiler ever failed to ignite. 12 failed ignitions per day wouldn’t give a build up to be worried about.
That ~20 lines of code kept it working for several years until the boiler was replaced.
Our boiler has a pump to cycle hot water around the house - this makes it so you get warm water right away when you turn on a faucet and also prevents pipes in exterior walls from freezing in the winter.
This stopped working, the pump is fine but the boiler was no longer triggering it.
I just wired up mains through an esp32 relay board to the pump and configured a regular timer via esphome.
Temperature based logic would be even better but I didn't find a good way to measure pipe temperature yet.
IIRC, I used an RTD that I had left over from a 3D printer upgrade, but an 18B20 would fine as well. A 10K NTC resistor might even be good enough. For what I needed (and I think for what you need), just fixing the sensor to the outside of the pipe [if metal] will give you a usable signal. That sensor was just metal HVAC taped to the front cast iron door of the burner chamber.
But a dead-simple timer solution gets you pretty far as you know.
I have a mix of shame and pride that the chart (everything in the rectangle) is entirely hand-coded SVG elements emitted by the ESP web request handler.
I'm thiiiiiiis close to installing a circulating pump. I plan to power it off the bathroom lightswitch, which I might just replace with a motion sensor.
`envsubst` on a k8s manifest, for templating. The space for templating/dynamic k8s manifests is complex, needlessly so I felt. But this... just works. It has been running in CI for a couple months now, deploying to prod. I'm sure the day that breaks down will come, but it's not here yet.
I’ve been working on the same thing for a few months now.
Not only is it more customizable / less complicated than helm / other solutions, but GNU gettext is almost 30?! years old at this point, and environment variables are probably realistically double that age. They aint going anywhere anytime soon.
Plus I feel that more complex logic removes value from the configs we are building, and so am not interested in many other tools.
I worked at a startup where the core backend was 1 giant serverless Function. For error handling and recovery, the Function ran in a while loop.
For all its faults, it worked and it was generating revenue. Enough that the startup got to a sizable Series A. That experience completely changed how I think about writing code at startups.
A customer of mine wraps their Python code into try except pass so it never stops because of errors, it just skips what would have run in the code after the exception. I added some logging so we're slowly understanding what fails and why.
this is great. if you look at older game source code you find things like this. things that we view as horrible hacks which are both extremely stable and perform well.
i see no reason to stop using these types of solutions, when appropriate.
Old games also didn't use a database, they saved everything in a giant text file.
I'm not sure if they were "extremely stable" though. Like Myspace, it might only work up until a certain point. What kills stuff is usually going viral.
Suppressing errors and continuing to run in a corrupt state is good for games (crashing is definitely not fun, continuing to run might be fun, unless it's the kind of game where you can get yourself locked out of winning long ahead of time) but not good for most other kinds of code - it's better for your accounting code to crash than to save the wrong numbers!
I made my own version of AWS workspaces inside AWS because workspaces is a buggy piece of shit and the client sucks. It's just an EC2 instance which can be started and stopped by a makefile that runs awscli and I query the IP address and open it in MS RDP!
At my company we used AWS workspaces for training classes. Whenever we'd have a class, we'd create 1 or 2 spare workspaces. If someone couldn't connect to their workspace, we'd give them the connection information for one of the spares.
I was so happy when we reworked our classes to no longer use AWS workspaces.
I use a table object and a OnAfterModifyRecord trigger to process OData calls in Navision 2018 (ERP System). For some reason I can not call actions manually so I write whatever I want to do into a table and process accordingly with triggers.
In the text file you have something you want to template (or "parametrize") from an outside variable, so you name that something like @@VAR@@ and then you can sed that @@VAR@@ :-)
That's how we do it. Not with sed exactly, but string replacement. One is a bull email sender that only supports VBScript, the other is C#, but the users aren't supposed (or need) to have full templating powers, so this way it's easier.
I think the m4 macro processor might be the more canonical way to do it. Unless you’re writing a shell script, in which case a “here-document” with embedded $VARIABLES is more straightforward.
Generated HTML email newsletters from Excel (in 2004).
It was a big old-fashioned bookseller trying to compete with Amazon. Software and the web was locked down tight, but they opened a daily report in Excel, and I built a VBA macro that generated the necessary HTML and published the images to an FTP server. Turned a 2 day job into a 10 minute one.
I had a GCP Cloud Run function that rendered videos. It was fine for one video per request but after that it slowed to a crawl and needed to shut down to clear out whatever was wrong. I assume a memory leak in MoviePy? Spent a couple of days looking at multiple options and trying different things, in the end I just duplicated the service so I had three of them and rotated which one we sent video renders to and do each render one at a time. It was by far the cheapest solution, means we processed them in parallel rather than serially so it was faster, all in all it worked a treat.
This reminds me of a service I recently found that was routinely crashing out and being restarted automatically. I fixed the crash, but it turns out it had ALWAYS been crashing on a reliable schedule - and keeping the service alive longer created a plethora of other issues, memory leaks being just one of them.
That was a structural crash and I should not have addressed it.
At Fastmail the ops team we ran fail overs all the time just to get our failures so reliable they worked no matter what. Only once in my tenure did a fail over fail and in that case there was a --yolo flag
At reddit we would randomly select a process to kill every 10 minutes out of the 10 or so on each machine, just so they would all get a restart in case we didn't do a deployment for a few days.
At Amazon they schedule service bounces during code freeze for any service that is known to have memory leaks because it's easier than finding the leak, which isn't usually an issue since it gets deployed so often.
Oooh, you’ve just reminded me of the email server at my first dev job. It would crash every few days and no one could work out why. In the end someone just wrote a cron job type thing to restart it it once a day, problem solved!
Hah welcome to cloudrun! I was evaluating it a few years ago to host some internal corporate app.
It worked great and was way easier to deploy than k8s setups. However after some testing we found out that the core logic of the app - a long running process, would just crawl to a halt after some time.
It turned out google wanted to push you to use their (paid) queue / pubsub solutions, but they didn’t want it to be _too_ obvious, so cloudrun would actually throttle its cpu sometime after the request it was spawned was returned.
Our logic was based on pushing stuff in a queue and having it be processed outside of a request, but google just f*ked with that solution.
And it would have been fine if that was upfront info, but it was buried in a doc page somewhere obscure, small print…
that was the whole point of cloud run, which was obvious when you looked at the pricing for it: If an instance is not processing requests, the CPU is not allocated and you're not charged.
I was part of a team which had to make web interactives for an old desktop-only Java-based CMS which published out to HTML. This was back before cross-publishing to formats like Apple News was important; we only had to worry about things working on the browser version.
The CMS didn't support any kind of HTML/JS embed, and had quite a short character limit per "Text" block. But luckily, that block didn't filter out _all_ inline HTML - only some characters.
So, a bootstrap "Loading" element was inserted, along with a script tag which would bring in the rest of the resources and insert those in to the page where the bootstrap was placed. This quickly became a versatile, re-usable loader, and allowed us to launch the features. But all this grew from a very inelegant hack which just happened to work.
Reminds me when we started implementing features as an Oracle trigger. It was meant to be “just a trigger” but then there are so many edge cases when you do an end—run around application code that it took a couple of week total. Boss was like “couple of weeks for a trigger!”
Monkey patching vendor code. They agreed their code didn’t work and produced wrong results, but the correct version would be slower, so they didn’t want to change it.
So I dynamically replaced the part of their code that was wrong. That monkey patch has run years and is still going :)
The company I was working for purchased some kind of email spamming program to send mass emails. However they quickly found that when more than a small number (10 to 15,000?) of addresses were in the list it took forever.
Support wasn’t being helpful, so I decided to look at it. It was some kind of PHP application and what I eventually found was it was doing a database query doing table scans for everything because there was a missing index that was incredibly obvious.
I added the index, the program worked well into huge numbers of emails addresses, and we informed the vendor.
The program was soon updated and it fixed the problem as well. Also all the PHP code was now obfuscated/encrypted.
As a 12 year old: I tried to overclock my first "good" own computer (AMD Duron 1200 MHz). System wouldn't start at 1600 MHz and I didn't know BIOS reset exists.
I ended up putting the computer in the freezer and let it cool down for an hour. I placed the CRT display on top and the power/VGA keyboard cable going into the freezer. I managed to set it back to the original frequency before it died.
When I was a teenager my friend would throw his laptop into the freezer for a few minutes every hour when we were playing games. He probably threw it in there hundreds of times, and it worked fine for years.
I don't know why but this reminds me of how we picture-framed my friend's old Wifi chip after replacing it, because that chip failing all the time was basically the core feature of our group's gaming sessions.
You guys are really smart. When I was a kid I had a graphics card that would overheat and crash the computer when I played Lineage. So I would get down under my desk and blow on it...
Once my phone died from a cracked solder joint. I had cold veggie sausages in the hotel room fridge. Holding my phone against the sausages let me grab a couple more files off of it. Saved my OTP keys that way (I've fixed my backups now :) )
I used a WiFi smart switch and a USB thermometer to make a sous vide cooker. I plugged a slow cooker into the smart socket, put the thermometer in it, and wrote a program to turn the switch on/off depending on the temperature the thermometer registered.
A friend of mine brute-forced the access key on a particular piece of industrial hardware by using an X10 outlet relay module, predecessor to today's wifi-connected smart sockets. The key was only 4 digits long and it would lock out after 3 attempts and require a reboot, so he was looking at a few thousand reboots each taking a minute or so. Easy enough to run over a weekend.
It ended up wearing out one relay, but he had a spare and just swapped it in and restarted the script from where it hung up. For the cost of like $30 and a few days, it gave us a new unobtanium unlocked unit.
In a previous role, I automated an unholy amount of business processes by adding doGet() / doPost() handlers to expose google sheets as basic web services. It's a bit slow for large sheets, but was quite nice to work with and troubleshoot, and the built-in history in google sheets allowed me to experiment with little risk of data loss/corruption.
Define works? Ive seen stupid and not working but convinced it's working until proven otherwise...
I use to work part time restoring Fiat, Porsche, and VW rares for an old head out in the mid west, lots of "stupid but works" in those old
cars... Mercedes Benz once (1980s or so) employed glass containers to solve for fuel pressure problems. Insane coolant loop designs or early Fuel Injection systems that develop "Ghosts" lol...
I've implemented recently just the same thing, but for SVG -> PNG conversion. I found that SVG rendering support is crap in every conversion tool and library I've tried. Apparently even Chrome has some basic features missing, when doing text on path for example. So far Selenium + headless Firefox performs the best ¯\_(ツ)_/¯
I wrote a Python package [1] that does something similar! It allows the generation of images from HTML+CSS strings or files (or even other files like SVGs) and could probably handle PDF generation too.
It uses the headless version of Chrome/Chromium or Edge behind the scenes.
Writing this package made me realize that even big projects (such as Chromium) sometimes have features that just don't work. Edge headless wouldn't let you take screenshots up until recently, and I still encountered issues with Firefox last time I tried to add support for it in the package. I also stumbled upon weird behaviors of Chrome CDP when trying to implement an alternative to using the headless mode, and these issues eventually fixed themselves after some Chrome updates.
Yeah it's the same concept, instead of .screenshot you do .pdf in pupetteer.
But with pdfs the money is on getting those headers and footers consistent and on every page, so you do need some handcrafted html and print styling for that (hint: the answer is tables).
I've seen a bit of SaaS and legacy websites-with-invoice-system doing that, with e.g. wkhtmltopdf. It isn't a lightweight solution, but it's a good hammer for a strange nail, a lot of off-the-shelf report systems suck.
I did the same. We had a tool that would let you export to pdf. That pdf would be sent to our customers. Initially we just used the print functionality in the users browser, but that caused output to vary based on the browser/os used.
People complained that the PDFs generated were slightly different. So instead I had the client send over the entire html in a post request and open it up in a headless chrome with --print-to-pdf and then sent it back to the client.
I mean browsers are built for and the best at displaying html+css. Given that they are "living standards", very few other programs can hope to keep up.
I had Chromium component added to a project just to show the users the help file which was a giant PDF document. The PDF file was from a 3rd part vendor who didn't know better/refused to change the system so we had to show it "as is" to the users. Any PDF reader component we tried failed because the PDF file had some crappy features in it that none of those component knew how to parse. Chromium engine, for its hate that gets nowadays, had no problem with any of those PDF files.
A company I worked for had a website where you could order mobile phones and subscriptions from different providers. This was just a frontend, and behind the scenes, they just ordered them directly from those providers. But those providers had terrible sites still written for IE6 (this was in 2010 I think). And yet those sites where all they had (for some reason; I don't know the full background).
So what happened is: the customer would order their phone subscription on the front end, that would create a job file that would be sent to a scheduler that managed 10 Windows VMs that used a Ruby Watir script to direct IE6 to fill in the data from the job file on the old decrepit website.
It's the most horrific hack that I ever touched (I forgot exactly, but I had to make some adjustments to the system), but it worked perfectly for a couple of years until those providers finally updated their websites.
I ended up swapping it out to a generic in-line CPAP humidifier, but at the same time, realized I could partially automate the process of refilling the chamber (and not have to keep unhooking hoses) by adding an in-line oxygen tee, some aquarium plumbing, a check valve, and a 12 volt pump and switch.
In the morning I just hold a button and the tank magically refills itself ;)
(Also: The Dreamstation, while recalled, was personally de-foamed and repaired myself -- I don't trust Philips any further than I can throw them now. I now self-service my gear.)
My mom's place (about 100 miles from me) has a water heater that's of an age where it could fail, so I put together a Pico W and a water sensor. I had it notify me daily just to make sure it was still working. And for reasons unknown, every 8 days it would stop notifying. A reboot would resolve it. We tried logging errors and having it report upon reboot but I wasn't versed enough with Pi to figure out anything more than it being an HTTP POST error. So I changed the code so when it got to that error instead of logging it would just reboot itself, and all has been smooth since.
508 comments
[ 4.0 ms ] story [ 351 ms ] threadIn hindsight, it would have been better to use a local HTTP server. Seemed like overkill at the time.
Every time it happened, it made for a long heat up cycle to warm the water and rads and eventually the house.
So I built an Arduino controlled NC relay that removed power for 1 minute out of every 120. That was often enough to eliminate the effect of the fault, but not so often that I had concerns about filling too much gas if the boiler ever failed to ignite. 12 failed ignitions per day wouldn’t give a build up to be worried about.
That ~20 lines of code kept it working for several years until the boiler was replaced.
Our boiler has a pump to cycle hot water around the house - this makes it so you get warm water right away when you turn on a faucet and also prevents pipes in exterior walls from freezing in the winter.
This stopped working, the pump is fine but the boiler was no longer triggering it.
I just wired up mains through an esp32 relay board to the pump and configured a regular timer via esphome.
Temperature based logic would be even better but I didn't find a good way to measure pipe temperature yet.
IIRC, I used an RTD that I had left over from a 3D printer upgrade, but an 18B20 would fine as well. A 10K NTC resistor might even be good enough. For what I needed (and I think for what you need), just fixing the sensor to the outside of the pipe [if metal] will give you a usable signal. That sensor was just metal HVAC taped to the front cast iron door of the burner chamber.
But a dead-simple timer solution gets you pretty far as you know.
But as you say, timer works good enough and that means little motivation to continue to work on it -- countless other projects await :)
BTW I've also tuned the timer to run for longer in the morning to get a hot shower ready.
Edit: nice dashboard, what are you using for the chart? I like the vintage look.
I have a mix of shame and pride that the chart (everything in the rectangle) is entirely hand-coded SVG elements emitted by the ESP web request handler.
I’ve been working on the same thing for a few months now.
Not only is it more customizable / less complicated than helm / other solutions, but GNU gettext is almost 30?! years old at this point, and environment variables are probably realistically double that age. They aint going anywhere anytime soon.
Plus I feel that more complex logic removes value from the configs we are building, and so am not interested in many other tools.
For all its faults, it worked and it was generating revenue. Enough that the startup got to a sizable Series A. That experience completely changed how I think about writing code at startups.
i see no reason to stop using these types of solutions, when appropriate.
I'm not sure if they were "extremely stable" though. Like Myspace, it might only work up until a certain point. What kills stuff is usually going viral.
If you don’t trust a filesystem, you can’t trust anything that uses one.
Flat files don’t scale past a certain point, but that point is way higher than most believe it is.
And "fork off a new process just to do the save" helps move that point farther away.
I was so happy when we reworked our classes to no longer use AWS workspaces.
In the text file you have something you want to template (or "parametrize") from an outside variable, so you name that something like @@VAR@@ and then you can sed that @@VAR@@ :-)
It was a big old-fashioned bookseller trying to compete with Amazon. Software and the web was locked down tight, but they opened a daily report in Excel, and I built a VBA macro that generated the necessary HTML and published the images to an FTP server. Turned a 2 day job into a 10 minute one.
That was a structural crash and I should not have addressed it.
At Amazon they schedule service bounces during code freeze for any service that is known to have memory leaks because it's easier than finding the leak, which isn't usually an issue since it gets deployed so often.
It worked great and was way easier to deploy than k8s setups. However after some testing we found out that the core logic of the app - a long running process, would just crawl to a halt after some time.
It turned out google wanted to push you to use their (paid) queue / pubsub solutions, but they didn’t want it to be _too_ obvious, so cloudrun would actually throttle its cpu sometime after the request it was spawned was returned.
Our logic was based on pushing stuff in a queue and having it be processed outside of a request, but google just f*ked with that solution.
And it would have been fine if that was upfront info, but it was buried in a doc page somewhere obscure, small print…
Thats the time I realized how bad gcp can be…
The CMS didn't support any kind of HTML/JS embed, and had quite a short character limit per "Text" block. But luckily, that block didn't filter out _all_ inline HTML - only some characters.
So, a bootstrap "Loading" element was inserted, along with a script tag which would bring in the rest of the resources and insert those in to the page where the bootstrap was placed. This quickly became a versatile, re-usable loader, and allowed us to launch the features. But all this grew from a very inelegant hack which just happened to work.
So I dynamically replaced the part of their code that was wrong. That monkey patch has run years and is still going :)
The company I was working for purchased some kind of email spamming program to send mass emails. However they quickly found that when more than a small number (10 to 15,000?) of addresses were in the list it took forever.
Support wasn’t being helpful, so I decided to look at it. It was some kind of PHP application and what I eventually found was it was doing a database query doing table scans for everything because there was a missing index that was incredibly obvious.
I added the index, the program worked well into huge numbers of emails addresses, and we informed the vendor.
The program was soon updated and it fixed the problem as well. Also all the PHP code was now obfuscated/encrypted.
I guess they didn’t like my helpful nature.
We called it the Frozen Throne.
It ended up wearing out one relay, but he had a spare and just swapped it in and restarted the script from where it hung up. For the cost of like $30 and a few days, it gave us a new unobtanium unlocked unit.
I use to work part time restoring Fiat, Porsche, and VW rares for an old head out in the mid west, lots of "stupid but works" in those old cars... Mercedes Benz once (1980s or so) employed glass containers to solve for fuel pressure problems. Insane coolant loop designs or early Fuel Injection systems that develop "Ghosts" lol...
Turns out, if you want to turn html+css into pdfs quickly, doing via a browser engine is a "works really well" story.
Writing this package made me realize that even big projects (such as Chromium) sometimes have features that just don't work. Edge headless wouldn't let you take screenshots up until recently, and I still encountered issues with Firefox last time I tried to add support for it in the package. I also stumbled upon weird behaviors of Chrome CDP when trying to implement an alternative to using the headless mode, and these issues eventually fixed themselves after some Chrome updates.
[1] https://github.com/vgalin/html2image
But with pdfs the money is on getting those headers and footers consistent and on every page, so you do need some handcrafted html and print styling for that (hint: the answer is tables).
People complained that the PDFs generated were slightly different. So instead I had the client send over the entire html in a post request and open it up in a headless chrome with --print-to-pdf and then sent it back to the client.
But for real I'd get fired if I said...
So what happened is: the customer would order their phone subscription on the front end, that would create a job file that would be sent to a scheduler that managed 10 Windows VMs that used a Ruby Watir script to direct IE6 to fill in the data from the job file on the old decrepit website.
It's the most horrific hack that I ever touched (I forgot exactly, but I had to make some adjustments to the system), but it worked perfectly for a couple of years until those providers finally updated their websites.
I ended up swapping it out to a generic in-line CPAP humidifier, but at the same time, realized I could partially automate the process of refilling the chamber (and not have to keep unhooking hoses) by adding an in-line oxygen tee, some aquarium plumbing, a check valve, and a 12 volt pump and switch.
In the morning I just hold a button and the tank magically refills itself ;)
Introducing Semi-Autofill(tm): https://i.ibb.co/NmDbVvw/autofill.png
(Also: The Dreamstation, while recalled, was personally de-foamed and repaired myself -- I don't trust Philips any further than I can throw them now. I now self-service my gear.)