Totally. I live in a place where many (most?) ISP plans have limited monthly downloads. I'd be so pissed off if my monthly allowance was blown by this series of boneheaded decisions.
And in that case, the problem would not be discovered until 1) someone opened a bug report, which rarely happens, because any competent user would just disable auto-updates, and 2) that bug report would be investigated, which also rarely happens.
Contrarian approach: $8000 is not a lot in this context. What did the CEO think of this? Most of the time it is just a very small speed bump in the overall finances of the company.
Avoidable, unfortunate, but the cost of slowing down development progress e.g. 10% is much higher.
But agree that senior gatekeepers should know by heart some places where review needs to be extra careful. Like security pitfalls, exponential fallback of error handling, and yeah, probably this.
I'm sure it cost a lot more than $8000. That was only the direct visible cost to them. There were likely users hit with costs for the additional downloads, who never even knew what was the issue. Users working on a mobile hotspot who had to pay for extra data etc etc.
Imagine if that was Meta that had over 1B users with their messenger desktop app update functionality that did just that. The loss would be in the hundreds of millions.
> But agree that senior gatekeepers should know by heart some places where review needs to be extra careful. Like security pitfalls, exponential fallback of error handling, and yeah, probably this.
The lesson here is much better use of automated tests (The app likely has no tests at all) and proper use of basic testing principles like TDD would prevent such junior-level embarrassing bugs creeping up in production paid software.
That is the difference between a $100 problem vs a $200M problem.
See the case of Knight Capital [0] who lost $460M, due to a horrific deploy.
I doubt there’s a CEO. Despite the use of “we”, pretty sure this is one guy building the app. All the copyright notices and social media go back to one person.
How do you adjust your testing approach to catch cases like this? In my experience, timing related issues are hard to catch and can linger for years unnoticed.
I would mock/hook/monkey patch/whatever the functions to get the current time/elapsed time, simulate a period of time (a day/week/month/year/whatever), make sure the function to download the file is called the correct amount of times. That would probably catch this bug.
Although, after such a fuck up, I would be tempted to make a pre-release check that tests the compiled binary, not any unit test or whatever. Use LD_PRELOAD to hook the system timing functions(a quick google shows that libfaketime[0] exists, but I've never used it), launch the real program and speed up time to make sure it doesn't try to download more than once.
Similar to doix said, consider reading the time as IO and then rewrite the code in sans-IO style so you can inject the time.
Then it's a unit test that looks too obvious to exist until you read the ticket mentioned in the comment above it
No need for monkey patching or hooking or preload
But before that you add a couple checkmarks to the manual pre-release test list: "1 hour soak test" and "check network transfer meters before and after, expect under 50 MB used in 1 hour (see bug #6969)"
I worked on a product where there was basically no automated testing, just a huge product surface to click around with a bunch of options. Because of technical debt some of the options would trigger different code paths, but it was up to the developer to memorize all the code paths and test accordingly.
After I shipped a bug the Director of Engineering told me I should "test better" (by clicking around the app). This was about 1 step away from "just don't write bugs" IMO.
Yep, my first job was at a company like that. Huge Windows desktop app built in Delphi. No automated testing of any kind. No testing scripts either. Just a lot of clicking around.
My first job was exactly that, selling windows app in Delphi. I joined the new team working on .net windows apps and we had an army of people clicking on UI all day long.
They maintained their "test plan" on a custom software where they could report failures.
TBH, that was well done for what it was but really called for automation and lacked unit-testing.
I am forced to use a custom kv store for my current t project. That pos has a custom dsl, which can only be imported through a swing ui, by clicking five buttons. Also, the ui is for 1024 screens, they are tiny in my 4k monitor
I don't think the author is wrong for saying that certain kinds of code should be written carefully. I object to the implication that other code shouldn't.
From TFA: "Write your auto-updater code very carefully. Actually, write any code that has the potential to generate costs carefully." So the focus is on code that "generate[s] costs". I think this is a common delusion programmers have; that some code is inherently unrelated to security (or cost), so they can get lazy with it. I see it like gun safety. You have to always treat a gun like it's loaded, not because it always is (although sometimes it may be loaded when you don't expect it), but because it teaches you to always be careful, so you don't absent-mindedly fall back into bad habits when you handle a loaded one.
Telling people to write code carefully sounds simplistic but I believe for some people it's genuinely the right advice.
> We decided to take responsibility and offer to cover all the costs related to this situation.
Good on them. Most companies would cap their responsibility at a refund of their own service's fees, which is understandable as you can't really predict costs incurred by those using your service, but this is going above and beyond and it's great to see.
I would also put into question if you _really_ need to check for updates every 5 minutes. Once per startup is already enough, and if you're concerned about users who leave it on for days, it could easily be daily or even less often.
It's near-instant now not usually because of more incessant polling, but because it simply keeps the connection open (can last many hours without sending a single byte, depending also on the platform) and writes data onto it as needed (IMAP IDLE). This has gotten more efficient if anything
In the early days, it was a long distance call to either Lisbon or Porto, I only got a modem, when regional numbers were available to the district capitals.
However on BBS days was much worse, it was mostly long distace calls to someone around the country, and they usually only had a couple of connections available like five or so.
Ah another thing is that they adopted the same model as mobile phones, so at least we could pre-pay the calls, and went we run out of cash there was it, no surprise bills, even if frustated.
Yeah but that should be a variable anyways. Maybe even a variable provided by the server. But in this case it should be on demand. with the old version cached and only downloading the new one when there is a new version once a day.
First thing I thought as well. Every 5 minutes for a screen recording software is an absurd frequency. I doubt they release multiple new versions per day.
A 5 minute update check interval is usage-reporting in disguise. Way fewer people would turn off a setting labeled “check for updates” than one labeled “report usage statistics”.
Eh, this one is probably ignorance over malice. It's super common to see people who need to make an arbitrary interval choice go with 300 out of habit.
Screen Studio can collect basic usage data to help us improve the app, but you can opt out of it during the first launch. You can also opt out at any time in the app settings.
In fact, assume the opposite unless you have a reason to assume otherwise (aka a close personal relationship). Giving strangers/businesses that you have no connection to the benefit of the doubt when they harm you is a good way to get taken advantage of.
Never contort your reasoning to attribute to incompetence what is much better explained by malice. Especially when politics or money is involved, malice should be the assumed default.
To be as user friendly as possible, always ask if user wants automatic background updates or not. If you can’t update without user noticing it, please implement manual updates as two mechanisms:
1) Emergency update for remote exploit fixes only
2) Regular updates
The emergency update can show a popup, but only once. It should explain the security risk. But allow user to decline, as you should never interrupt work in progress. After decline leave an always visible small warning banner in the app until approved.
The regular update should never popup, only show a very mild update reminder that is NOT always visible, instead behind a menu that is frequently used. Do not show notification badges, they frustrate people with inbox type 0 condition.
This is the most user friendly way of suggesting manual updates.
You have to understand, if user has 30 pieces of software, they have to update every day of the month. That is not a good overall user experience.
> You have to understand, if user has 30 pieces of software, they have to update every day of the month. That is not a good overall user experience.
That's not an user issue tho, it's a "packaging and distribution of updates" issue which coincidentally has been solved for other OS:es using a package manager.
I'd also question if the updater needs to download the update before the user saying they want it. Why not check against a simple endpoint if a newer version is available and if so, prompt the user that an update could be downloaded and then download it. This would also allow the user to delay the update if they are on metered connections.
Their users do not care about their screen recording studio anywhere near as much as the devs who wrote it do.
Once a month is probably plenty.
Personally, I disable auto-update on everything wherever possible, because the likelihood of annoying changes is much greater than welcome changes for almost all software I use, in my experience.
What might be fun is figuring out all the ways this bug could have been avoided.
Another way to avoid this problem would have been using a form of “content addressable storage”. For those who are new, this is just a fancy way of saying make sure to store/distribute the hash (ex. Sha256) of what you’re distributing and store it on disk in a way that content can be effectively deduplicated by name.
It’s probably not so easy as to make it a rule, but most of the time, an update download should probably do this
Note that billing alerts protect against unexpected network traffic, not directly against bugs and bad design in the software. Update checking remains a terrible idea.
I am always kind of a stickler about code reviews. I once had a manager tell me that I should leave more to QA with an offhand comment along the lines of "what is the worst that could happen" to which I replied without missing a beat "We all lose our jobs. We are always one bad line of code away from losing our jobs"
The number of times I have caught junior or even experienced devs writing potential PII leaks is absolutely wild. It's just crazy easy in most systems to open yourself up to potential legal issues.
When I work on my own code, at home, with no-one to assist or review, I write tests, and open a PR anyway, and review it myself, sometimes the next day with fresh eyes, or even 10 minutes later after a quick walk in and out of the room and a glass of water.
If you're not confident you can review a piece of code you wrote and spot a potentially disastrous bug like the one in OP, write more tests.
These days you can dump your code/diff to ChatGPT and ask it to criticize. Often most of it is junk -- irrelevant or incorrect suggestions. But the 1 in 10 suggestions is actually useful and correct, which can be immensely helpful.
Code reviews kill velocity - introduce context switching, and are make work, it feels like you’re doing something to make a PR etc but your not.
The context it makes the most sense is accepting code from strangers in a low trust environment.
The alternative to trying to prevent mistakes is making it easy to find and correct them. Run CI on code after it’s been merged and send out emails if it’s failed. At the end of a day produce a summary of changes and review them asynchronously. Use QA, test environments, etc.
> Code reviews kill velocity - introduce context switching, and are make work
This is the same point three times, and I don't agree with it. This is like saying tests kill velocity, there's nothing high velocity about introducing bugs to your code base.
Everything introduces context switching, there's nothing special about code reviews that makes it worse than answering emails, but I'm not going to ignore an important email because of "context switching."
Everyone makes mistakes, code reviews are a way to catch those. They can also spread out the knowledge of the code base to multiple people. This is really important at small companies.
CI is great, but I have yet to see a good CI tool that catches the things I do.
> Why not require two or three reviews if they are so helpful at finding mistakes?
Places do? a lot of opensource projects have the concept of dual reviews, and a lot of code bases have CODEOWNERS to ensure the people with the context review the code, so you could have 5-10 reviewers if you do a large PR
> No it isn’t. Fake work, synchronization, and context switching are all separate problems
Context switching is a problem because it...kills velocity. Fake work is a problem because it kills velocity. You're saying it's time that could be better spent elsewhere, but trying to make it sound wider. I disagree with the premise.
Synchronization is a new word, unrelated to what you originally wrote.
> How many times have you gone back to address review comments and introduced a regression because you no longer have the context in your head?
Never? I am not unable to code in a branch after a few days away from it. If I were, I would want reviews for sure! Maybe you have had reviews where people are suggesting large, unnecessary structural changes, which I agree would be a waste of time. We're just looking for bug fixes and acceptably readable code. I wouldn't want reviewers opining on a new architecture they read about that morning.
I guess they say never, because if you have descriptive commit messages and write good PR description, it's easy to regain the context surrounding the change. It's all about communication, both to others and your future self.
> Why not require two or three reviews if they are so helpful at finding mistakes?
For secure software, e.g. ASIL-D, you will absolutely have a minimum 2 reviewers. And that’s just for the development branch. Merging to a release branch requires additional sign offs from the release manager, safety manager, and QA.
By design the process slows down “velocity”, but it definitely increases code quality and reduces bugs.
Once again let me reframe the mindset. Trying to get a perfect change where you anticipate every possible thing that will go wrong beforehand is impossible - or at least extremely costly. The alternative is to spend your effort on making it easy to find and fix problems after.
You are not anticipating every possible bugs. It's mostly a learning experience for you and the team if it's done correctly. Someone may proposes another approach, highlight certain aspects that needs to be done "right" (definition may vary), let you know possible pitfalls, etc... It's not always LGTM.
This feels like a strange sense of priorities which would be satirised in a New Yorker/Far Side single-panel comic: “Sure, my mistake brought down the business and killed a dozen people, but I’m not sure you appreciate how fast I did it”.
Code should be correct and efficient. Monkeys banging their heads against a keyboard may produce code fast, but it will be brittle and you’ll have to pay the cost for it later. Of course, too many people view “later” as “when I’m no longer here and it’s no longer my problem”, which is why most of the world’s software feels like it’s held together with spit.
> would be satirised in a New Yorker/Far Side single-panel comic:
Thanks for taking my experience and comment seriously and challenging your preconceptions.
> Code should be correct and efficient.
When it ships to customers. The goal is to find the bugs before then. Having a stable branch can be accomplished in many ways besides gating each merge with a review.
Do you have any studies to show how effective synchronous code review is in preventing mistakes? If they are such a good idea why not do 2 or 3?
> Thanks for taking my experience and comment seriously and challenging your preconceptions.
I apologise if my comment read as mean. I wanted to make the joke and it may have overshadowed the point.
> Do you have any studies to show how effective synchronous code review is in preventing mistakes?
I could’ve been clearer. I’m not advocating for code reviews, I’m advocating for not placing “velocity” so high on the list of priorities.
> If they are such a good idea why not do 2 or 3?
This argument doesn‘t really make sense, though. You’ve probably heard the expression “measure twice, cut once”—you don’t keep measuring over and over, you do it just enough to ensure it’s right.
> Thanks for taking my experience and comment seriously and challenging your preconception
I'm honestly curious what you're experience level is? I've never known a developer with more than a couple years of experience valuing velocity over financial stability
The up-front cost of code review can be easily be tripled or quadrupled when it’s distributed over several weeks after the fact in the form of unplanned work, each instance of which incurs its own cost of context switching, as well as the cost of potential rework.
The purpose of such a review is a deliberate bottleneck in the earlier stage of development to stop it becoming a much larger bottleneck further down the line. Blocking one PR is a lot cheaper than blocking an entire release, and having a human in the loop there can ensure the change is in alignment in terms of architecture and engineering practices.
CI/CD isn’t the only way to do it but shifting left is generally beneficial even with the most archaic processes.
To catch bugs and avoid the bottleneck that is code review, you’re just moving the bottleneck to some point after merging the code, not before.
Like it or not you still have to stop what you’re doing to identify a bug and then fix it, which takes time away from planned feature work. You’re not optimising anything, you’re just adding fragility to the process.
As I said before, an issue localised to a PR in review blocks one person. An issue that has spread to staging or prod blocks the entire team.
You can still get delta updates with Sparkle in an electron app. I am using it, and liking it a lot more than Electron Updater so far: https://www.hydraulic.dev
Well when I follow a desktop link on my phone, it redirects me to the mobile version, despite the URL specifically asking to serve the desktop site, it just doesn't work the other way around. Plus I never asked to see the mobile site, I followed a link someone else posted
The scale is astounding. I was briefly interested in the person that caused the error then immediately realized it was irrelevant because if a mechanism doesn't exist to catch an issue like that, then any company is living on borrowed time.
I read that as "A single line of code costs $8000" which, from the comments seems like a few others had the same thought. Reading the article it is not costs and the original title is "One line of code that did cost $8,000", so as some others have pointed out it is a bug that cost $8000.
> While refactoring it, I forgot to add the code to stop the 5-minute interval after the new version file was available and downloaded.
I’m sorry but it’s exactly cases like these that should be covered by some kind of test, especially When diving into a refactor. Admittedly it’s nice to hear people share their mistakes and horror stories, I would get some stick for this at work.
I assume most of that 2PB network traffic was not egress, right? Otherwise how did it "only" cost you $8k on Google Cloud?
Even at a cost of 0.02$ per GB, which is usually a few times lower than the actual prices I could find there, that would still result in an invoice of about $40k...
The first 500TB should have cost $35k already. At that point pricing goes from $0.06/GB to "contact us". So I'd have expected google to charge $80k or so for the whole thing. (Unless google decided to forgive most of the cost)
These articles are great, but I have to one-up the blog: I recently helped a small dev team clean up a one-line mistake that cost them $95,000... which they didn't notice for three months.
The relevance is that instead of checking for a change every 5 minutes, the delay wasn't working at all, so the check ran as fast as possible in a tight loop. This was between a server and a blob storage account, so there was no network bottleneck to slow things down either.
It turns out that if you read a few megabytes 1,000 times per second all day, every day, those fractions of a cent per request are going to add up!
>Add special signals you can change on your server, which the app will understand, such as a forced update that will install without asking the user.
I understand the reasoning, but that makes it feel a bit too close to a C&C server for my liking. If the update server ever gets compromised, I imagine this could increase the damage done drastically.
> Screen Studio is a screen recorder for macOS. It is desktop app. It means we need some auto-updater to allow users to install the latest app version easily.
No, it doesn't mean that.
Auto updater introduced series of bad outcomes.
- Downloading update without consent, causing traffic for client.
- Not only that, the download keeps repeating itself every 5 minutes? You did at least detect whether user is on metered connection, right... ?
- A bug where update popup interrupts flow
- A popup is a bad thing on itself you do to your users. I think it is OK when closing the app and let the rest be done in background.
- Some people actually pay attention to outgoing connections apps make and even a simple update check every 5 minutes is excessive. Why even do it while app is running? Do on startup and ask on close. Again some complexity: Assume you're not on network, do it in background and don't bother retrying much.
- Additional complexity for app that caused all of the above. And it came with a price tag to developer.
Wouldn't app store be perfect way to handle updates in this case to offload the complexity there?
That was a thing I thought was missing from this writeup. Ideally you only roll up the update to a small percent of users. You then check to see if anything broke (no idea how long to wait, 1 day?). Then you increase the percent a little more (say, 1% to 5%) and wait a day again and check. Finally you update everyone (who has updates on)
yes obviously something as mature as the App store supports phased rollout. I believe it is even the default setting once you reach certain audience sizes. Updates are always spread over 7 days slowly increasing the numbers
App store updates are perfect: no unnecessary complications, no unnecessary work (assuming Screen Studio is published and properly updated in the app store), and the worst case scenario is notifications about a new Screen Studio version ending up in a Screen Studio recording in progress.
Thinking of it, the discussed do-it-yourself update checking is so stupid that malice and/or other serious bugs should be assumed.
Exactly. The AppStore already exists and does updates (either automatically or manually, configurable by the user). The developer didn't have to lift a finger to get this functionality. Imagine sitting down and spending time adding functionality to your application that is already provided for free by the operating system, and then after all that, doing it incorrectly!
Starting from the paid developer accounts, the Apple app store isn't "provided for free by the operating system" and it is a source of endless busywork, fear and suffering, but the argument stands: a professional Macintosh software vendor uses the app store because Macintosh users expect it, so it can be assumed that "properly" publishing new software version to the app store is a sunken cost that should be made as useful as possible.
By "provided for free" I mean the App Store comes with the OS, costs nothing (monetarily) to the developer over the existing annual Apple Developer Program fee, which pretty much all macOS developers pay anyway, and can be counted on to exist on all macOS installations.
Yeah no, publishing to the App Store is a nightmare in cost and time. I can 100% guarantee they still saved money on 30% fees even after this $8000 snafu.
Screen Studio has 32k followers, lets say 6% are end users, 2000 users at $229, that is $137k in App Store fees.
I am going to say writing your own app update script is a wash time wise, as getting your app published is not trivial, especially for an app that requires as many permissions as screen studio.
Some people don’t like using the AppStore. I like to keep backups of installers so I can control the version. And if it gets pulled from the AppStore, I’ll always have a copy.
> malice and/or other serious bugs should be assumed
Going back to the blog post and re-reading it with this possibility in mind is quite a trip.
> It turns out thousands of our users had the app running in the background, even though they were not using it or checking it for weeks (!). It meant thousands of users had auto-updater constantly running and downloading the new version file (250MB) over and over again every 5 minutes
This could easily have been data exfiltration from client computers instead, and few (besides the guy whose internet contract got cancelled for heavy traffic) would have noticed.
I find the misbehavior of indie/boutique MacOS apps always insisting on starting at login very irritating. Unless the app needs to run some heavy background preparation steps before becoming usable, there is literally no sense it starting at login. Also when dormant, check for update(once every 24h), and nag the user if they want to update, but please do not auto download! A lot of non-tech folks use 128/256GB versions of macbook with trillions of photos already clogging their device, an app downloading new updates to add to the pain unless the user asks to do so is outright malice.
And even when nothing changed?!? Fucking lazy developers aka "I have an idle ≥1Gb/s pipe to the download server". What happened to rsync/zsync/zstd (with dictionary)? There are so many good tools freely available to reduce wasted bandwidth when you insist on reinventing the wheel. sigh
If you’re a small shop or solo dev, it is real hard to justify going native on three platforms when electron gives it for (near) free. And outside of HN, no one seems to blink at a 250MB bundle.
There are alternatives like Tauri that use the system browser and allow substantially smaller bundles, but they’re not nearly as mature as Electron, and you will get cross platform UI bugs (some of which vary by user’s OS version!) from the lack of standardization.
I’d actually seen this project before because the author did a nice write up on using React portal to portal into electron windows[1], which is something I decided to do in my app.
I’d just assumed his was a cross platform project.
I go into more detail in a sibling comment, but Tauri does not provide a standardized web runtime. The webview you get depends on your OS and OS version. They’re all “WebKit”, but definitely do not all render the same. I have built a Tauri app and switched to Electron after encountering multiple x-plat rendering bugs.
Different versions of electron bundle different versions of chromium. There can/will be rendering differences between them.
Tauri is an alternative framework that uses whatever web view the OS provides, saving ~200mb bundle size. On Mac that’s a (likely outdated) version of Safari. On Windows it’ll be Edge. Not sure what Linux uses, I’d guess it varies by distro.
The promise of Electron (and it’s an amazing value prop) is that your HTML/JS UI will always look and work the same as in your dev environment, no matter what OS the host is running.
I don’t have the time or inclination to test my app on the most recent 3 releases of the most popular operating systems every time I change something in the view layer. With Electron, I trade bundle size for not having to do so.
I do think alternatives like Tauri are compelling for simple apps with limited UI, or where a few UI glitches are acceptable (e.g. an internal app). Or for teams that can support the QA burden.
I’m not excusing this bug. There are several poor decisions that went into this issue, but my contention is that using electron (with the resulting 250mb bundle) is not one of them.
You should probably not roll your own auto-updater.
If you do, checking every 5 minutes for updates is waaaay too often (and likely hurts battery life by triggering the radio).
And triggering a download without a user-prompt also feels hostile to me.
The app size compounds the problem here, but the core issue is bad choices around auto-updating
FWIW the transitive dependencies of the nixOS ffmpeg add up to 764MB, but dynamically linking is always much larger than statically linking, and that calculation will include more than just the shared-libraries.
Also note that he app includes an ffmpeg that is 39MB uncompressed.
Per [1] That would be 39MB (uncompressed; probably about half that compressed) to include ffmpeg-darwin-arm64, since OS X doesn't ship with ffmpeg installed.
> Wouldn't app store be perfect way to handle updates
But then the HN crowd would complain "why use an app store? that's gate keeping, apple could remove your app any day, just give me a download link, and so on..."
You can? Don’t check for updates every 5 minutes. Daily or even weekly would be sufficient for an app like this (if auto-updater is even necessary at all.. just show a notification)
422 comments
[ 86.7 ms ] story [ 7952 ms ] threadThe title should have been: "how a single line of code cost our users probably more than $8000"
Just amazed. Yea ‘write code carefully’ as if suggesting that’ll fix it is a rookie mistake.
So so frustrating when developers treat user machines like their test bed!
Avoidable, unfortunate, but the cost of slowing down development progress e.g. 10% is much higher.
But agree that senior gatekeepers should know by heart some places where review needs to be extra careful. Like security pitfalls, exponential fallback of error handling, and yeah, probably this.
> But agree that senior gatekeepers should know by heart some places where review needs to be extra careful. Like security pitfalls, exponential fallback of error handling, and yeah, probably this.
The lesson here is much better use of automated tests (The app likely has no tests at all) and proper use of basic testing principles like TDD would prevent such junior-level embarrassing bugs creeping up in production paid software.
That is the difference between a $100 problem vs a $200M problem.
See the case of Knight Capital [0] who lost $460M, due to a horrific deploy.
[0] https://www.henricodolfing.com/2019/06/project-failure-case-...
I doubt there’s a CEO. Despite the use of “we”, pretty sure this is one guy building the app. All the copyright notices and social media go back to one person.
Although, after such a fuck up, I would be tempted to make a pre-release check that tests the compiled binary, not any unit test or whatever. Use LD_PRELOAD to hook the system timing functions(a quick google shows that libfaketime[0] exists, but I've never used it), launch the real program and speed up time to make sure it doesn't try to download more than once.
[0] https://github.com/wolfcw/libfaketime
Then it's a unit test that looks too obvious to exist until you read the ticket mentioned in the comment above it
No need for monkey patching or hooking or preload
But before that you add a couple checkmarks to the manual pre-release test list: "1 hour soak test" and "check network transfer meters before and after, expect under 50 MB used in 1 hour (see bug #6969)"
In Linux they're under /sys/class/net I think
After I shipped a bug the Director of Engineering told me I should "test better" (by clicking around the app). This was about 1 step away from "just don't write bugs" IMO.
TBH, that was well done for what it was but really called for automation and lacked unit-testing.
I wish I could teach everything I learned the hard way at that job
I don't get the impression they did any testing at all.
From TFA: "Write your auto-updater code very carefully. Actually, write any code that has the potential to generate costs carefully." So the focus is on code that "generate[s] costs". I think this is a common delusion programmers have; that some code is inherently unrelated to security (or cost), so they can get lazy with it. I see it like gun safety. You have to always treat a gun like it's loaded, not because it always is (although sometimes it may be loaded when you don't expect it), but because it teaches you to always be careful, so you don't absent-mindedly fall back into bad habits when you handle a loaded one.
Telling people to write code carefully sounds simplistic but I believe for some people it's genuinely the right advice.
Good on them. Most companies would cap their responsibility at a refund of their own service's fees, which is understandable as you can't really predict costs incurred by those using your service, but this is going above and beyond and it's great to see.
How the times have changed ..
The "send and receive" button is seared into my brain
I was in Spain at the time, and at first you had to connect to the Internet through a phone number in France.
Did you guys have something like that?
However on BBS days was much worse, it was mostly long distace calls to someone around the country, and they usually only had a couple of connections available like five or so.
Ah another thing is that they adopted the same model as mobile phones, so at least we could pre-pay the calls, and went we run out of cash there was it, no surprise bills, even if frustated.
Screen Studio can collect basic usage data to help us improve the app, but you can opt out of it during the first launch. You can also opt out at any time in the app settings.
1) Emergency update for remote exploit fixes only
2) Regular updates
The emergency update can show a popup, but only once. It should explain the security risk. But allow user to decline, as you should never interrupt work in progress. After decline leave an always visible small warning banner in the app until approved.
The regular update should never popup, only show a very mild update reminder that is NOT always visible, instead behind a menu that is frequently used. Do not show notification badges, they frustrate people with inbox type 0 condition.
This is the most user friendly way of suggesting manual updates.
You have to understand, if user has 30 pieces of software, they have to update every day of the month. That is not a good overall user experience.
That's not an user issue tho, it's a "packaging and distribution of updates" issue which coincidentally has been solved for other OS:es using a package manager.
If the update interval had been 1 day+, they probably wouldn't have noticed after one month when they had a 5 minute update check interval.
This could have easily been avoided by prompting the user for an update, not silently downloading it in the background... over and over.
Their users do not care about their screen recording studio anywhere near as much as the devs who wrote it do.
Once a month is probably plenty.
Personally, I disable auto-update on everything wherever possible, because the likelihood of annoying changes is much greater than welcome changes for almost all software I use, in my experience.
It is sort of fun (for $8,000) as it was “just” a screenshotter, but imagine this with bank app or any other heavily installed app.
All cloud providers should have alerts for excessive use of network by default. And they should ask developers if they really want to turn alerts off.
I remember Mapbox app that cost much more, just because provider did charge by months… and it was a great dispute who’s fault it was…
What might be fun is figuring out all the ways this bug could have been avoided.
Another way to avoid this problem would have been using a form of “content addressable storage”. For those who are new, this is just a fancy way of saying make sure to store/distribute the hash (ex. Sha256) of what you’re distributing and store it on disk in a way that content can be effectively deduplicated by name.
It’s probably not so easy as to make it a rule, but most of the time, an update download should probably do this
The most obvious one is setting up billing alerts.
Past a certain level of complexity, you're often better off focusing on mitigation that trying to avoid every instance of a certain kind of error.
The number of times I have caught junior or even experienced devs writing potential PII leaks is absolutely wild. It's just crazy easy in most systems to open yourself up to potential legal issues.
The website makes it seem like it's a one person shop.
If you're not confident you can review a piece of code you wrote and spot a potentially disastrous bug like the one in OP, write more tests.
The context it makes the most sense is accepting code from strangers in a low trust environment.
The alternative to trying to prevent mistakes is making it easy to find and correct them. Run CI on code after it’s been merged and send out emails if it’s failed. At the end of a day produce a summary of changes and review them asynchronously. Use QA, test environments, etc.
This is the same point three times, and I don't agree with it. This is like saying tests kill velocity, there's nothing high velocity about introducing bugs to your code base.
Everything introduces context switching, there's nothing special about code reviews that makes it worse than answering emails, but I'm not going to ignore an important email because of "context switching."
Everyone makes mistakes, code reviews are a way to catch those. They can also spread out the knowledge of the code base to multiple people. This is really important at small companies.
CI is great, but I have yet to see a good CI tool that catches the things I do.
No it isn’t. Fake work, synchronization, and context switching are all separate problems.
> code reviews are a way to catch those
I said you can do reviews - but there is no reason to stop work to do them.
Why not require two or three reviews if they are so helpful at finding mistakes?
I agree everyone makes mistakes - that’s why I would design a process around fixing mistakes, not screening for perfection.
How many times have you gone back to address review comments and introduced a regression because you no longer have the context in your head?
Diminishing returns, of course. I have worked places where two reviews were required and it was not especially more burdensome than one, though.
I catch so many major errors in code review ~every day that it's bizarre to me that someone is advocating for zero code review.
Places do? a lot of opensource projects have the concept of dual reviews, and a lot of code bases have CODEOWNERS to ensure the people with the context review the code, so you could have 5-10 reviewers if you do a large PR
Context switching is a problem because it...kills velocity. Fake work is a problem because it kills velocity. You're saying it's time that could be better spent elsewhere, but trying to make it sound wider. I disagree with the premise.
Synchronization is a new word, unrelated to what you originally wrote.
> How many times have you gone back to address review comments and introduced a regression because you no longer have the context in your head?
Never? I am not unable to code in a branch after a few days away from it. If I were, I would want reviews for sure! Maybe you have had reviews where people are suggesting large, unnecessary structural changes, which I agree would be a waste of time. We're just looking for bug fixes and acceptably readable code. I wouldn't want reviewers opining on a new architecture they read about that morning.
I believe you can figure it out.
> Never?
Ok well I’m trying to talk to people who have that problem. Because I and my team do.
For secure software, e.g. ASIL-D, you will absolutely have a minimum 2 reviewers. And that’s just for the development branch. Merging to a release branch requires additional sign offs from the release manager, safety manager, and QA.
By design the process slows down “velocity”, but it definitely increases code quality and reduces bugs.
This feels like a strange sense of priorities which would be satirised in a New Yorker/Far Side single-panel comic: “Sure, my mistake brought down the business and killed a dozen people, but I’m not sure you appreciate how fast I did it”.
Code should be correct and efficient. Monkeys banging their heads against a keyboard may produce code fast, but it will be brittle and you’ll have to pay the cost for it later. Of course, too many people view “later” as “when I’m no longer here and it’s no longer my problem”, which is why most of the world’s software feels like it’s held together with spit.
Thanks for taking my experience and comment seriously and challenging your preconceptions.
> Code should be correct and efficient.
When it ships to customers. The goal is to find the bugs before then. Having a stable branch can be accomplished in many ways besides gating each merge with a review.
Do you have any studies to show how effective synchronous code review is in preventing mistakes? If they are such a good idea why not do 2 or 3?
I apologise if my comment read as mean. I wanted to make the joke and it may have overshadowed the point.
> Do you have any studies to show how effective synchronous code review is in preventing mistakes?
I could’ve been clearer. I’m not advocating for code reviews, I’m advocating for not placing “velocity” so high on the list of priorities.
> If they are such a good idea why not do 2 or 3?
This argument doesn‘t really make sense, though. You’ve probably heard the expression “measure twice, cut once”—you don’t keep measuring over and over, you do it just enough to ensure it’s right.
Well my comment is against synchronous code reviews. So we are not in disagreement.
> you do it just enough to ensure it’s right.
I agree. Each layer of review etc is a cost and has benefits. You want to pick an appropriate level.
I'm honestly curious what you're experience level is? I've never known a developer with more than a couple years of experience valuing velocity over financial stability
Yes, they kill your velocity. However, the velocity of a team can be massively increased by shipping small things a lot more often.
Stable branches that sit around for weeks are the real velocity killer, and make things a lot more risky on deployment.
The purpose of such a review is a deliberate bottleneck in the earlier stage of development to stop it becoming a much larger bottleneck further down the line. Blocking one PR is a lot cheaper than blocking an entire release, and having a human in the loop there can ensure the change is in alignment in terms of architecture and engineering practices.
CI/CD isn’t the only way to do it but shifting left is generally beneficial even with the most archaic processes.
You’re taking a more extreme position than the one I’m stating. You can review every day or every hour if you want.
> a deliberate bottleneck in the earlier stage
Wouldn’t it be better if we could catch bugs AND avoid the bottleneck? That’s the vision. Good intentions may disagree about how to accomplish that.
Like it or not you still have to stop what you’re doing to identify a bug and then fix it, which takes time away from planned feature work. You’re not optimising anything, you’re just adding fragility to the process.
As I said before, an issue localised to a PR in review blocks one person. An issue that has spread to staging or prod blocks the entire team.
The author seemed to enjoy calculating the massive bandwidth numbers, but didn’t stop to question whether 5 minutes was a totally ridiculous.
Previous discussion: https://news.ycombinator.com/item?id=35858778
Electron really messed up a few things in this world
If just some JavaScript files change, you don't need to redownload the entire Chromium blob.
https://en.m.wikipedia.org/wiki/Knight_Capital_Group#2012_st...
440m usd
The url specifically asks Wikipedia to serve the mobile site.
Set up daily emails.
Set up cost anomaly alerts.
I’m sorry but it’s exactly cases like these that should be covered by some kind of test, especially When diving into a refactor. Admittedly it’s nice to hear people share their mistakes and horror stories, I would get some stick for this at work.
Curious where the high-water mark is across all HNers (:
Our team had a bug that cost us about $120k over a week.
Another bug running on a large system had an unmeasurable cost. (Could $K, could be $M)
The relevance is that instead of checking for a change every 5 minutes, the delay wasn't working at all, so the check ran as fast as possible in a tight loop. This was between a server and a blob storage account, so there was no network bottleneck to slow things down either.
It turns out that if you read a few megabytes 1,000 times per second all day, every day, those fractions of a cent per request are going to add up!
I understand the reasoning, but that makes it feel a bit too close to a C&C server for my liking. If the update server ever gets compromised, I imagine this could increase the damage done drastically.
No, it doesn't mean that.
Auto updater introduced series of bad outcomes.
- Downloading update without consent, causing traffic for client.
- Not only that, the download keeps repeating itself every 5 minutes? You did at least detect whether user is on metered connection, right... ?
- A bug where update popup interrupts flow
- A popup is a bad thing on itself you do to your users. I think it is OK when closing the app and let the rest be done in background.
- Some people actually pay attention to outgoing connections apps make and even a simple update check every 5 minutes is excessive. Why even do it while app is running? Do on startup and ask on close. Again some complexity: Assume you're not on network, do it in background and don't bother retrying much.
- Additional complexity for app that caused all of the above. And it came with a price tag to developer.
Wouldn't app store be perfect way to handle updates in this case to offload the complexity there?
That was a thing I thought was missing from this writeup. Ideally you only roll up the update to a small percent of users. You then check to see if anything broke (no idea how long to wait, 1 day?). Then you increase the percent a little more (say, 1% to 5%) and wait a day again and check. Finally you update everyone (who has updates on)
Thinking of it, the discussed do-it-yourself update checking is so stupid that malice and/or other serious bugs should be assumed.
Screen Studio has 32k followers, lets say 6% are end users, 2000 users at $229, that is $137k in App Store fees.
I am going to say writing your own app update script is a wash time wise, as getting your app published is not trivial, especially for an app that requires as many permissions as screen studio.
Going back to the blog post and re-reading it with this possibility in mind is quite a trip.
> It turns out thousands of our users had the app running in the background, even though they were not using it or checking it for weeks (!). It meant thousands of users had auto-updater constantly running and downloading the new version file (250MB) over and over again every 5 minutes
This could easily have been data exfiltration from client computers instead, and few (besides the guy whose internet contract got cancelled for heavy traffic) would have noticed.
If you’re a small shop or solo dev, it is real hard to justify going native on three platforms when electron gives it for (near) free. And outside of HN, no one seems to blink at a 250MB bundle.
There are alternatives like Tauri that use the system browser and allow substantially smaller bundles, but they’re not nearly as mature as Electron, and you will get cross platform UI bugs (some of which vary by user’s OS version!) from the lack of standardization.
I’d actually seen this project before because the author did a nice write up on using React portal to portal into electron windows[1], which is something I decided to do in my app.
I’d just assumed his was a cross platform project.
1: https://pietrasiak.com/creating-multi-window-electron-apps-u...
I can remember when I would have to leave a 250MB download running overnight.
Before that, I can remember when it would have filled my primary hard drive more than six times over.
... Why can't the app-specific code just get plugged into a common, reusable Electron client?
Tauri is an alternative framework that uses whatever web view the OS provides, saving ~200mb bundle size. On Mac that’s a (likely outdated) version of Safari. On Windows it’ll be Edge. Not sure what Linux uses, I’d guess it varies by distro.
The promise of Electron (and it’s an amazing value prop) is that your HTML/JS UI will always look and work the same as in your dev environment, no matter what OS the host is running.
I don’t have the time or inclination to test my app on the most recent 3 releases of the most popular operating systems every time I change something in the view layer. With Electron, I trade bundle size for not having to do so.
I do think alternatives like Tauri are compelling for simple apps with limited UI, or where a few UI glitches are acceptable (e.g. an internal app). Or for teams that can support the QA burden.
Please, many people connect to the internet via a mobile phone hotspot, at least occasionally.
This bug would likely cause you to go through your entire monthly data in a few hours or less.
You should probably not roll your own auto-updater.
If you do, checking every 5 minutes for updates is waaaay too often (and likely hurts battery life by triggering the radio).
And triggering a download without a user-prompt also feels hostile to me.
The app size compounds the problem here, but the core issue is bad choices around auto-updating
Except like 1 or maybe 2 billion people with slow or expensive internet.
Open QuickTime and hit Command-Shift-N. Press record.
FWIW the transitive dependencies of the nixOS ffmpeg add up to 764MB, but dynamically linking is always much larger than statically linking, and that calculation will include more than just the shared-libraries.
Also note that he app includes an ffmpeg that is 39MB uncompressed.
1: https://johnvansickle.com/ffmpeg/ (based on the arm64 build, since TFA is an arm64 app).
1: https://news.ycombinator.com/item?id=43839120
But then the HN crowd would complain "why use an app store? that's gate keeping, apple could remove your app any day, just give me a download link, and so on..."
You literally can't win.