This is such a great blog post. Succinct and to the point. Read it from start to finish without pause even though the stack used (which is great!) is only remotely applicable to my workflow.
Yes. Plus, from reading database outage postmortems, I was not comfortable making the "do we failover now?" decision automatic. Think about the brownouts, where the primary is still up, but slow. Or it experiences intermittent packet loss.
I've automated the mechanics of the failover, but it still must be initiated manually.
That‘s a good decision! Developers nowadays fear a failing machine so much because it‘s happening more often on the cloud than dedicated servers. I wouldn‘t use automatic failover too. Cleaning up the mess can take hours and days, a small downtime of a few minutes/hours is better.
I‘ve seen one developer using telegram for ha-failover which is great! Just a message and the scripts are executed for a failover. You can do it in seconds without even being on a computer.
Project I relate to, doing switch to Patroni managed DB (first of many) right today, in like a 5 hours, I may share observations from almost first hand experience later, if you care.
The more tools you have the more difficult it is to maintain yourself.
In a larger team with dedicated DevOps you might be right. But in my case where I maintain frontend, backend, databases, security, hosting and the servers I cannot have too much complexity. And i find container tools too complex to fully understand as a full stack software developer (not an expert in anything). Same for heavy weight frontend or backend frameworks. I like to keep it simple without too many abstractions on all layers. So far I managed to build apps used by millions of daily users for about 10 years.
I think you and the people disagreeing with you are both right, in a sense. These tools can absolutely simplify and improve even a small deployment…for someone with significant experience using them. But there’s a lot of learning that has to happen to get to that level of experience. And for a solo operator or small team without that experience, investing that time into learning may not be the best investment.
You see this contextual disagreement a ton here on HN. Those who gave taken the time to learn these tools and have practical experience using them for real deployments see the alternative as primitive, error prone and fundamentally limiting. Those without the experience see the tools as overly-complex distractions. Both are true, depending on your situation. Like almost all tech decisions, there are no universally correct answers. The best decisions are those tailored to the specific circumstances.
From what I observe, developer prefer to stay in API abstractions than spending time on servers/other infrastructure. Partially I can understand their choice.
What do you mean? The economics and decision-making are different between one-person companies through small and large organizations. When you have to know to to do everything, and fix anything, it is a problem when there is complexity, and you lack control. As organizations get larger, there is more value from abstractions and more leeway for their use.
In many ways it is a luxury to be able to use said abstractions and be able to open a ticket with someone else. Of course, a single-person startup could use the same tech, but it seriously impacts their costs and they don't have time to deal with the sprawl of abstractions.
I believe it’s the opposite? In a large org, you can have a dedicate team managing your servers. When you’re a one man business, you want to abstract as much as possible to focus on value proposition.
True. Although more often than not it's the opposite in my experience. Large organizations can afford to spend the money on GCP/aws/Azure and deploy thousands of services. They can also afford the money to hire infrastructure engineers to maintain such services. Also, in large companies everything needs to be implemented "the same way", so no matter if you want to deploy a tiny service or a huge one, you must do it the same way other services in the company are deployed.
Whereas when you are a one-man business, all you have is (free) time and prefer to spend as little money as possible while your business is not profitable. This means: walk away from dedicated databases, managed servers, huge aws bills, etc. You do everything "the old" way: rent a server (bare metal or cloud) and maintain it yourself.
In theory, a one-man business should "abstract as much as possible". In practice, it doesn't work out like this because you are responsible for everything about the business and it's easier to actually operate and maintain the relatively small number of bare-metal servers that run the business. Your advantage as a solo developer is that you can move fast at any time, and it's harder to do when you are encumbered with abstractions. That's not to say build versus buy isn't an evaluation that happens. There are strategic bets, maybe on analytics, or things that help, but aren't core to what you build.
This pattern plays out with many indie app developers as well, both now with iOS apps, and back in the day with Delphi shareware. The cost of taking a dependency on a 3rd party library turns out to be much higher for the solo developer than larger orgs. That's not to say they don't make bets, but they don't build the way that someone at a larger org would do because they have to maintain a higher degree of direct control.
Marco Arment (https://marco.org/) has elaborated a lot on this topic. You can also get a feel for the limits on the upper bounds of his developer experience. Many on HN are not willing to work at a low-level with such boring and dull technologies.
I know it's a bit silly, but anyone building out cloud services should look at this post and think about how easy it is to set up this. This is just slightly more complex than a static site, has a bit of heterogenity, but otherwise is a lot of known tools.
Can your stack provide all of this without breaking the bank for the people signing up? Can it be done without having to learn a bunch of bespoke systems? I think that Heroku's success in particular is totally down to managing this bespoke-ness balance (that is still kinda missing from container-based setups).
No they aren't. Even seemingly basic ones like Notion hide a lot of complexity.
What you're seeing is survivorship bias where the SaaS companies that are doing well are the ones that can provide a simple, clean UI whilst adding the functionality that you need for the product to be useful.
> I wish every project was as basic as a web server and database but it's pretty rare these days.
Is it? What functionality are you thinking about that absolutely require more than that?
I find that most ideas even today can be solved with a web server + DB, while developers today like to over-engineer things and think about scaling a service for a million users while they haven't even figured out the value proposition of their product/service yet.
The last couple places I worked also had a queueing system for async execution/background tasks.
A lot of apps use an "AI" or "machine learning" component which usually just means a second heterogeneous internal service.
For a little bit larger companies it's pretty common to have a data analytics or data warehouse component.
In addition, a web server + database usually require some monitoring system. After you have more than a handful of servers, probably also log aggregation.
If you're just a single or handful of people working on something, you probably don't need any of that.
Once you get a little bigger (or downtime becomes expensive) those things get tacked on quickly.
Also pragmatically choosing data centres in one physical location (even though this is a health checking service) rather than immediately going for highly available Aurora Serverless multi-AZ or something that would inflate the bill 10x.
Maybe in the past, sure. But these days people rely far more on their web apps.
Having a 1-2 day outage because your architecture is not highly-available across data centres like this one just doesn't cut it. And multi-day outages are very much real because of the thundering herd effect as companies rapidly try to move between data centres exhausting capacity.
And much of the complexity in modern day architectures come from high availability and security.
It's already multi-datacenter, since the servers are distributed across multiple datacenters on the Falkenstein campus (and they also have other locations in their eco-system).
Majority by what metric? Solo projects? Enterprise? What? That’s an incredibly broad brush that doesn’t match my experience at all. Could it be you just haven’t been involved in many complicated projects?
To be fair, the author is using several SaaS apps for core parts of his business: monitoring, email, DNS, etc. He's just "self-hosting" the more expensive parts (e.g, the EC2 parts) to get more for his money. This is great if you feel comfortable sysadmin'ing and dba'ing servers. A lot of people don't feel comfortable doing this and prefer to use ELB instead of HAProxy, RDS instead of DB servers, etc. To each his own.
What happens when the datacenter catches fire? (OVH in 2021).
If you're running a site like this, it's fine to be down for a bit, people will forgive you as you restore from backup (looks like a very solid foundation the site has).
The nice bit about this is, that you can deploy the same thing everywhere.
on other bare metals, or on AWS, azure, ...
As long as you have database backup, git code, and build-scripts.
I run something similar (although smaller).
My disaster recovery is:
I have everything prepared, to go on AWS, if necessary. Would take less, than 1 hr to be back up (database size being the biggest time sink). If I wanted to minimize that time, I could have small DB replica running, so I would just have to run last day of wall files. But for my purposes everything less than a day is good enough.
And then you can take a few days, to find something cheaper, to migrate to.
Pretty much this. I can quickly set up an entire new environment for my SaaS in under an hour; on a good day probably in about 10 minutes. It will run on any Linux environment: AWS, DigitalOcean, Linode, Azure, what-have-you.
At my last job we had one of those hyperfancy devops setups with all the fancy devops tools. Literally no one in the company knew how to spin up a new environment. I'm not exaggerating: no one knew how to run a dev environment and when they had to set up a new region for legal purposes it took weeks for the team. All of that was initially set up in an age of legends in the mythical past of a year and a half ago.
Theoretically it would all failover and scale to the moon and back. Emphasis on theoretically because no one seemed to understand how it all worked, so who knows how well it would behave when it failed. There was actually some major downtime in my time there which was rationalized away as "growing pains", but in in my opinion a major factor was just that no one really understood how it al worked.
If something goes wrong with a simple system then often the diagnosis and fix is simple (in this case: just deploy a new environment). If something goes wrong with a complex system then all of this is much harder.
Not saying you can't use these tools correctly or that there isn't an appropriate use for it, but it's a good case study on how the complexity can spin out of control if you're not careful in how you apply it.
I've been using healthchecks.io for years now. Awesome tool. Small scope, does the things it intends to do really well. Can highly recommend.
I'm surprised that the machines are that large, there must be a bunch of people using the service nowadays :) I remember the setup being even smaller in previous blog posts. But I really like it. Small, easy to reason about. Makes debugging when stuff goes wrong quite easy.
I'm a bit surprised that there's no config management listed though - seems like it really is just Fabric + a bunch of scripts. But hey, if it works, great!
Can you share more details about what the 4 HAProxy servers are doing? I run a SaaS too, but on a single beefy machine handling web and DB stuff. I'm curious what I'm missing out on not using HAProxy in front.
* the traffic from the monitored systems comes with spikes. Looking at netdata graphs, currently the baseline is 600 requests/s, but there is a 2000 requests/s spike every minute, and 4000 requests/s spike every 10 minutes.
* want to maintain redundancy and capacity even when a load balancer is removed from DNS rotation (due to network problems, or for upgrade)
There are spare resources on the servers, especially RAM, and I could pack more things on fewer hosts. But, with Hetzner prices, why bother? :-)
I didn't realize you were getting that many requests per second. That's quite remarkable when you only have 18 app cores available. In my experience, you're lucky if you can get Django to service a request in 10ms. So I'd expect closer to 40 cores to handle that level of traffic. I suppose since it is spiky, and not a particularly latency sensitive application, you can get by with fewer cores.
Damn, I never thought about it, but I guess the next cron/scheduling system should really be configurable down to the second, and if it is not defined set a random value between 0-59 seconds as a default. That might make the entire internet traffic and global compute use less spiky.
Why uWSGI? Every time I've talked to someone who uses it and got them to try gunicorn they've seen quite big performance improvement for no code change and switched.
I understand having multiple proxies, but it seems a bit odd to have more proxy servers than web servers, no? Do they also do caching so that of the up-to 4,000 requests/second, a lot of them never reach the web servers? Presumably the Python code is more resource intensive than HAProxy.
Another happy healthchecks.io user here, for many many years. The service has always worked great for me and I root for Peteris. Can happily recommend.
With ubuntu and the automatic security updates you don‘t have to spend any time maintaining it. I haven‘t done anything in months, the system is applying security updates automatically.
Hardware redundancy aside (and the good cost efficiency at Hetzner), reading the title, I'm getting the impression, the bus factor at healthchecks.io seems to be 1.
If I'd run a one man show type of business, I'd love to have some kind of plan B in case I'd be incapacitated for more than half a day. THAT would be an interesting read to me.
What volume of data are you storing in postgres? Any reason not to use a hosted postgres provider? At first glance it seems like a lot of compute for the service but curious about scale
Around 200 write tx/s as a baseline. Spikes to 2000 write tx/s at the start of every minute, and 4000 write tx/s every 10 minutes.
> Any reason not to use a hosted postgres provider?
* Cost
* Schrems II
* From what I remember, both Google Cloud SQL and AWS RDS used to have mandatory maintenance windows. The fail-over was not instant, so there was some unavoidable downtime every month. This was a while ago – maybe it is different now.
I respect that they use a combination of git tools just for version controlling. I understand that you can do everything with the git cli but my experience is that the more I tried to limit myself to the cli the messy my version controlling become.
For the healthcheck's integration solution (email, signal, discord....), how did you go about doing that? I would love to hear about all that. I did find a section about Signal's integration.
Started with just the email integration, and added other integration types over time, one by one. A few were contributed as GitHub PRs.
The Signal one took by far the most effort to get going. But, for ideological reasons, I really wanted to have it :-) Unlike most other services, Signal doesn't have public HTTP API for sending messages. Instead you have to run your own local Signal client and send messages through it. Healthchecks is using signal-cli: https://github.com/AsamK/signal-cli
>A dedicated laptop inside a dedicated backpack, for dealing with emergencies while away from the main PC.
This line here shows much thought has gone into "what if...." situations. Bravo!
I too have a similar setup - I have a contractual responsibility to my clients, and if anything goes wrong with my main equipment, I can resume ASAP. This includes a dedicated smartphone for hotspotting if my internet connection dies.
When I started, Stripe was not yet available in my country, Latvia (it now is).
Personally I've had good experience with Braintree. Particularly their support has been impressively good – they take time to respond, but you can tell the support agents have deep knowledge of their system, they have access to tools to troubleshoot problems, and they don't hesitate to escalate to engineering.
> HAProxy 2.2 for terminating TLS connections, and load balancing between app servers. Enables easy rolling updates of application servers.
Wonder why he is not using hetzner's load balancer [1]. At least cost-wise, the savings are huge (4xAX41-NVME are around 160€; LB31 - the most expensive one - is 36€)
I believe the load balancer is pretty new and it is probably the only service that allows connecting to dedicated servers -- which is separate from their cloud offering.
It's kind of interesting that there's more horsepower/cores in the load balancers than in the downstream web/app servers...4xAX41 vs 3xAX41. So I imagine something is different about the workload there versus a typical load balancer.
That product has a serious lack of docs, and I know from experience some off-cloud vendor balancers have crappy architectures with low overall throughput limits. Their product could be a shared CPU VPS under the hood or something deeply baked into their network architecture, guilty until proven innocent
I never realized that Healthchecks.io was a one man show.
I'm almost ashamed to say that i've been using the free tier for years. I moved from self hosted to the free tier when i killed off all of self hosted services, and i don't have much legitimate use for it, having it attached to a backup of a Raspberry Pi running HomeBridge in my summerhouse :)
I do use it for a purpose it probably was not designed for :) In my summerhouse, one of the circuit breakers trips every now and then (1-2 times per year) for no apparent reason, and since the fridge is connected to that breaker, i very much fear arriving at a summer house where the fridge has been unpowered for a couple of weeks.
The solution to that of course, was to setup a systemd timer on a Raspberry Pi (that also runs HomeBridge), which pings Healthchecks.io every 15 minutes. If it misses 2 pings it raises an alert.
So far i've had 2 notifications in the 2+ years it's been running, and both have been internet connection related, so monitoring the situation has apparently changed something :)
I use it for the same purpose as well, a friend left on a trip and his fridge lost power, spoiling thousands of dollars worth of his small baking business materials.
I set up a phone with Tasker, if the phone stops charging or fails to check in, it sends an alert.
Luckily, and as you say, monitoring changed the situation, as it never failed since.
Compared to the office buildings leaving their entire office illuminated all night even though the parking lot is empty, this is less than miniscule. In the grand scheme of things, yes, it matters. But start with the big fish and you'll be less hungry.
While I agree with the overall sentiment (fix the big problems first to maximize payoff), the issue is one of attitude. Keeping a "save where possible" mindset, no matter how large the immediate payoff (within reasonable limits) has a much greater chance of success.
You have that effect in many areas of life. A quite obvious example is working out. Of course it won't make much of a difference if you skip today's workout or abort your 40 min run today halfway cause you don't feel like it. But it's the mindset/attitude. It makes you a quitter. It enables you to do the same thing again. And again. And in the end, the cumulative damage of the change in mindset is magnitudes worse than the single event.
> Keeping a "save where possible" mindset, no matter how large the immediate payoff (within reasonable limits) has a much greater chance of success.
Ironically that is what kept me from setting up solar power. Being in Scandinavia, solar has a somewhat limited potential given that days are 7 hours long during winter, and December often has less than 20 hours of sunshine in total. My calculations for the "payback" time of the solar panels said that _MAYBE_ they would have saved something before their "expiry date" some 20 years into the future.
Considering the "energy and material waste" involved in creating solar panels, this task can be handled by energy companies much more efficiently than i ever can.
I am still considering a share in a windmill though :)
This isn't the same thing - you've actually calculated that you're unlikely to save in this situation. You made a sensible decision that the payback period is so long for your own uses that it runs the risk of not actually paying back/being beneficial. The "save where possible" mindset would be more like "I calculated it would be better, but my impact is less than my workplace doing X, so I won't".
Everyone should be calculating payback on these investments every couple of years, because the calculus changes depending on incentives/tech/other factors (e.g. EV purchases, or massive cost of living changes)
I’m not saying it wouldn’t produce energy, and it would probably also contribute most of the year, but the time where I need the most energy (heating and light) is also the time where it is least likely to produce any.
We have an annual power consumption of 6-8k kWh, with about half during winter (heat pump, yay). More energy is required during night time due to dropping temperatures and it being dark, so realistically, I could probably save about half.
At normal energy prices of €0.3/kWh, that means I would save €1200/year, and that’s without taking into account reduced electricity taxes for electric heating.
Taking those into account, all electricity used beyond 4000 kWh/year would be around €0.12/kWh. Taking that into account, I could save €480/year, still assuming I could save 4000 kWh/year out of 8000 total.
A solar panel installation _without_ batteries is about €13500, and saving €480/year means it would take 28 years to get back the investment.
I’m aware batteries can change that equation somewhat in my favor, but it’s ever harder to calculate anything usable with those as it depends if the battery can get charged during the day. In any case, a solar panel installation with batteries is around €20000.
Yes, the ROI takes years to achieve but the alternative is to keep throwing money at the power company (and some of that energy will likely be from coal nowadays).
Since I bought an electric car I've been drooling over the idea of having the car powered by the sun. I go into the office only on occasion so I don't need to charge fast either. But you need an available roof and an okay topography.
But in Scandinavia, I bet most energy goes to heating so improving insulation and adding ground thermal will probably be more efficient. The idea of a sun powered car is just so enticing, however. It's kinda sci-fi.
> but the alternative is to keep throwing money at the power company (and some of that energy will likely be from coal nowadays).
This is Denmark, where a large part of our power grid is based on renewables, and backup power is based on natural gas. There is no coal involved. When we import power it's from Norway/Sweden or Germany, so either renewables or nuclear. We're not at Norwegian CO2 levels (yet), but it's getting better :)
> Since I bought an electric car I've been drooling over the idea of having the car powered by the sun.
Denmark has outlawed sales of new ICE cars from 2030, so at that point solar will make MUCH more sense, as i can charge my car for "free", and the time to ROI will be much shorter.
> But in Scandinavia, I bet most energy goes to heating so improving insulation and adding ground thermal will probably be more efficient.
I thought so to, but currently, in no small part due to the Ukraine situation, lots of people are replacing oil and natural gas heaters with heat pumps or central heating, and the central heating providers are installing large heat pumps as well.
As part as getting a heat pump, my house (built 1970's and renovated 2010's) has a total heat demand of roughly 13000 kWh. Assuming a heat pump with a SCOP of 3.5, that means i will need about 3700 kWh of electricity to heat it, which means that our total electricity consumption ends up around 7500 kWh, so about half the energy (in my case) being used for heating.
We tested it, and with our usual usage pattern of using the summerhouse every 2 weeks, the brand new fridge uses less power while powered for the 12 days between departure and arrival, than it does cooling down when powered on at arrival.
Despite being brand new, it uses 6-8 hours to get from 18C to 5C. I guess because it's brand new it is well insulated, so less cooling power is needed once it's cooled down, which translates to a weaker compressor with less power consumption.
As for the Raspberry Pi, it runs idle for 99% of the time, consuming _at most_ 1.15W. That amounts to 0.8 kWh in a month, which is about as much as a TV uses in 2.5 hours, or about a quarter the power consumption of a Sonos One Speaker sitting idle [1].
The total power consumption of the summerhouse uninhabited is ~1.8kWh/day, which includes the heat pump, fridge, internet modem and router, Raspberry Pi, Security Camera and a couple of hubs for IoT.
Definitely the case that the newer more energy efficient coolers and freezers use smaller heat pumps. It can be a real problem if you keep it empty and add a bunch of ‘warm’ thermal mass too, as it can take a day or so to get it down to safe temperatures again in some pathological cases.
I started leaving a gallon jug of water (frozen) in mine as it helps give it enough thermal mass to even out the spikes.
My chest freezer maxes out at 180 watts TDP for instance, which is pretty anemic if you’re trying to freeze a pot of room temperature stew.
> this waste of electricity is very selfish considering climate change
Moving yourself to the summer house every two weeks probably has far more climate impact than the fridge.
Taking to a logical conclusion: you shouldn’t have a summer house at all. (That’s not a position that I hold, but one that seems more logical than “have a summer house; just empty out and unplug the fridge while you’re not there…”)
We don't keep easily perishable goods in there like meat or vegetables, but some of them does require cooling to stay fresh.
> just empty the fridge and turn it off while not in the summer house for a few weeks
We tried that, but the fridge uses 6-8 hours to reach target temperature despite being brand new, and keeping food fresh without cooling for 10 hours is not easy.
It actually uses less power being powered on continiously for 2 weeks (12 days from departure to arrival), than it does being powered off and working overtime for 6 hours trying to cool down. It's not much less power, but still less.
> It actually uses less power being powered on continiously for 2 weeks (12 days from departure to arrival), than it does being powered off and working overtime for 6 hours trying to cool down. It's not much less power, but still less.
I suppose you mean energy rather than power? (Cause the argument makes less sense with power.)
That's an interesting phenomenon then since it goes counter to my (rather naive) understanding of physics. There could be some anomaly with the fridges efficiency curve that causes this. Maybe it's really inefficient cooling down the contents from room temperature while being really efficient maintaining the cool temp? After all, the latter is its main purpose..
> Maybe it's really inefficient cooling down the contents from room temperature while being really efficient maintaining the cool temp? After all, the latter is its main purpose..
Normal power consumption of it (assuming 18C room temperature) is less than 0.4 kWh/day. That amounts to 4.8 kWh in a 12 day period. Cooling it down from 18C takes 5.6 kWh (measured).
Temperature varies of course, and the lower the temperature the lower the power consumption, and the heat pump keeps a minimum temperature of 12C. During long summer days, the indoor temperature can easily reach 26C, which will require more power, and i have not measured every possible scenario.
I'm not ruling out that actually cooling down the thing during winter will require less energy than keeping it running, but i'm fairly certain that doing the same from 26C will require more power than keeping it running.
This seems like a lot. It's 20 megajoules, so even at a COP of 1, it's enough energy to cool down 20 megajoules / (4182 J/kg/K) / 18 K = 265 kilos of water by 18 kelvin.
And a refrigerator is supposed to attain a COP better than 1, and I suspect you have less thermal mass around than 265 kilos of water.
Something is wrong-- either in measurement, or heat exchangers very badly occluded by dust, etc, or some other fault.
It does sound like a lot, and I’ve only measured once.
The fridge is in a cabinet, enclosed on 3 sides, so maybe it’s not letting enough hot air out ?
I could be wrong, but I think it was stated as using 184 kWh / year (at room temp 22C, optimal conditions), which is just over 0.5 kWh/day, and considering that the summerhouse is usually cooler than that (12-16C) when unoccupied, I would assume it used even less power.
As it is, for this entire week, the heat pump has (self) reported using 1.8 kWh/day, and according to my power company, my daily consumption has been 1.8 kWh/day. Considering that there are 5-7 IoT things plugged in, the fridge probably isn’t using much power.
> The fridge is in a cabinet, enclosed on 3 sides, so maybe it’s not letting enough hot air out ?
Seems likely. Once it succeeds in warming its cabinet, a big share of that heat leaks back into the fridge. (And efficiency falls because of the bigger temperature difference).
You would always expect to be better off letting the refrigerator warm up, because:
* Every bit of warming that finds its way into a powered-off refrigerator is heat the refrigerator would need to take out anyways. And the heat flowing in declines as the inside warms up.
* There's standby power, etc.
* A refrigerator is expected to be more efficient removing heat when the temperature difference between inside and outside is less.
As a counter argument, imagine an infinitely good insulation and 0 standby power (and a spherical cow inside, for illustration purposes). Then keeping it "running" is clearly better than letting the fridge warm up since power draw is 0 in the former case. Obviously the conditions are not as perfect, but with a very modern fridge, maybe you get sufficiently close that keeping running indeed is superior.
I'm not arguing that that's the case here. I'm arguing that you don't "always" expect things to be worse off letting it run. There is a threshold, when approaching the mentioned fantasy conditions, beyond which letting it run may be better.
Perfect insulation doesn't help if you open the door?
If you're saying 'well with perfect insulation it doesn't matter if it has power' - well yeah. It doesn't matter if it has power. So you aren't wasting any either leaving it on.
Heat losses are proportional to the temperature difference integrated over time, and the refrigerator is more efficient in taking heat from the inside when it is close to the temperature of the room. This means that you are better off letting it warm up, because this results in both types of losses decreasing.
This continues all the way up to perfect insulation, though ultimately it's breakeven with perfect insulation.
> Perfect insulation doesn't help if you open the door?
This doesn't change the picture, but worse, it's just completely unrelated to the scenario. Who's opening the door when everyone has left for weeks?
Uh no. The heat pump may be more efficient during the period it has the largest temperature differential, but that is ensuring it has the largest possible heat differential up front. If you don’t do that (aka keep it on), there is far less total heat to move, unless you’d have it sit so long that you’d get the same total heat transfer. Which if the insulation is good, may take many weeks.
Which seems unlikely in this case.
My chest freezer for instance turns on for less than 30 minutes a day even with people getting in and out. When I defrost it, it takes a day+ to cool down again.
The reason someone would leave the door open is to ensure no one accidentally left a jug of milk in there when they turned it off. Which would pretty much neutralize any possible gains in sheer grossness if nothing else.
> you don’t do that (aka keep it on), there is far less total heat to move, unless you’d have it sit so long that you’d get the same total heat transfer.
I think you're a little confused. I'm a little lost at even how to minimally explain it to you. The rate at which heat leaks into the refrigerator declines as the refrigerator heats up, so this metric too improves with time.
> Which if the insulation is good, may take many weeks.
Implausible. "Many weeks"?
The best 20 cu ft refrigerators use about 375 kilowatt-hours per year according to energystar.gov. At a coefficient of performance of 10, that's an average of 400W flowing into the fridge and needing to be removed by the refrigerator.
At 20K of temperature difference, that's about 20W/degK. The contents will warm very quickly, absent phase change.
Having ice in there will slow things down until the ice melts; then it will warm very quickly. The typical numbers cited say you have 2-3 days with a full freezer without power before things have mostly thawed and become unsafe-- this is well over half of the way to room temperature in terms of energy budget (and in the best case of a full freezer where there's the most thermal mass inside).
(Specific heat capacity of water ~4200 J/(kg*k); heat of fusion of water ~334000 J/(kg*k)).
None of this changes the energy balance of the problem, though.
Sounds like a win-win to me. Either you installing the thing alerts you to the problem, or instslling it makes the whole problem disappear. Either way it’ll be fixed.
I'm really proud of this project. Personally, it has been a case study for a long time. It's the absolute epitome of pragmatic engineering solutions that actually works and turn them into a SaaS as an open source project. Well done!
I use vanilla Wireguard (the "wg" command and the "wg-quick" service). I set up new hosts and update peer configuration using Fabric tasks. It may sound messy, but works fine in practice. For example, to set up Wireguard on a new host:
* On the new host, I run a Fabric task which generates a key pair and spits out the public key. The private key never leaves the server.
* I paste the public key in a peer configuration template.
* On every host that must be able to contact the new host, I run another Fabric task which updates the peer configuration from the template ("wg syncconf").
One thing to watch out is any services that bind to the Wireguard network interface. I had to make sure on reboot they start after wg-quick.
I can't say definitely, but my gut feeling is positively.
A. What if another operator takes the source code, and starts a competing commercial service?
I've seen very few (I think 1 or 2) instances of somebody attempting a commercial product based on Healthchecks open-source code. I think that's because it's just a lot of work to run the service professionally, and then even more work to find users and get people to pay for it.
B. What if a potential customer decides to self-host instead?
I do see a good amount of enthusiasts and companies self-hosting their private Healthchecks instance. I'm fine with that. For one thing, the self-hosting users are all potential future clients of the hosted service. They are already familiar and happy with the product, I just need to sell the "as a service" part.
179 comments
[ 2.9 ms ] story [ 211 ms ] threadI've automated the mechanics of the failover, but it still must be initiated manually.
I‘ve seen one developer using telegram for ha-failover which is great! Just a message and the scripts are executed for a failover. You can do it in seconds without even being on a computer.
It is so cool to see one-person SaaS business running on old school bare metal servers without needing any fancy devOps / containerize tooling.
I follow a simple rule of thumb - the more abstractions you introduce the more complex it gets to maintain
Life is easier if you can get away with lesser abstractions.
You see this contextual disagreement a ton here on HN. Those who gave taken the time to learn these tools and have practical experience using them for real deployments see the alternative as primitive, error prone and fundamentally limiting. Those without the experience see the tools as overly-complex distractions. Both are true, depending on your situation. Like almost all tech decisions, there are no universally correct answers. The best decisions are those tailored to the specific circumstances.
In many ways it is a luxury to be able to use said abstractions and be able to open a ticket with someone else. Of course, a single-person startup could use the same tech, but it seriously impacts their costs and they don't have time to deal with the sprawl of abstractions.
Whereas when you are a one-man business, all you have is (free) time and prefer to spend as little money as possible while your business is not profitable. This means: walk away from dedicated databases, managed servers, huge aws bills, etc. You do everything "the old" way: rent a server (bare metal or cloud) and maintain it yourself.
This pattern plays out with many indie app developers as well, both now with iOS apps, and back in the day with Delphi shareware. The cost of taking a dependency on a 3rd party library turns out to be much higher for the solo developer than larger orgs. That's not to say they don't make bets, but they don't build the way that someone at a larger org would do because they have to maintain a higher degree of direct control.
Marco Arment (https://marco.org/) has elaborated a lot on this topic. You can also get a feel for the limits on the upper bounds of his developer experience. Many on HN are not willing to work at a low-level with such boring and dull technologies.
Off topic, any pgDash alternatives for MYSQL?
I know it's a bit silly, but anyone building out cloud services should look at this post and think about how easy it is to set up this. This is just slightly more complex than a static site, has a bit of heterogenity, but otherwise is a lot of known tools.
Can your stack provide all of this without breaking the bank for the people signing up? Can it be done without having to learn a bunch of bespoke systems? I think that Heroku's success in particular is totally down to managing this bespoke-ness balance (that is still kinda missing from container-based setups).
That's because their product is only slightly more complex than a static site.
I wish every project was as basic as a web server and database but it's pretty rare these days.
What you're seeing is survivorship bias where the SaaS companies that are doing well are the ones that can provide a simple, clean UI whilst adding the functionality that you need for the product to be useful.
Is it? What functionality are you thinking about that absolutely require more than that?
I find that most ideas even today can be solved with a web server + DB, while developers today like to over-engineer things and think about scaling a service for a million users while they haven't even figured out the value proposition of their product/service yet.
A lot of apps use an "AI" or "machine learning" component which usually just means a second heterogeneous internal service.
For a little bit larger companies it's pretty common to have a data analytics or data warehouse component.
In addition, a web server + database usually require some monitoring system. After you have more than a handful of servers, probably also log aggregation.
If you're just a single or handful of people working on something, you probably don't need any of that.
Once you get a little bigger (or downtime becomes expensive) those things get tacked on quickly.
I feel like the overwhelming majority of web based software projects can be catered for with exactly this setup.
If it's rare it's because everybody's drunk the Kool-aid
Having a 1-2 day outage because your architecture is not highly-available across data centres like this one just doesn't cut it. And multi-day outages are very much real because of the thundering herd effect as companies rapidly try to move between data centres exhausting capacity.
And much of the complexity in modern day architectures come from high availability and security.
i.e. they would likely be sharing the same power and network infrastructure.
So not redundant by any normal definition.
But you gotta pay more than 500$, way more.
If you're running a site like this, it's fine to be down for a bit, people will forgive you as you restore from backup (looks like a very solid foundation the site has).
on other bare metals, or on AWS, azure, ...
As long as you have database backup, git code, and build-scripts.
I run something similar (although smaller). My disaster recovery is: I have everything prepared, to go on AWS, if necessary. Would take less, than 1 hr to be back up (database size being the biggest time sink). If I wanted to minimize that time, I could have small DB replica running, so I would just have to run last day of wall files. But for my purposes everything less than a day is good enough.
And then you can take a few days, to find something cheaper, to migrate to.
At my last job we had one of those hyperfancy devops setups with all the fancy devops tools. Literally no one in the company knew how to spin up a new environment. I'm not exaggerating: no one knew how to run a dev environment and when they had to set up a new region for legal purposes it took weeks for the team. All of that was initially set up in an age of legends in the mythical past of a year and a half ago.
Theoretically it would all failover and scale to the moon and back. Emphasis on theoretically because no one seemed to understand how it all worked, so who knows how well it would behave when it failed. There was actually some major downtime in my time there which was rationalized away as "growing pains", but in in my opinion a major factor was just that no one really understood how it al worked.
If something goes wrong with a simple system then often the diagnosis and fix is simple (in this case: just deploy a new environment). If something goes wrong with a complex system then all of this is much harder.
Not saying you can't use these tools correctly or that there isn't an appropriate use for it, but it's a good case study on how the complexity can spin out of control if you're not careful in how you apply it.
He has daily backups for that reason - those can be changed to hourly or even every 15-mins if needed.
I'm surprised that the machines are that large, there must be a bunch of people using the service nowadays :) I remember the setup being even smaller in previous blog posts. But I really like it. Small, easy to reason about. Makes debugging when stuff goes wrong quite easy.
I'm a bit surprised that there's no config management listed though - seems like it really is just Fabric + a bunch of scripts. But hey, if it works, great!
* the traffic from the monitored systems comes with spikes. Looking at netdata graphs, currently the baseline is 600 requests/s, but there is a 2000 requests/s spike every minute, and 4000 requests/s spike every 10 minutes.
* want to maintain redundancy and capacity even when a load balancer is removed from DNS rotation (due to network problems, or for upgrade)
There are spare resources on the servers, especially RAM, and I could pack more things on fewer hosts. But, with Hetzner prices, why bother? :-)
Are "app" servers the haproxy ones or the "web servers", or other ones not mentioned?
If I'd run a one man show type of business, I'd love to have some kind of plan B in case I'd be incapacitated for more than half a day. THAT would be an interesting read to me.
> Any reason not to use a hosted postgres provider?
* Cost
* Schrems II
* From what I remember, both Google Cloud SQL and AWS RDS used to have mandatory maintenance windows. The fail-over was not instant, so there was some unavoidable downtime every month. This was a while ago – maybe it is different now.
For the healthcheck's integration solution (email, signal, discord....), how did you go about doing that? I would love to hear about all that. I did find a section about Signal's integration.
The Signal one took by far the most effort to get going. But, for ideological reasons, I really wanted to have it :-) Unlike most other services, Signal doesn't have public HTTP API for sending messages. Instead you have to run your own local Signal client and send messages through it. Healthchecks is using signal-cli: https://github.com/AsamK/signal-cli
This line here shows much thought has gone into "what if...." situations. Bravo!
I too have a similar setup - I have a contractual responsibility to my clients, and if anything goes wrong with my main equipment, I can resume ASAP. This includes a dedicated smartphone for hotspotting if my internet connection dies.
Personally I've had good experience with Braintree. Particularly their support has been impressively good – they take time to respond, but you can tell the support agents have deep knowledge of their system, they have access to tools to troubleshoot problems, and they don't hesitate to escalate to engineering.
Wonder why he is not using hetzner's load balancer [1]. At least cost-wise, the savings are huge (4xAX41-NVME are around 160€; LB31 - the most expensive one - is 36€)
[1] https://www.hetzner.com/cloud/load-balancer
I do use it for a purpose it probably was not designed for :) In my summerhouse, one of the circuit breakers trips every now and then (1-2 times per year) for no apparent reason, and since the fridge is connected to that breaker, i very much fear arriving at a summer house where the fridge has been unpowered for a couple of weeks.
The solution to that of course, was to setup a systemd timer on a Raspberry Pi (that also runs HomeBridge), which pings Healthchecks.io every 15 minutes. If it misses 2 pings it raises an alert.
So far i've had 2 notifications in the 2+ years it's been running, and both have been internet connection related, so monitoring the situation has apparently changed something :)
I set up a phone with Tasker, if the phone stops charging or fails to check in, it sends an alert.
Luckily, and as you say, monitoring changed the situation, as it never failed since.
Usually there's little any single person can do, but this waste of electricity is very selfish considering climate change.
You have that effect in many areas of life. A quite obvious example is working out. Of course it won't make much of a difference if you skip today's workout or abort your 40 min run today halfway cause you don't feel like it. But it's the mindset/attitude. It makes you a quitter. It enables you to do the same thing again. And again. And in the end, the cumulative damage of the change in mindset is magnitudes worse than the single event.
Ironically that is what kept me from setting up solar power. Being in Scandinavia, solar has a somewhat limited potential given that days are 7 hours long during winter, and December often has less than 20 hours of sunshine in total. My calculations for the "payback" time of the solar panels said that _MAYBE_ they would have saved something before their "expiry date" some 20 years into the future.
Considering the "energy and material waste" involved in creating solar panels, this task can be handled by energy companies much more efficiently than i ever can.
I am still considering a share in a windmill though :)
Everyone should be calculating payback on these investments every couple of years, because the calculus changes depending on incentives/tech/other factors (e.g. EV purchases, or massive cost of living changes)
First Google result: https://www.energy.gov/eere/solar/articles/busted-common-sol...
We have an annual power consumption of 6-8k kWh, with about half during winter (heat pump, yay). More energy is required during night time due to dropping temperatures and it being dark, so realistically, I could probably save about half.
At normal energy prices of €0.3/kWh, that means I would save €1200/year, and that’s without taking into account reduced electricity taxes for electric heating.
Taking those into account, all electricity used beyond 4000 kWh/year would be around €0.12/kWh. Taking that into account, I could save €480/year, still assuming I could save 4000 kWh/year out of 8000 total.
A solar panel installation _without_ batteries is about €13500, and saving €480/year means it would take 28 years to get back the investment.
I’m aware batteries can change that equation somewhat in my favor, but it’s ever harder to calculate anything usable with those as it depends if the battery can get charged during the day. In any case, a solar panel installation with batteries is around €20000.
Since I bought an electric car I've been drooling over the idea of having the car powered by the sun. I go into the office only on occasion so I don't need to charge fast either. But you need an available roof and an okay topography.
But in Scandinavia, I bet most energy goes to heating so improving insulation and adding ground thermal will probably be more efficient. The idea of a sun powered car is just so enticing, however. It's kinda sci-fi.
This is Denmark, where a large part of our power grid is based on renewables, and backup power is based on natural gas. There is no coal involved. When we import power it's from Norway/Sweden or Germany, so either renewables or nuclear. We're not at Norwegian CO2 levels (yet), but it's getting better :)
You can check for yourself here : https://app.electricitymap.org/map
> Since I bought an electric car I've been drooling over the idea of having the car powered by the sun.
Denmark has outlawed sales of new ICE cars from 2030, so at that point solar will make MUCH more sense, as i can charge my car for "free", and the time to ROI will be much shorter.
> But in Scandinavia, I bet most energy goes to heating so improving insulation and adding ground thermal will probably be more efficient.
I thought so to, but currently, in no small part due to the Ukraine situation, lots of people are replacing oil and natural gas heaters with heat pumps or central heating, and the central heating providers are installing large heat pumps as well.
As part as getting a heat pump, my house (built 1970's and renovated 2010's) has a total heat demand of roughly 13000 kWh. Assuming a heat pump with a SCOP of 3.5, that means i will need about 3700 kWh of electricity to heat it, which means that our total electricity consumption ends up around 7500 kWh, so about half the energy (in my case) being used for heating.
In my experience, anyway.
We tested it, and with our usual usage pattern of using the summerhouse every 2 weeks, the brand new fridge uses less power while powered for the 12 days between departure and arrival, than it does cooling down when powered on at arrival.
Despite being brand new, it uses 6-8 hours to get from 18C to 5C. I guess because it's brand new it is well insulated, so less cooling power is needed once it's cooled down, which translates to a weaker compressor with less power consumption.
As for the Raspberry Pi, it runs idle for 99% of the time, consuming _at most_ 1.15W. That amounts to 0.8 kWh in a month, which is about as much as a TV uses in 2.5 hours, or about a quarter the power consumption of a Sonos One Speaker sitting idle [1].
The total power consumption of the summerhouse uninhabited is ~1.8kWh/day, which includes the heat pump, fridge, internet modem and router, Raspberry Pi, Security Camera and a couple of hubs for IoT.
[1]: https://support.sonos.com/s/article/256?language=en_US
I started leaving a gallon jug of water (frozen) in mine as it helps give it enough thermal mass to even out the spikes.
My chest freezer maxes out at 180 watts TDP for instance, which is pretty anemic if you’re trying to freeze a pot of room temperature stew.
Owning a "summer" house?
Moving yourself to the summer house every two weeks probably has far more climate impact than the fridge.
Taking to a logical conclusion: you shouldn’t have a summer house at all. (That’s not a position that I hold, but one that seems more logical than “have a summer house; just empty out and unplug the fridge while you’re not there…”)
Another solution to this problem is to just empty the fridge and turn it off while not in the summer house for a few weeks.
> just empty the fridge and turn it off while not in the summer house for a few weeks
We tried that, but the fridge uses 6-8 hours to reach target temperature despite being brand new, and keeping food fresh without cooling for 10 hours is not easy.
It actually uses less power being powered on continiously for 2 weeks (12 days from departure to arrival), than it does being powered off and working overtime for 6 hours trying to cool down. It's not much less power, but still less.
I suppose you mean energy rather than power? (Cause the argument makes less sense with power.)
That's an interesting phenomenon then since it goes counter to my (rather naive) understanding of physics. There could be some anomaly with the fridges efficiency curve that causes this. Maybe it's really inefficient cooling down the contents from room temperature while being really efficient maintaining the cool temp? After all, the latter is its main purpose..
Normal power consumption of it (assuming 18C room temperature) is less than 0.4 kWh/day. That amounts to 4.8 kWh in a 12 day period. Cooling it down from 18C takes 5.6 kWh (measured).
Temperature varies of course, and the lower the temperature the lower the power consumption, and the heat pump keeps a minimum temperature of 12C. During long summer days, the indoor temperature can easily reach 26C, which will require more power, and i have not measured every possible scenario.
I'm not ruling out that actually cooling down the thing during winter will require less energy than keeping it running, but i'm fairly certain that doing the same from 26C will require more power than keeping it running.
This seems like a lot. It's 20 megajoules, so even at a COP of 1, it's enough energy to cool down 20 megajoules / (4182 J/kg/K) / 18 K = 265 kilos of water by 18 kelvin.
And a refrigerator is supposed to attain a COP better than 1, and I suspect you have less thermal mass around than 265 kilos of water.
Something is wrong-- either in measurement, or heat exchangers very badly occluded by dust, etc, or some other fault.
The fridge is in a cabinet, enclosed on 3 sides, so maybe it’s not letting enough hot air out ?
I could be wrong, but I think it was stated as using 184 kWh / year (at room temp 22C, optimal conditions), which is just over 0.5 kWh/day, and considering that the summerhouse is usually cooler than that (12-16C) when unoccupied, I would assume it used even less power.
As it is, for this entire week, the heat pump has (self) reported using 1.8 kWh/day, and according to my power company, my daily consumption has been 1.8 kWh/day. Considering that there are 5-7 IoT things plugged in, the fridge probably isn’t using much power.
Seems likely. Once it succeeds in warming its cabinet, a big share of that heat leaks back into the fridge. (And efficiency falls because of the bigger temperature difference).
* Every bit of warming that finds its way into a powered-off refrigerator is heat the refrigerator would need to take out anyways. And the heat flowing in declines as the inside warms up.
* There's standby power, etc.
* A refrigerator is expected to be more efficient removing heat when the temperature difference between inside and outside is less.
I'm not arguing that that's the case here. I'm arguing that you don't "always" expect things to be worse off letting it run. There is a threshold, when approaching the mentioned fantasy conditions, beyond which letting it run may be better.
Then it would never warm up, and at this limit the two cases are equivalent (powered and unpowered).
> when approaching the mentioned fantasy conditions, beyond which letting it run may be better.
Anywhere short of the limit, what I said holds. Anything better than infinite insulation, I don't know how to reason about.
Perfect insulation doesn't help if you open the door?
If you're saying 'well with perfect insulation it doesn't matter if it has power' - well yeah. It doesn't matter if it has power. So you aren't wasting any either leaving it on.
This continues all the way up to perfect insulation, though ultimately it's breakeven with perfect insulation.
> Perfect insulation doesn't help if you open the door?
This doesn't change the picture, but worse, it's just completely unrelated to the scenario. Who's opening the door when everyone has left for weeks?
Which seems unlikely in this case.
My chest freezer for instance turns on for less than 30 minutes a day even with people getting in and out. When I defrost it, it takes a day+ to cool down again.
The reason someone would leave the door open is to ensure no one accidentally left a jug of milk in there when they turned it off. Which would pretty much neutralize any possible gains in sheer grossness if nothing else.
I think you're a little confused. I'm a little lost at even how to minimally explain it to you. The rate at which heat leaks into the refrigerator declines as the refrigerator heats up, so this metric too improves with time.
> Which if the insulation is good, may take many weeks.
Implausible. "Many weeks"?
The best 20 cu ft refrigerators use about 375 kilowatt-hours per year according to energystar.gov. At a coefficient of performance of 10, that's an average of 400W flowing into the fridge and needing to be removed by the refrigerator.
At 20K of temperature difference, that's about 20W/degK. The contents will warm very quickly, absent phase change.
Having ice in there will slow things down until the ice melts; then it will warm very quickly. The typical numbers cited say you have 2-3 days with a full freezer without power before things have mostly thawed and become unsafe-- this is well over half of the way to room temperature in terms of energy budget (and in the best case of a full freezer where there's the most thermal mass inside).
(Specific heat capacity of water ~4200 J/(kg*k); heat of fusion of water ~334000 J/(kg*k)).
None of this changes the energy balance of the problem, though.
* On the new host, I run a Fabric task which generates a key pair and spits out the public key. The private key never leaves the server.
* I paste the public key in a peer configuration template.
* On every host that must be able to contact the new host, I run another Fabric task which updates the peer configuration from the template ("wg syncconf").
One thing to watch out is any services that bind to the Wireguard network interface. I had to make sure on reboot they start after wg-quick.
A. What if another operator takes the source code, and starts a competing commercial service?
I've seen very few (I think 1 or 2) instances of somebody attempting a commercial product based on Healthchecks open-source code. I think that's because it's just a lot of work to run the service professionally, and then even more work to find users and get people to pay for it.
B. What if a potential customer decides to self-host instead?
I do see a good amount of enthusiasts and companies self-hosting their private Healthchecks instance. I'm fine with that. For one thing, the self-hosting users are all potential future clients of the hosted service. They are already familiar and happy with the product, I just need to sell the "as a service" part.