HN Explorer

Ask HN: How do you make sure your servers are up as a single founder?

448 points by thr0waway998877 ↗ HN
I'm running a small business on AWS as a solo founder. It's just me. Yesterday I had a service interruption while I was in the London subway. Luckily, I was able to sign in to the AWS console and resolve the issue.

But it does (again) raise the question I'd rather not think about. What if something happens to me and there's another outage that I can't fix?

So - how do you make sure that your servers are up as a one person founder? Can I pay someone to monitor my AWS deploy and make sure it's healthy?

232 comments

[ 1.9 ms ] story [ 239 ms ] thread
freetonik
Heroku. I just pay for the privilege of not thinking (almost) about such issues.
dubcanada
Also services like platform.sh and others.
randlet
Heroku looks really expensive if you need e.g. a private network per client (compared to AWS VPC). Between $12k-$36k if I'm reading this correctly: https://elements.heroku.com/addons/heroku-private-spaces
malyk
Yes, if you need that.

Many many businesses can survive on a $25 a month 1x dyno and the free database tier (and free sendgrid, free newrelic, free whatever other addon) and can be pretty sure their site will basically never go down.

It's a fantastic security blanket, but...as your business grows you'll start to pay. The question then becomes "If I do all this myself on AWS (or azure or gcp) how much time am I going to be sacrificing time against building my business dealing with random infrastructure crap?" Or "at what point does it make sense to hire someone to focus on all this infrastructure and how much would that cost vs. just paying for heroku?"

jarl-ragnar
I use uptime robot http://uptimerobot.com for monitoring, they have a free plan or paid if you want faster checks.

If it's truly critical to have no down time then you probably need to build that resilience in to your architecture.

tnolet
You need monitoring. But it will not keep your servers from dying.

Using serverless, PaaS like Heroku or similar will help.

lepah
by not managing servers, use a PAAS such as heroku, it significantly reduces devops allowing you more time to focus on what matters, ie product market fit.
kerberos84
Yes there are plenty of start-ups doing this. You can also use AWS's build-in functionality to achieve this. You can write a Lambda function which checks your server status. Even better which calls your end-point for health check if you want more detailed monitoring
_tkzm
I know the situation. I haven't got to the production stage yet but I totally get it. Beside using Kubernetes, Nomad or some other scheduler, you will always have to invest your own time to resolve issues manually. You could have triggers that would invoke ansible playbooks if you don't want to handle any of the aforementioned but in the end the type of business simply requires maintenance - there is no way around that. A real human being has to be keeping an eye on the entire architecture and make sure it is running as it is supposed to.
deif
Other people are suggesting alternative platforms when you could simply have an AWS autoscaling group. If a server goes down it simply relaunches a saved image.
maxk42
It's a good solution, but it's a bitch-and-a-half to set up.
cddotdotslash
Back when I was working on everything myself, I deployed everything through AWS Lambda and API Gateway, with all my static assets on S3 and CloudFront. I had exactly zero infrastructure issues over the course of two years and never dealt with security patches, SSH'ing, etc. If I were doing billions of requests, it may not have been the most cost effective, but it helped me scale without worrying about typical devops issues. Updates, testing, rollbacks, etc were also extremely easy.
nickthemagicman
Did you ever have any problems with the function concurrency limit?
scarface74
It’s a soft limit. You can request a larger limit and get it increased within 30 minutes.
nickthemagicman
Awesome thanks! Sorry to bug but this is very interesting to me. Did you use a framework like Serverless?
scarface74
No. I’m of the belief that when you choose a platform, go all in. Given a choice between the Serverless platform and AWS’s Serverless Application Model, I would choose SAM.

I also work for a company that pays for the business support plan, they will help you to an extent with any weirdness in their own platform - but not with third party utilities.

Another advantage with using SAM, is that you can create and configure your lambda/API Gateway from the web console and then export your SAM/CloudFormation template.

All that being said, for APIs, I don’t use either. I would recommend using standard frameworks like C#/WebAPI, JS/Node, Python/Flask/Django instead and using the proxy integration libraries that AWS provides. It lets you develop/debug locally like you are accustom to and it gives you the optionality to move your APIs to Fargate (Serverless Docker) or EC2 instances without any code changes:

C#/Web API

https://aws.amazon.com/blogs/developer/deploy-an-existing-as...

Node:

https://github.com/awslabs/aws-serverless-express

Python/Flask

https://dev.to/apcelent/deploying-flask-on-aws-lambda-4k42

I’ve been told that there are similar frameworks for other languages but those are the three that I use.

If you have any other questions, feel free to email me. My address is in my profile. I’m not a consultant trying to sell anything....

nickthemagicman
Thanks so much! This is really interesting. I think acloud.guru has a similar set up and they pay basically nothing for their millions of QPS. Going to research this more for my next project.
ryanolsonx
How do you keep things fast?

Lambda functions can have cold starts that introduce latency. How do you manage that?

(From my small amount of experience - please prove me wrong.)

tabakd
It will only cold start the first time. You also need to make sure that if your lambda can reuse the existing db connection when receiving multiple multiple requests to save time.
cldellow
If you're doing a SPA, you can paper over the cold starts a bit, since the app itself will render, and it'll be background requests to load data that are impacted.

That still sucks, so then you can (hopefully) cache some things so that _some_ data begins to stream in. Or you can make it so your very high priority stuff has minimal dependencies - you can get a Lambda cold start in < 1s if your app only uses the standard library.

But still, in my experience, cold starts are a thing. If you have a high-traffic app or use Lambda warming, you decrease the # of people who experience a cold start, but at the end of the day, your p99 is going to be worse than a vanilla VM solution, because _some_ people will get cold starts. For some apps, that's OK - think line of business app where the first few pages can be served from static materials or cached materials, and you trigger the requests in the background.

scarface74
If latency is a big deal, don’t use lambda. I went from not knowing anything about Docker to having a Fargate (Serverless Docker) NodeJS app running in less than a day using a walkthrough I found.

But then again, I did already know the other fiddly bits of AWS.

ElFitz
First, caching. Can be done via API Gateway or Cloudfront.

Second, minimizing the function's code size. Both by making small functions and then optimizing them (there are plugins for the serverless framework to do that)

Third, using a language with a minimized cold start. Some can have cold start latences lower than 1 second [^1]

[1]: https://mikhail.io/serverless/coldstarts/aws/

garethmcc
I am going to add that Lambda cold starts are pretty overblown unless you really do have an application that is super sensitive to latency (i.e. you are doing stock trading or something else pretty latency crazy). Most applications can use Lambda and pretty much never worry about cold starts. Consistent traffic keeps Lambda's warm and I've seen services at their busiest suffer less than 1% coldstarts.
cddotdotslash
I setup a CloudWatch event rule to ping the function every 10 minutes, which keeps the function warm without costing much at all.
xwdv
For your case I recommend you use Poor Man’s High Availability method, an auto scaling group of size 1.
mistrial9
can you say more about this? pointers?
benevol
Interesting, I wouldn't have expected this answer. Can you elaborate?
scarface74
Autoscaling with a min/max of 1 will cause the instance to terminate after a number of failed health checks and start a new instance.
m00dy
You can try to use AWS Elasticbeanstalk. It can recover from failures automatically by spawning new nodes right behind the load balancer.
mister_hn
yes you can do. Or try to automatize as much as possible:

- add health check mechanisms

- if health check is broken => restart service

- if restart service doesn't help after X retry => redeploy previous state (if any available)

Try to use Kubernetes or Docker Swarm if possible, combined with Terraform

tachion
Restarting the service and redeploying it should be absolutely the last resort and aren't really sound advice, mainly, because you are losing the invaluable crashed state of the system, that may be vital (sometimes logs are not enough) to discover _why_ the system crashed in the first place and then delivering a fix for that particular issue. Once that's done, you incorporate this into your infrastructure automation (having which goes without saying) be it Ansible, Terraform, Kubernetes or whatever else.

Otherwise you allow the problem to persist, pile up with other issues (also fixed by restarts, I assume) and implementing automated restarts in that manner reduces not only your uptime in uncontrollable manner, but also your code/infrastructure quality, increasing your tech debt beyond the point of recovery.

Friends don't let friends fixing things by restarting them ;)

scarface74
Restarting the service and redeploying it should be absolutely the last resort and aren't really sound advice, mainly, because you are losing the invaluable crashed

I’m speaking in terms of AWS translate to your chosen infrastructure.

At the bare minimum you should have two redundant servers behind an autoscaling group with a min/max of two with health checks.

When you need to get something up now and you want to keep the crash state, configure the crash instance to be taken out the autoscaling group but not terminate and start up a new instance. You can then troubleshoot.

somesomesome
[deleted]
scarface74
If you’re a solo developer, why would you have something as complicated as k8s? I’m referring to dead simple VMs.
mister_hn
> Restarting the service and redeploying it should be absolutely the last resort and aren't really sound advice, mainly, because you are losing the invaluable crashed state of the system, that may be vital

I assume you have a separated logging mechanism, where all logs are collected, independently from the restarted service. Don't forget to log the state of your system as much as possible for post-mortem analysis

dkersten
Most of the suggestions here is suggesting ways of restarting services when they go down, which is a good start, but that doesn't actually solve the issue I hit last night...

My system integrates with an external system and what happened is this external system started sending me unexpected data, which my system wasn't able to handle, because I didn't expect it so never thought to test for it -- the issue was that I was trying to insert IDs into a uuid database field, but this new data had non-uuid IDs. Because the original IDs were always generated by me, I was able to guarantee that the data was correct, but this new data was not generated by me. Of course, sufficient defensive programming would have avoided this as this database error shouldn't have prevented other stuff from working, but my point is that mistakes get made (we're humans after all) and things do get overlooked.

The problem is, restarting my service doesn't prevent this external data from getting received again, so it would simply break again as soon as more is sent and the system would be in this endless reboot loop until a human fixes the root cause.

That's a problem that I worry about, no matter how hard I try to make my system auto-healing and resilient (I don't know of any way to fix it other than putting great care into programming defensively), but again, we're human, so something will always slip through eventually...

Some people are suggesting to out-source an on-call person. That seems to me like the only way around this particular case. (The other suggestions can still be used to reduce the amount of times this person gets paged, though)

Clubber
Defensive programming (or I call it fail-safe programming) is a must for any type of service / daemon.

I employ a healthy dose of exception trapping and logging, and I get an email whenever it happens.

People aren't perfect, but you can anticipate a lot of failures. It usually involves bad data as in your case. Each time you get bitten, change your code so it fails gracefully.

dkersten
Agreed. Be as defensive as possible, trap all exceptions (this allowed me to identify the problem and fix it very quickly, but I still had to step in and fix it) and validate absolutely everything no matter how unlikely it seems. Also go over every system and ask "what if an unexpected error happens, will it take the system down? will it prevent other requests/tasks/users from working?"
Clubber
Also, if you are running on linux, check out supervisor. I use it to keep my WebAPI's up. If it crashes, it starts it up again.
gwbas1c
Glib answer: Don't work alone!

There's two ways to think about this:

1 - Your product might actually be too complex for a single-person business. You could rotate being on call for situations like this. This means that you'd have to make sure that sales are big enough to support an additional parter or two.

2 - Perhaps you need to simplify your product? Think more critically about error handling? I don't know the details about this part of your service, but if I assume that these bad UUIDs came from HTTP POSTs, why does a series of wonky HTTP posts bring down your entire service? Typically, something like this would trigger some kind of unhandled error that's caught higher up in your web framework and returns some kind of 5xx error.

This paragraph is very C# centric, but it should translate to other languages as well: Typically, I layer my error handling. Each operation is wrapped in a general exception handler that catches EVERYTHING and has some very basic logging. (ASP.Net does this and returns a 5xx error if your code has an unhandled exception.) Furthermore, as I get closer to actual operations that can fail, I catch exceptions that I can anticipate. Finally, I have basic sanity checks for things like making sure a string is really a UUID.

Without knowing much of your service's architecture, it just sounds like you need some high-level error handling. You probably have 100s of other little weird bugs, so high level error handling needs to do the equivalent of returning a 5xx error and logging, so you can fix it when you're able to.

dkersten
My point was less about the specific issue I hit and more that 1) external circumstances that a restart won't resolve can cause failures, because 2) we're human and no matter how hard we try, even with a large team, things do slip through.

The difference with having a large team is less that all possible failure cases will get protected against (although more eyes and code review does help), but more that someone can always be available to fix it when something unexpected happens.

In my particular case, the majority of the system kept running fine. The part that failed was a streaming system which receives updates in realtime from an external system. The error actually was localised to one particular type of updates, but that type stopped working because I didn't protect defensively enough against errors in that one particular case (I do have my database queries protected against errors, but this one slipped through). This caused other systems to not get these updates, so things that relied on them stopped working. Its not that they crashed, they just never received the updates they were waiting for.

Of course the fix is to trap all exceptions, log/notify, ignore and continue, so that at least one piece of bad update doesn't affect other updates, but again, my main point was that we're human, so can't possibly protect against everything that might cause a non-recoverable (without human intervention) error.

> Finally, I have basic sanity checks for things like making sure a string is really a UUID

Yes, I did add this too after I hit this issue and its a good point: validate EVERYTHING even if you generate it and think you can assume it will be good.

> Don't work alone!

That's the real solution, but sometimes its not possible.

Thanks for your detailed response, though, its appreciated.

gwbas1c
> In my particular case, the majority of the system kept running fine. The part that failed was a streaming system which receives updates in realtime from an external system.

Is your product too complicated for a single-person business?

As a solo programmer, I can write and develop extremely complicated systems. These systems can be so complicated that I don't have time to run them, find customers, support customers, ect.

That, ultimately, is why I don't see myself running a single-person business anytime soon. I really enjoy complicated programming, and if I have to also handle ops, support, sales, ect, then what I program needs to be too simple to remain interesting.

tachion
This is exactly the reason why restarts shouldn't be ever considered a fix, as I've elaborated on a bit more in my other comment in this thread. They fix noting, but give an illusion of it.
kd5bjo
For situations like this, it's extremely useful to have a human-review queue that the automated system can drop jobs into. It keeps the rest of the system running, and lets the support staff (you) fix the problem during normal business hours.

This is how we handled errors in our video archiving system at Justin.tv, and to my knowledge we never lost a single frame that made it to the broadcast servers. The raw bits were streamed to disk as they came in, and only got removed once the VOD servers had the final version— any errors would retry a few times and then get flagged for manual processing. We did have a few close calls where something broke the whole archiving system, and the broadcast servers came dangerously close to shutting down due to full disks, though.

tootie
Always treat third-party systems like they're full of nitroglycerin. Double check all response codes, expect the unexpected, degrade gracefully when it hits the fan. You're always better off serving up a nice 500 error page than spinning forever or returning a false positive to users. And make sure you have a clear SLA with them and can escalate/mitigate/compensate when they don't fulfill it.
wstuartcl
This. Write guards like the external integration is an active malicious adversary -- because when they change api, go down, have their own issues they may as well be attacking your integration.
bjacobt
I used to run a service solo that processed data from multiple external sources and as you said you need to program defensively when dealing with input.

I handled it with a pipeline that did the following, 1. validation, 2. transform data if needed, 3. load the data. If validation failed, the data would get "quarantined" and I would get an email notification, or slack notification if urgent.

I don't generally write a lot of unit tests, but all my validation and transformation logic would have 100% coverage because things always change and you need to make sure future updates never break the system

lacker
Some of the comments are suggesting totally different technologies. Don’t do that. You can stay on AWS and achieve the reliability you need. This isn’t the sort of problem that should lead you to rebuild your whole stack.

The question you should be asking is, how can I make my service automatically recover from this problem. It depends why exactly it crashed. If a simple restart fixes the problem, there are different ways you can automate this process, like Kubernetes or just writing scripts.

I’m happy to give more detailed advice if you would like, my email is in my profile.

davecap1
Have you thought about hiring someone remote in the same or different timezone to be on-call for outages? I'm sure there are many people around that would be able to help with this. You could hire someone on a retainer who can be on-call via PagerDuty or something.
dhimes
I outsource to Tummy.com They've been terrific, and I don't have to worry about anything I don't want to.
hellcow
The only way to achieve high availability is to have redundancy of all things.

Random things will go wrong that you can't predict. Boxes will die suddenly and without reason, even after months of working fine without changes, and always at the worst possible moment. Your system needs to be built to withstand that.

I'll take the opposite approach of everyone here and recommend against serverless, kubernetes, and Heroku/PAAS.

You are a solo founder. You should understand your infra from the ground up (note: not understand an API, or a config syntax, but how the underlying systems actually work in great detail). It needs to be simple conceptually for you to do that. If anything goes wrong, you need to be able to identify the cause and fix it quickly.

I've gone through this first-hand and know all the trade-offs. If you'd like, I'm happy to discuss architecture decisions on a call. Email is in my profile.

scarface74
No you don’t need to understand your infra from the ground up - especially as a solo founder. You should offload as much of the grunt work as you can afford to so you can concentrate on your business domain.

If something “goes wrong” or you don’t understand how to implement something with managed services, support is just a ticket and a live chat/phone call away. I can speak from personal experience that AWS business support is great even when there isn’t a problem and you just want an “easy button” for someone to tell you what’s wrong with your configuration.

CoolGuySteve
It depends on the service I think. I've had ECS errors that took AWS support days to figure out (turns out some permission quota thing was overriding some ECS thing).

All in all, I think maybe I might have to find some other batch processing system.

scarface74
If you are using regular ECS with EC2 - as opposed to Fargate, it just provisions regular EC2 instance with an agent already installed. You can ssh/rdp into the instance and troubleshoot it.

But yeah I did have a doozy of an issue with ECS but it was completely my fault. I create a cross account policy for ECR but left out the account that actually contained the registry. Then my containers were in a private subnet without any access to the internet (by design they were behind a load balancer) but they couldn’t get to the ECR endpoint. I just had to either assign a public IP address or use a private link.

Support helped me with both.

bszupnick
As a solo bootstrapping technical founder, I am in LOVE with Heroku.

My product is single-tenant, which can be tough infrastructure-wise since each app/customer needs a cluster of servers and services (Postgres and RabbitMQ). The Heroku pipelines enables me to have a testing app, staging app, and when I want to push staging to production, I can push the code to all my single-tenant production apps with the push of a button.

In theory I could do this with Bitbucket or Gitlab CI/CD pipelines, but this enabled me to focus on app development instead of devops.

That's just my preference, of course.

fulafel
IMO a single founder doesn't need "high availability" as the term is usually used (five nines stuff with high operational complexity). If you can get it cheaply by using eg Heroku, it's a nice bonus.
Scarbutt
The only way to achieve high availability is to have redundancy of all things.

It depends on your availability requirements, for me, it's more important and simpler to be able to deploy again(with data) under hour than dealing with HA.

samvher
It really depends on the failure mode and the cost of failure. As mentioned by others you can encounter issues in external services which you have no control over and the best you can do in that case is fail gracefully until you're able to deal with the issue. If it's easy to detect failure, and a restart fixes the problem, it can be quite straightforward to set up some monitoring scripts that take care of this for you, and even if it's more complicated than a restart some monitoring can at least notify you by email or SMS. Keeping your tech simple and/or having high test coverage or formal verification can reduce your error rate. Similarly you can introduce fault tolerance into the system with something like Erlang's OTP or monitored containers in an orchestrator (K8s, Docker Swarm, some cloud solution). If failures are expensive you might want to take on staff to deal with them, if the cost is low you might just want to accept occasional downtime (though you'll want to think about how you report that to your users).
PopeDotNinja
+1 for Erlang. Learning how to write OTP apps in Erlang taught me so much about building reliable systems.
nlg
I agree in general with the responses encouraging better usage of managed platforms. I've run a SaaS app for a couple of years using a combination of AWS Elasticbeanstalk (Flask and Django) and AWS Lambda. Server resource related downtime has been minimal and recovery is quick/automated. Even hosting on Lambda you can run into issues without layers of redundancy (Lambda may be fine but a Route 53 outage would prevent you from hitting that endpoint if you're using that for DNS).

Before thinking about handing over management of the deployment, I would encourage you to think about what the root cause of the outage is and whether something in the app will create that situation again. I invested in setting up DataDog monitoring for all hosts with alerts on key resource metrics that were causing issues (CPU was biggest issue for me).

The other thing that's worked well for me is just keeping things simple. As a solo founder, time spent with customers is more valuable than time spent on infrastructure (assuming all is running well). It's a little dated, but I still think this is a good path to follow as you're building your customer base. A simple stack will let you spend more time learning how your product can help your customers best.

http://highscalability.com/blog/2016/1/11/a-beginners-guide-...

kevinyun
We are considering Datadog, and nothing else seems to compare to them, but they seem extremely expensive. As a small startup/solo founder, did your implementation justify costs?
gamache
$15 per server per month isn't that extreme. I've used Datadog in production for 3+ years (starting from a headcount of 10) and can honestly say they've been worth every penny. Observability is irreplacable.
heliodor
Plenty of people are happy using the industry-standard open-source tools:

- metrics system (Prometheus, InfluxDB, Graphite, etc.)

- dashboards (Grafana)

- alerts (both the metrics system and Grafana can handle this)

If you're not a big-data company, you can self-host instances of these products reliably.

Disclosure: My startup, https://HostedMetrics.com, provides turnkey hosted versions of these software packages.

I'd be happy to explain the options that are out there and provide any advice you need. Get in touch: contact info is in my profile.

byteshock
Your product doesn’t even come close to what datadog offers....
heliodor
Datadog has ~9000 customers. I imagine the respective open-source tools have many more customers than that, which says something about the perceived value of the two approaches. Also, Datadog wraps many tools into one whereas the open-source solutions have singular purposes. It's an apples to oranges comparison.
nlg
Yes, it's been an incredibly valuable tool for me. I get the most out of the host monitoring and the logging / alerting. The APM is nice, but I'm not using it nearly as much as the other pieces.

Just getting logging centralized alone has saved me tons of time, which is in turn more time spent on the product. I've been able to use the log parsing to setup metrics that tell me when an outside integration is acting up and isolate which paths. Take a day to really learn how their logs work and you'll be able to generate metrics / advanced event alerting in no time.

I was hesitant to pay the premium, but the peace of mind has been worth it. You can piece the same thing together with open source tooling. But then you've got another thing to manage.

byteshock
Datadog is the best all around tool with APM and log monitoring built in. I highly recommend it as it will be substantially cheaper than paying for separate monitoring, apm, logs, etc. The support is also a lot more receptive towards small business owners unlike other platforms such as New Relic.

Also if you send custom stats to datadog such as login activity, you can use their Anomaly detection to find suspicious behavior.

Disclaimer: we use datadog at my company and have tried all the other popular options. Hands down datadog is the most feature rich and user friendly.

cpursley
Use heroku as long as it's cost affective. Every time I've moved from heroku to another platform for "cost savings" I always end up spending much more time that I'd planned just maintaining it.
peterk5
I build my projects on Google App Engine and it has been stable and reliable without much administration. The platform is not without its challenges, especially with the Gen 2 rollout, but no issues related to administration/interruption. PaaS could be a good place to explore...
markyc
seconding this, if you don't need lots of resources it makes sense. I pair it with Amazon cloudfront and so far, almost one year in with zero problems. By far the biggest win for me is the peace of mind
nathan_f77
I'm working on FormAPI [1] as a solo founder. I started on Heroku, but Heroku was a bit unreliable and I had some random outages that I couldn't predict or control. (This was even while using dynos in their professional tier.)

I also had a lot of free AWS credits, so I migrated to AWS. I didn't want to write all my terraform templates from scratch, so I spent a lot of time looking for something that already existed, and I found Convox [2].

Convox provides an open source PaaS [3] that you can install into your own AWS account, and it works amazing well. They use a lot of AWS services instead of re-inventing the wheel (CloudFormation, ECS, Fargate, EC2, S3.) It also helps you provision any resources (S3 buckets, RDS, ElastiCache), and everything is set up with production-ready defaults.

I've been able to achieve 100% uptime for over 12 months, and I barely need to think about my infrastructure. There's even been a few failed deployments where I needed to manually go into CloudFormation and roll something back (which were totally my fault), but ECS keeps the old version running without any downtime. Convox is also rolling out support for EKS, so I'm planning to switch from ECS to Kubernetes in the near future (and Convox should make that completely painless, since they handle everything behind the scenes.)

[1] https://formapi.io

[2] https://convox.com

[3] https://github.com/convox/rack

asdkhadsj
Convox looks nice! My only issue is that it seems tied to AWS?

It would be nice to have a stateless tool abstract Terraform a bit, to let you use more providers as a basic PaaS.

tachion
You start by identifying the reasons behind why your application/service may fail and then design and implement the infrastructure for it, that can withstand certain failures for a cost you can bear. If a failure of a piece of infrastructure costs you £1 per day, you might be OK with paying £5/day for the infrastructure to handle such failure. But would be be OK with paying £50 for the same thing?

It's all the matter of defining requirements, then solutions and tradeoffs of those solutions and then implementing it with best practices in mind (automation, testing, monitoring, backups, etc.).

Hit me up if you want to discuss it over a pint! :)

foxhop
You have identified a single point of failure (yourself), you either need to accept the risk or hire a person on retainer.

I'm in the same boat with my solo founder projects (links in profile).

ransom1538
1. Stay on AWS only.

2. Pay for a Business Support plan. https://aws.amazon.com/premiumsupport/pricing/

3. Call business support about something "how do I restart my server" - so you know how to file a ticket, get a feel for how quick the response is and how it works.

Do not over think this. EG: terraform templates

asdkhadsj
re: terraform, that's only part of the picture right? What do you recommend for provisioning? I assume Terraform + Packer? I looked into these a while back and they seemed good.

My only concern was that my target was very low-cost setups, and I wanted something like Packer but let me provision multiple images onto a single machine. Eg, if I just used Packer, as far as I could tell I would have to have 1 machine per image. It sounds odd, but I didn't want to pay $5*Services, especially when the number of users and load was very small. Being able to deploy in a PaaS fashion to something like Docker on the machine seemed best.

But then I was looking at Terraform + Packer + DockerSomething, and things went back to feeling less simple.

scarface74
Don’t do Terraform. If you are already on AWS. Use CloudFormation. You are paying for the business support plan. They aren’t going to support your Terraform deployment.

I haven’t used Packer, but could you use CodeBuild/CodeDeploy/CodePipeline? Again if you’re on AWS you might as well take advantage of their support.

You can deploy Docker images using CloudFormation either to Fargate and not have to worry about servers or to EC2 instances (? I haven’t tried).

jasonkester
I build my stuff on top of a stack that hardly ever goes down.

All my SaaS products run on a Windows server, with SQL Server as a database and ASP.NET on IIS running the public sites. You can probably come up with a lot of uncharitable things to say about those technologies, but "flimsy" and "fragile" likely aren't in the list.

As a result, when things go seriously wrong, the application pool will recycle itself and the site will spring back to life a few seconds later. Actual "downtime", of the sort that I learn about before it has fixed itself might happen maybe once ever couple years. At least, I seem to remember it having happened at least once or twice in the last 15 years of running this way.

There's a Staging box in the cage, spun up and ready to go at a moment's notice, in case that ever changes. But thus far it has led a very lonely life.

scarface74
I agree. I’ve spent close to 20 years hosting on IIS and have never had downtown attributable to it - especially since the app pool will recycle. Most of the issues I’ve seen were caused by people who didn’t know how to use EF correctly or badly tuned databases/queries.

Even now, the only “Linux deployments” I do are either with Lambda or Fargate (Serverless Docker). I just don’t like managing servers. These days even my EC2 instances are disposable.

rasz
Code Red was 18 years ago, Blaster 16, Witty worm 15. All ate ISS servers for breakfast in highly automated fashion. 20 years of IIS with no downtown indeed.
scarface74
I know about the Unicode hack - where you could basically encode DOS commands in the url window and it would run on the server. We got lucky. It renamed all pages using the default names. We didn’t have any.
gscott
Webpages on the server should be read-only... just in case!
jedieaston
I’m curious, how much did you have to pay for your licenses? If you are a single founder with a new startup, it seems like you’d need a bunch of money to be able to use Microsoft in production, unless you can use something like BizSpark (if that’s still a thing?).
juliangoldsmith
Last I knew, Visual Studio Professional was about $500 for a single user. SQL Server can get expensive depending on what you're hosting, but IIS and ASP.NET don't cost anything. All you'd need is a Windows box for those.
bdcravens
SQL Server on RDS includes the SQL Server license and is very cost effective for what you get. (Backups, point in time restore, etc). They also offer EC2 with SQL Server but I’d rather pay the slight premium for the additional management.
tbyehl
BizSpark is not still a thing. The replacement program, Microsoft for Startups, requires being associated with a partnered "startup-enabling organization."
[deleted]
(comment deleted)
jasonkester
Not much. Maybe a few grand every few years when the hardware needs upgrading. Not really enough to notice in the big picture.
thrower123
A standard sized Azure App Service runs about a $75 a month. That will let you host about as many ASP.NET sites on it as you want - and there's no VM that you have to manage. Scaling up or out is pretty painless if you need to

A SQL database on Azure runs about $5-$15 a month, depending on how you configure it.

Once you throw in SSL certs and DNS management, you're probably looking at about $100 a month.

EastSmith
For SQL database the lowest I could find right now was costing around $600. Can you please share a link for the $5-15 price range?

https://azure.microsoft.com/en-us/pricing/details/sql-databa...

thrower123
You'll want to switch to the Single Database and change the pricing model from vCore to DTU. Then you can pick Basic or Standard and adjust the size and compute resources.

Basic will do quite a lot; I use one to load test our application and haven't hit limits.

gscott
I use Leaseweb and they include a line item for the license price. It is all very affordable. I pay about $65 a month for a Windows 2016 server that I run IIS and SQL Server on for all of my projects I have going on. Very fast, affordable, etc. My SQL Server came from MS Bizspark for free (MSDN).
twooclock
Contabo VM with win server 2016 data center (includes sql sever 2016) costs me 16€/month.
robocat
If using Microsoft be very wary of the cost of understanding the licencing - especially for the "free" options.

Many years ago we used BizSpark, and we wasted far too many days and dollars trying to understand and use the licencing (and using the abysmal web interface that was mandatory). We couldn't end up jumping through the right hoops to use products in production, even though were target audience for the products when we started our business. The licences had very broad exceptions that gave Microsoft the right to pull the plug at any time.

I'm sure things are different now, but I'm also sure they are the same.