> Curious to know your thoughts on how you manage your infrastructure.
What I quite like about your repo:
- there is a separate API and background job instance
- there is a separate web image, to not always couple front end deployments to back end
- there are specialized data stores like Redis (or maybe RabbitMQ or MinIO in a different type of project)
- Dozzle seems nice https://dozzle.dev/ (I use Portainer mostly, but seems useful)
What I think works quite nicely in general:
- starting out with a monolithic back end but making it modular with feature flags (e.g. FEATURE_REPORTS, FEATURE_EMAILS, FEATURE_API), so that you can deploy vastly different types of workloads in separate containers BUT not duplicate your data model and don't need to extract shared code libraries (yet) and if you ever need to split the codebase into multiple separate ones, then it won't be *too* hard to do that
- having a clear API (RESTful or otherwise) as the contract between a separate back end and front end deployment, so that even if your SPA technology gets deprecated (AngularJS, anyone?) then you can migrate to something, unlike when doing SSR and everything being coupled
- the same applies to NOT having the same container build process have both the front end and back end build (I've seen a Java project install a specific Node version through Maven and then the build dragging on cause Maven ends up processing thousands of files as a part of the build)
- using the right tool for the job: many might create full text search, key-value storage, message queues, JSON document storage, even blob storage all with PostgreSQL and that might be okay; others will go for separate instances of ElasticSearch, Redis, RabbitMQ, something S3 compatible and so on, probably a tradeoff between using well known libraries and tools vs building everything yourself against a single DB instance
- in my experience, many projects out there are served perfectly fine by a single server so Docker Compose feels like the logical tool to start out with, if multiple instances indeed become necessary, there is always Docker Swarm (yes, still works, very simple), Hashicorp Nomad or K3s or one of the other more manageable Kubernetes distros
- self-hosted (or self-hostable) software in general is pretty cool and gives you a bunch of freedom, though using managed cloud services will also be pleasant for many, more expensive upfront but less so in regards to your own time spent managing the stack; the former also lends itself nicely to being able to launch a local dev environment with the full stack, which feels like a superpower (being able to really test out breaking migrations, look at what happens with the whole stack etc.)
- having some APM and tracing is nice, something like Apache Skywalking was pretty simple to setup, though there are more advanced options out there (e.g. cloud version of Sentry, because good luck running that locally)
- having some uptime monitoring is also very nice, something like Uptime Kuma is just very pleasant to use
- heck, if you really wanted to, you could even self-host a mail server: https://github.com/docker-mailserver/docker-mailserver (though that can be viewed as a hobbyist thing), or have MailCatcher / Inbucket or something for development locally
I'm a big fan of the modular monolith pattern, but haven't used feature flags for the purpose you're describing. Do you use any specific tools or frameworks for that? I'd also imagine there would be calls between features from within the same codebase, do those become network calls? And how does this interact with your Docker Compose/single server recommendation?
> Do you use any specific tools or frameworks for that?
You don't need to, you can just enable/disable certain features during app startup, based on what's in the environment variables/configuration, though many frameworks have built in functionality for something like that, for example: https://www.baeldung.com/spring-conditional-annotations
If I wanted to allow toggling access to the API, then I'd have an environment variable like FEATURE_API and during startup would check for it and, if not set with a value of "true", then just not call the code that initializes the corresponding functionality.
It's really nice when frameworks/libraries make this obvious, like https://www.dropwizard.io/en/stable/getting-started.html#reg... but it might get harder with some of the "convention over configuration" based ones, where you have to fight against the defaults.
> I'd also imagine there would be calls between features from within the same codebase, do those become network calls?
It depends on how you architect things!
There's nothing preventing you from using the service layer pattern for grouping logic, and accessing multiple services in each of your features as needed, and poking the different bits of your data model (assuming it's all the same DB).
If you are at the point where you need more than the same shared instance of a DB, then you'd probably need a message queue of some sort in the middle, RabbitMQ is really nice in that regard. Though at that point you're probably leaning more in the direction of things like eventual consistency and giving up using foreign keys as well.
> And how does this interact with your Docker Compose/single server recommendation?
Pretty nicely, in my experience!
When developing things locally, you can enable all of the needed FEATURE_* flags on your laptop, then it's more like a true monolith then.
Want to deploy it all on a single server when the scale is not too big? Do the same with Docker Compose, or maybe have separate containers on the same node, each with one of the features on, so the logs are more clean and the resource usage per feature is more obvious, and the impact of one feature misbehaving is more limited.
The scale is getting bigger? Docker Swarm will let you scale out horizontally (or Nomad/K8s, maybe with K3s) and you can just move some of those containers to separate nodes, or have multiple ones running in parallel, assuming the workload is parallelizable (serving user API requests, vs some centralized sequential process).
Funny I drew the same conclusion. Previously a cloud architect at Microsoft, now I don't use Azure anymore for the project I am working on right now.
Rather, I have decided to opt for Supabase instead. Probably over the long time it may cause issues for my startup - but even more realistically my startup is going to fail and my increased developer velocity by using simple tooling like this will allow me to figure out why my idea doesn't work in in a shorter amount of time, so I can go on to my next pursuit.
To be honest I think even using docker is overengineering.
Who is going to get a new job without k8s on their resume. :)
Seriously, I think a lot of people do things the hard way to learn large scale infrastructure. Another common reason is 'things will be much easier when we scale to a massive number of clients', or we can dynamically scale up on demand.
These are all valid to the people building this, just not as much to founders or professional CTOs.
Just take a look at the level of complexity in home lab subreddits!
I don’t quite get if people do it for interest, for love of the tech, or if they are technocratic and believe in levelling up their skill to get k8s on their CV like you say.
K8s is painful to get started, and painful to learn. But once you have it up you can just keep adding stuff to it.
I run a k8s cluster at home. Part of it yes, is to apply my existing skills and keep them fresh. But part of it is that kubernetes can be easier long term.
Ive got magical hard drive storage with rook ceph. I can yoink a hard drive out of my servers and nothing happens to my workloads.
I can do maintenance on one of the servers with 0 down time.
All of my config for what I have deployed is in git.
I manage VMS and kubernetes at work, and im not going to pretend that kubernetes isnt complex, but it's complex up front instead of down the road. VMs run into complexity when things change. I'm sure you can make VMS good but then why not use something like kubernetes, you will have to reinvent a lot of the stuff that's already in kubernetes.
It's a hammer for sure and not everything is a nail, but it can be really powerful and useful even for home labs.
What you also can do is starting with just a single node, incredibly easy to install with e.g. https://k3s.io/. You still have to invest the upfront effort to understand how it works but you can already reap a lot of benefits with a lot less complexity.
Kubernetes does not force you into the distributed systems hell, you can go that route later, or never.
Kubernetes/k3s on a single node turns what could have been immutable 1-step upgrades into multi-step mutable upgrades, since kubernetes's software itself and all the management components you need are a mutable layer on top of the operating system.
a) It doesn't have to be mutable. You can easily setup k3s on a single node, install the apps and bake an AMI or equivalent. And using something like ArgoCD or GitOps will ensure that your k8s stack is in sync with a tracked and managed Git repository.
b) In what world is upgrading your entire platform ever a single step. Even for a basic Python app you still have Python itself plus dependencies. And then of course whatever front end web server you're using.
I don't run k8s at home, but I have worked in k8s-heavy environments and studied it deeply. This is the accurate, nuanced take.
Few but not no people will ever run into problems at the kind of scale k8s operates at. Plus, learning how it "expects" the programs running inside its Pods to behave is kind of like learning how Django or Rails "expect" a web app to work - it's a more complicated style than just writing your own totally custom, hermetically-sealed Python apps for your personal use, sure, but it also comes with a slew of benefits in case you ever do hit that level of scale and want to move over.
Or, maybe you look over the app you're writing and say "Fat chance." In which case you can justify e.g. not making everything an API endpoint, keeping a ton of state mucking about, etc. But I still feel that's an improvement over not even realizing the questions are being asked.
Is that really true anymore? Even self hosting k8s these days (e.g with rke/rke2) is a single yaml file and one command to deploy an entire cluster.. Maybe back when we all used kubespray and networking was more complicated (to the user at least) etc.. But today? I don't think so.
Using a hosted offering is even easier, literally a couple of clicks, a ./gcloud-cli or terraform apply -- again not very hard and all the cloud providers provide you with example code you just need to plug some machine sizes etc into..
Dev setup? Install orbstack and click 'kubernetes' and you're done, your IDE (likely) will automagically pick up your kubeconfig and you can go right ahead creating services, deployments, jobs, whatevers...
I’m sure there are countless other benefits. But how many layers of abstraction, services and things that need configuring are their compared to basic RAID to get support for magical hard disks that can be yoinked without affecting workloads?
Aren't disks so large those days that losing a disk almost means you will lose a second disk during resilvering unless that by "basic raid" you're doing not-basic-raid things such as btrfs raid1c3?
> Compared to basic RAID to get support for magical hard disks that can be yoinked without affecting workloads?
These things aren't mutually exclusive though. I've spent the last few years working with kubernetes at work and running a 'simple'(but with tons of containers and weird edge cases / uses) unraid server at home for all of my needs. At some point I flipped over from 'jeez kubernetes is just too much, almost nobody should ever use this' to 'wow I have to migrate 99% of my home services to a cluster, this is driving me nuts.' I haven't quite gotten around to that migration, but I do think that k8s cluster for services / temporary storage / parallel jobs and separate unraid box that runs NFS (and doesn't do much else) is going to be a great setup for a home lab.
> But once you have it up you can just keep adding stuff to it.
I dunno why, but the k8s in my workplace keeps breaking in painful ways. It also has an endless supply of breaking points that makes life painful for anybody that depends on what runs in it, but aren't detected by the people that manage it.
Honestly, that second part is an exclusivity there, but I have never seen people "just keeping adding things to it" on practice.
It depends on how well you know k8s and what your stack is. Rancher is an extra complex version of k8s. Longhorn is pretty fragile in my experience, so is canal. But chillum and eks don't really have the same reliability issues in my experience.
It's because it is complex. And in the long run, things become simpler. The only difficulty is the initial setup and once you are past that, the overall maintenance workload just becomes easier compared to a single VM setup
You can even think of it as paying insurance premiums upfront. You get to "make a claim" if the requirements do grow into the sort of need that suit such a cluster/complex setup.
But, on the same insurance theme; I am not sure paying 10K a year to insure my 5K car makes a lot of sense, because, in the long run, I might write my car off.
K8s is painful to manage. It's a lot less painful than getting paged in the middle of the night because your server is down - And much much less than realizing that you've been down for an entire day and didn't notice. (K8s isn't even a complete solution to these problems! Just one part of a complete ~balanced breakfast~ production stack)
You don't need k8s for all of that, but there's not a simpler solution than k8s that handles as much.
Excuse my harshness but people doing it needlessly is just unprofessional waste and abuse.
Some people seem to have no concern with the needs and timetables of the would be customers but instead burn through cash building fancy nonsense.
It's like going in to a car mechanic for tires and then finding out it took 3 weeks because the guy wanted to put on low rider hydraulics and spinner hubcaps for his personal enrichment.
The worst part is it's inherently ambiguous to the next people. They don't know if the reason something is there is because it's needed or because it's just shiny bling.
I am certainly not saying everything you say is not all true. My comment is dark humour. I really like your last point. Years ago I replaced a huge hadoop cluster data processing job with a single app on one machine with a few CPUs, that reduced a job that took over 8 hours to 20 minutes. What is even dumber is, it was just a python script and gnu parallel, which used to be perl.
…but if the bosses at competing mechanic shops hire based on quality of low riders a mechanic can install, of course they'll practice on the paying customers.
I quit working about 1.5 years ago. I think I still love computers while I simultaneously hate "the web". Don't get me wrong, to my amazement people have called me the best web developer they've ever met and I routinely get put on web like things at every company I go to - hardware, logistics, finance, I've been trying to run away from it but it keeps finding me and I think I hate it.
I've got this allergic reaction to bullshit and fetishize successful products and customer satisfaction. I think we've both changed; I'm different than I was 20 years ago and so is web development.
Tight applications with minimal tools that can be pivoted and changed swiftly which require competence and finesse to administer where you don't create developer debt, these are out of fashion.
All profitable hacker spaces professionalize as romantic magic becomes a liability.
I'm a middle aged divorced man, not divorced from a person, but from a profession and I've been trying to date around with new loves.
> I think a lot of people do things the hard way to learn large scale infrastructure
Having seen some of these half-rolled, first-time-understood k8s deployments, and the multi-year projects to unravel the mess that was created, overflowing with anti-patterns and other incorrect ways of doing things, I think I would prefer a narrower scope of true experienced professionals (or at least some experienced pros that can help guide the ship for their mentees) working on and designing k8s infra.
And for those that don't need it (the vast majority of startups, small businesses, regular-sized businesses, etc), just stick to the easier-to-use paradigms out there.
Nubank, the Brazilian bank unicorn, described their approach as “if this works, it’s because we reached massive scale quickly” (paraphrased) and started with an architecture that would support that from the beginning. They were very happy with their choices and have blogged about them in detail.
This is a case where “things will be much easier when we scale to a massive number of clients” turned out to be true.
In my experience, having done it both ways, first on VM's, then on lots of fully or mostly managed services, I generally prefer the latter because systems tend to be a lot more "self-healing" - because they're someone elses responsibility. This has had a dramatic effect on improving my sanity and sleeping well at night. I only wish I could migrate to an even more fully managed stack that's more reliable and still less work. The cases where I haven't been able to are either too expensive or would be too difficult to migrate.
Everyone is building like they are the next Facebook or Google. To be honest, if you get to that point, you will have the money to rebuild the environment. But, a startup should go with simple. I miss the days when RAILS was king just for this reason.
The added complexity is overkill. Just keep it simple. Simple to deploy, simple to maintain, simple to test, etc. Sounds silly, but in the long run, it works.
I agree that we are overthinking about infrastructure. Boring stack like traditional RDMS, single server with regular backup, few bash script for deployment is fine for normal startup that targets to non-tech customer. They will serve you well at least one or two years, then you will know what should be improve.
One of the big surprise is database like PostgreSQL can handle like 100tps very well with cheap hardware cost. That mean you can handle up to 86 millions transaction per day.
I dunno. I've seen a _lot_ of business ideas fail, which could have much less expensively failed using PHP and MySQL on shared cPanel hosting than they did using AWS/Azure/GCP.
Yeah, that won't scale to a million QPS, or even 10 QPS. But way more businesses fail because they never achieve 100 Queries Per Day, instead of failing because they fell over at 10 or 1,000 or 1,000,000 QPS.
I mean, hell, Twitter (back in the day) was famous for The Fail Whale.
Getting enough traffic is harder and more important than your "web scale architecture" for your startup. Making actual cash money off your traffic is harder and more important than your "web scale architecture" (ideally by selling them something they want, but making cash money through advertising or by impressing VCs with stories of growth and future value counts too).
There is precisely _zero_ chance that if you ever get within 2 or 3 orders of magnitude of "a million QPS" - that the code you and your cofounder wrote won't have been completely thrown away and rewritten by the 20 or 100 person engineering department that is now supporting your "1000 QPS" business.
I think this goes for any technology group with any stage of company. I work in networking and genuinely of the product I sell, my customers only need a small amount of core functionality and default settings - the rest is “bells and whistles”.
But still, no matter what, the odd customer demands they need all these complexities turned on for no discernible reason.
IMO it’s a far better approach with any platform to deploy the minimum and turn things on if you need to as you develop.
Incidentally, I’ve been exposed to “traditional” cloud platforms (Azure, GCP, AWS) through work and tried a few times to use them for personal projects in recent years and get bewildered by the number of toggles in the interface and strange (to me) paradigms. I recently tried Cloudflare Workers as a test of an idea and was surprised how simple it was.
I thought the same thing until recently. Apparently there's a "Docker Swarm version 2" around, and it was the original (version 1) Docker Swarm that was deprecated:
Yes, swarm is not deprecated. I haven't used it myself yet, but I read elsewhere that swarm offers an easy way to manage secrets with containers. Some people run their 1 container in a swarm cluster with 1 node just for this feature. I see it's even officially suggested as a Note in the doc:
> Docker secrets are only available to swarm services, not to standalone containers. To use this feature, *consider adapting your container to run as a service. Stateful containers can typically run with a scale of 1 without changing the container code.*
Basically doing this for a small startup - there are some complexities around autoscaling task queues with gpus and whatnot, but the heart of it is on a single VM (nginx, webapp, postgres, redis). We're b2b, so there's very little traffic anyway.
The additional benefit is devs can run all the same stuff on a Linux laptop (or Linux VM on some other platform) - and everyone can have their own VM in the cloud if they like to demo or test stuff using all the same setup. Bootstrapping a new system is checking in their ssh key and running a shell script.
Easy to debug, not complex or expensive, and we could vertically scale it all quite a ways before needing to scale horizontally. It's not for everyone, but seed stage and earlier - totally appropriate imo.
True, I use it mainly for a few convenience things - holding ephemeral monitoring data, distributed locks, redis streams for some pub/sub stuff, sorted sets can be handy - things I could do in Postgres, but are a bit simpler in Redis.
I like this but one of the issues with this approach is if no Docker images like traditional configuration management tool, you are going for a world of pain.
Docker and Docker images have tons of best practices already defined for plenty of use cases. If it's already containerized; then, jumping to any orchestrator that supports OCI images is more about adjusting the business to a new set of operations.
Like everything, it's context dependent, but wowzers my life has improved so much since I got on board the Flatcar or Bottlerocket train of immutable OS. Flatcar (née CoreOS) does ship with docker but is still mostly a general purpose OS but Bottlerocket is about as "cloud native" as it comes, shipping with kubelet and even the host processes run in containers. For my purposes (being a k8s fanboy) that's just perfect since it's one less bootstrapping step I need to take on my own
Both are Apache 2 and the Flatcar folks are excellent to work with
Sure, but again, complexity - stuff people have to learn/maintain/upgrade, etc. ymmv
Running and configuring VMs isn't hard to do correctly, it just takes discipline to never "hack it in the moment" - or if you do, can that change in your config system.
> it just takes discipline to never "hack it in the moment" - or if you do, can that change in your config system.
Yup, and I'm glad your experience has been different from mine but mine has been that tired and stressed people are anything but disciplined, so nipping a few "I'll just apt-get ..." in the bud goes a long way. So does Reverse Uptime (or its friend, Chaos Engineering)
I have a custom deployment system which idempotently configures an Ubuntu LTS VM. All the config templates are checked into source control. I don't configure anything by hand - it's either handled in this thing or via a small user-data script run at provisioning time.
It's one level of indirection away from "check in a public key" in that the user can rotate their own keys without needing git churn
Also, and I recognize this is departing quite a bit from what you were describing, ssh key leases are absolutely awesome because it addresses the offboarding scenario much better than having to reconcile evicting those same keys: https://github.com/hashicorp/vault/blob/v1.12.11/website/con... and while digging up that link I also discovered that Vault will allegedly do single-use passwords, too <https://github.com/hashicorp/vault/blob/v1.12.11/website/con...>, but since I am firmly in the "PasswordLogin no" camp, caveat emptor with that one
if you take the time to understand k8s and have a straightforward k8s deployment, these things aren't really a problem - and you don't have to do the custom sysadmin timesinks that need to go into the "simple" suggestion. What is suggested here is "easy". But it is not simple: it proliferates custom work.
I have had great success with a very simple kube deployment:
- GKE (EKS works well but requires adding an autoscaler tool)
- Grafana + Loki + Prometheus for logs + metrics
- cert-manager for SSL
- nginx-ingress for routing
- external-dns for autosetup DNS
I manage these with helm. I might, one day, get around to using the Prometheus Operator thing, but it doesn't seem to do anything for me except add a layer of hassle.
New deployments of my software roll out nicely. If I need to scale, cut a branch for testing, I roll into a new namespace easily, with TLS autosetup, DNS autosetup, logging to GCP bucket... no problem.
I've done the "roll out an easy node and run" thing before, and I regret it, badly, because the back half of the project was wrangling all these stupid little operational things that are a helm install away on k8s.
So if you're doing a startup: roll out a nice simple k8s deployment, don't muck it up with controllers, operators, service meshes, auto cicds, gitops, etc. *KISS*.
If you're trying to spin a number of small products: just use the same cluster with different DNS.
(note: if this seems particularly appealing to you, reach out, I'm happy to talk. This is a very straightforward toolset that has lasted me years and years, and I don't anticipate having to change it much for a while)
> I manage these with helm. I might, one day, get around to using the Prometheus Operator thing, but it doesn't seem to do anything for me except add a layer of hassle.
One big advantage of the operator is that its custom resources are practically kind of standard by now. This means helm charts for a lot of software ship those and integrating that piece of software into your monitoring is a matter of setting a few flags to true.
The go to solution for a k8s monitoring setup is https://github.com/prometheus-community/helm-charts/tree/mai...
I just hosted a site on Elastic beanstalk. Didn’t need to really do anything honestly. Upload a zip file with python code that runs locally really well. Database is on RDS. It has and continues to work well for 5+ years and lots of productivity.
That's not a joke. Go is a fast compiled language, and Go programs are self-contained executables. So you don't need containers. FCGI is an orchestration system, like Kubernetes. It's single-machine, but will start up and shut down processes as the load changes. A crashed process will be restarted. Host the web pages on a static page server, and use client-side Javascript for any dynamic stuff. Good for maybe 20-100 transactions per second. The database will be the bottleneck.
Don't worry, I host serious stuff on a single machine, and am quite happy with it ;) What set me off a bit was the shared hosting. You don't want noisy neighbors, usually. That's worth a few bucks.
Agreed, VPS providers often blind users with super low prices, I didn't even notice this until I started hosting game servers where realtime performance is important. Always make sure that "iostat -c 1" column "%steal" is zero. Luckily there are providers which give guaranteed performance.
Production Go experience. My go to estimate is per node 1-5k http rps with a couple db calls, maybe a network call to an internal service or three, and serializing json. I use that before building for server count and cost estimates. Some services exceed that, never saw a server we made that couldn't do 1k rps.
Friends don't let friends use ruby|python|perl|php|...
Yes. This is the basis of privilege separation and differential rollouts. If you collapse all this down into a single server or even lambda you lose that. Once your service sees load you will want this badly.
> SQS and various background jobs backed by Lambda
Yes. This is the basis of serverless. The failure of one server is no longer a material concern to your operation. Well done you.
> Logs scattered across CloudWatch
Okay. I can't lie. CloudWatch is dogturds. There is no part of the service that is redeemable. I created a DyanmoDB table and created a library which puts log lines collected into "task records" into the table paritioned by lambda name and sorted by record time. Each lambda can configure the logging environment or use default which include a log entry expiration time. Then I created a command line utility which can query and or "tail" this table.
This work took me 3 days. It's paid off 1000x fold since I did it. You do sometimes have to roll your own out here. CloudWatch is strictly about logging cold start times now.
> Could this have been simplified to a single NodeJS container or Python Flask/FastAPI app with Redis for background tasks? Absolutely.
Could this have been simplified into something far more fragile than what is described? Absolutely. Why you'd want this is entirely beyond me.
I agree that cloudwatch is dogturds, but want to dive deeper for illustrative purposes:
Your dynamodb solution isn't foolproof. It has throughput limited to the partition granularity -> in your case the lambda name. It's also relatively expensive and fairly slow to query in bulk (DDB is designed for OLTP).
I don't have direct experience here, but I expect slapping grafana on top of any disk basked source is likely to be cheaper, faster, and have better ergonomics. Once your logging is too much for a disk to handle (this will be later than you would've outgrown ddb, but before you would've outgrown cloudwatch) then you can bring something fancy in.
The event volume is not particularly large as we tend to process things in batch and rarely on the edge of an event. I also wouldn't, for example, log API requests using this mechanism. We're nowhere near this being an issue as 20-30 lambdas is not a particular problem for us. Choose a good naming convention and build your own deployment infrastructure and it's no sweat.
> relatively expensive
Large object compression and/or offload to s3 is baked into our dynamodb interface library. Not that this matters as almost all log records end up being less than 4kb anyways.
> slow to query in bulk
Which is why time is part of the key. You're not often looking back more than an hour. There's bulk export back onto campus servers if you wanted that anyways. TTL is default 1 day. Running a "tail" is absurdly cheap, much cheaper than CloudWatch's laughable rate for their similar feature, a miss is 1/2 a read unit, and a hit is almost never more than 2.
> slapping grafana
I didn't need "observability." I need current state and recent deltas. This is particularly true when any changes are made. Otherwise my logs are pure annoyance and don't generally provide value. We optimized for the exceptionally narrow case we felt the cloud underserved in and left it at that.
> Once your service sees load you will want this badly.
> [...]
> The failure of one server is no longer a material concern to your operation.
Elsewhere in thread you say:
> The event volume is not particularly large as we tend to process things in batch and rarely on the edge of an event.
So the service is not actually under load, and it runs in batches so (temporary) failure is not actually a concern.
> This work took me 3 days. It's paid off 1000x fold since I did it.
Since Lambda was introduced less than 10 years ago, what you're saying here is that it'd be full time job for you for the past 10 years to maintain this (3000 days instead of three) if you have not gone the serverless way, which I find doubtful.
> Could this have been simplified into something far more fragile than what is described? Absolutely.
Considering the hyperboles in the rest of your comment, this sounds more like snark than a considered opinion.
(And also, why are people so kin on doing privilege separation by giving full privilege to a 3rd party and asking it to limit what each piece of code can do?)
There are only relative few startups or non startups which need complex infrastructure from a technical point of view...
In reality, there is a strong bias in favor of complex cloud infrastructure:
"We are a modern, native cloud company"
"More people mean (startup/manager/...) is more important"
"Needing an architect for the cloud first CRUD app means higher bills for customers"
"Resume driven development"
"Hype driven development"
... in a real sense, nearly everyone involved benefits from complex cloud infrastructure, where from a technical POV MySQL and PHP/Python/Ruby/Java are the correct choice.
One of the many reasons more senior developers who care for their craft burn out in this field.
But an application built in the high pressure environment of a startup also has the risk of becoming unmanageable, one or two years in. And to the extent you already have familiar tools to manage this complexity, I vote for using them. If you can divide and conquer your application complexity into a few different services, and you are already experienced in an appropriate application framework, that may not be such a bad choice. It helps focus on just one part of the application, and have multiple people work on the separate parts without stepping on each other.
I personally don't think that should include k8s. But ECS/Fargate with a simple build pipeline, all for that. "Complex" is the operative word in the article's title.
Hence I said "I personally" and "already have familiar tools".
Also, if you're fair... not all those AWS acronyms you're listing would be displaced by the single k8s cluster. (Maybe you weren't arguing to swap out complexity, rather that the complexity floodgates were open already anyway?)
And if one of those services is down, your entire application is down. You basically build a server made of abstract components ECR, S3, ALB, CF , all of which are able to fail.
Except it's not anywhere near as complex because you need to manage far far less using the AWS services than if you ran all of your own inside a k8s cluster. And even if you use k8s, you're probably already using most of those anyway. Who bothers building their own container hosting and file hosting at a startup?
Early in a project you see a lot of similar code paths, and so it’s often tempting to take the logic from two or three e.g. API routes and merge the “clean” abstraction into single piece of logic both routes can call.
Over-time this “clean” abstraction adopts a bunch of optional parameters based on the upstream API routes, leaving you with an omni-function that is more convoluted, and thus harder to change, than if the API routes weren’t overly optimized from the get-go.
As a personal rule, I’ll let myself copy something 3 times before taking a step back and figuring out a “better” way.
After reading all the comments here, the conclusion is to start simple, then switch to k8s and later to cloud-native only when your business has grown to 1000 and then 1 million daily customers respectively.
If someone thinks rolling their own infrastructure is "starting simple" than I have some land in Antartica my great, great uncle is trying to get rid of they might be interested in.
In "serverless" defense, I'll put a one data point from myself. I built https://crates.live 4-5 years ago. I used a "complex" tech stack. A single page web app. Hosted in Github pages as static HTML/JS. For the server side, I used Cloudflare workers (Wasm) to run a GraphQL server (kind of).
The result: It's still up after 5 years. I never looked back after I created the project. I do remember the endless other projects I did that have simply died now because I don't have time to maintain a server. And a server almost always end up crashing somehow.
Another thing, Pieter Levels has successful small apps that relies more on centralized audiences than infrastructure. He makes cool money but it's nowhere near startup-expected levels of money/cash/valuations. He is successful in the indie game but it'll be a mistake to extrapolate that to the VC/Silicon Valley startup game.
To counter your point I have a site running since 2019 that is still up with no input from me or anybody, it’s a dynamic site too. It’s running on docker on a vps at digitalocean. If you build a rock solid configuration it will stand the test of time.
If you'll allow me, I'd like to shill my company for a second. We provide all the benefits of "single server deployment" while providing the scalability of the "30 lambdas" solution.
A single VM is all fine and well until your hacky go-fast code allows an issue with a single request to take down your service. Serverless requests are isolated and will limit the blast radius of your hacky code as you iterate quickly.
I've seen it plenty. A request to process an Excel file or generate a PDF etc. Basically anything generating or processing documents is a likely candidate. It might only affect a single application, but if you are running multiple apps on a box, it is often enough to cause an outage.
397 comments
[ 3.8 ms ] story [ 519 ms ] threadDo startups really need complex cloud architecture?
Inspired, I wrote a blog exploring simpler approaches and created a docker-compose template for deployment
Curious to know your thoughts on how you manage your infrastructure. How do you simplify it? How do you balance?
What I quite like about your repo:
What I think works quite nicely in general:You don't need to, you can just enable/disable certain features during app startup, based on what's in the environment variables/configuration, though many frameworks have built in functionality for something like that, for example: https://www.baeldung.com/spring-conditional-annotations
If I wanted to allow toggling access to the API, then I'd have an environment variable like FEATURE_API and during startup would check for it and, if not set with a value of "true", then just not call the code that initializes the corresponding functionality.
It's really nice when frameworks/libraries make this obvious, like https://www.dropwizard.io/en/stable/getting-started.html#reg... but it might get harder with some of the "convention over configuration" based ones, where you have to fight against the defaults.
> I'd also imagine there would be calls between features from within the same codebase, do those become network calls?
It depends on how you architect things!
There's nothing preventing you from using the service layer pattern for grouping logic, and accessing multiple services in each of your features as needed, and poking the different bits of your data model (assuming it's all the same DB).
If you are at the point where you need more than the same shared instance of a DB, then you'd probably need a message queue of some sort in the middle, RabbitMQ is really nice in that regard. Though at that point you're probably leaning more in the direction of things like eventual consistency and giving up using foreign keys as well.
> And how does this interact with your Docker Compose/single server recommendation?
Pretty nicely, in my experience!
When developing things locally, you can enable all of the needed FEATURE_* flags on your laptop, then it's more like a true monolith then.
Want to deploy it all on a single server when the scale is not too big? Do the same with Docker Compose, or maybe have separate containers on the same node, each with one of the features on, so the logs are more clean and the resource usage per feature is more obvious, and the impact of one feature misbehaving is more limited.
The scale is getting bigger? Docker Swarm will let you scale out horizontally (or Nomad/K8s, maybe with K3s) and you can just move some of those containers to separate nodes, or have multiple ones running in parallel, assuming the workload is parallelizable (serving user API requests, vs some centralized sequential process).
At some point you'll also need to consider splitting things further in your database layer, but that's most likely way down the road, like: https://about.gitlab.com/blog/2022/06/02/splitting-database-...
Rather, I have decided to opt for Supabase instead. Probably over the long time it may cause issues for my startup - but even more realistically my startup is going to fail and my increased developer velocity by using simple tooling like this will allow me to figure out why my idea doesn't work in in a shorter amount of time, so I can go on to my next pursuit.
To be honest I think even using docker is overengineering.
Seriously, I think a lot of people do things the hard way to learn large scale infrastructure. Another common reason is 'things will be much easier when we scale to a massive number of clients', or we can dynamically scale up on demand.
These are all valid to the people building this, just not as much to founders or professional CTOs.
I don’t quite get if people do it for interest, for love of the tech, or if they are technocratic and believe in levelling up their skill to get k8s on their CV like you say.
All I think is “this looks painful to manage”!
I run a k8s cluster at home. Part of it yes, is to apply my existing skills and keep them fresh. But part of it is that kubernetes can be easier long term.
Ive got magical hard drive storage with rook ceph. I can yoink a hard drive out of my servers and nothing happens to my workloads.
I can do maintenance on one of the servers with 0 down time.
All of my config for what I have deployed is in git.
I manage VMS and kubernetes at work, and im not going to pretend that kubernetes isnt complex, but it's complex up front instead of down the road. VMs run into complexity when things change. I'm sure you can make VMS good but then why not use something like kubernetes, you will have to reinvent a lot of the stuff that's already in kubernetes.
It's a hammer for sure and not everything is a nail, but it can be really powerful and useful even for home labs.
Kubernetes does not force you into the distributed systems hell, you can go that route later, or never.
b) In what world is upgrading your entire platform ever a single step. Even for a basic Python app you still have Python itself plus dependencies. And then of course whatever front end web server you're using.
Few but not no people will ever run into problems at the kind of scale k8s operates at. Plus, learning how it "expects" the programs running inside its Pods to behave is kind of like learning how Django or Rails "expect" a web app to work - it's a more complicated style than just writing your own totally custom, hermetically-sealed Python apps for your personal use, sure, but it also comes with a slew of benefits in case you ever do hit that level of scale and want to move over.
Or, maybe you look over the app you're writing and say "Fat chance." In which case you can justify e.g. not making everything an API endpoint, keeping a ton of state mucking about, etc. But I still feel that's an improvement over not even realizing the questions are being asked.
Is that really true anymore? Even self hosting k8s these days (e.g with rke/rke2) is a single yaml file and one command to deploy an entire cluster.. Maybe back when we all used kubespray and networking was more complicated (to the user at least) etc.. But today? I don't think so.
Using a hosted offering is even easier, literally a couple of clicks, a ./gcloud-cli or terraform apply -- again not very hard and all the cloud providers provide you with example code you just need to plug some machine sizes etc into..
Dev setup? Install orbstack and click 'kubernetes' and you're done, your IDE (likely) will automagically pick up your kubeconfig and you can go right ahead creating services, deployments, jobs, whatevers...
You get basic loadbalancing, health checks, centralized and nearly out of the box logging and monitoring and tracing.
You get a streamlined build process (create a container image, have an image build, create your helm chart, done)
Your RAID commment is quite far away of what k8s makes k8s
These things aren't mutually exclusive though. I've spent the last few years working with kubernetes at work and running a 'simple'(but with tons of containers and weird edge cases / uses) unraid server at home for all of my needs. At some point I flipped over from 'jeez kubernetes is just too much, almost nobody should ever use this' to 'wow I have to migrate 99% of my home services to a cluster, this is driving me nuts.' I haven't quite gotten around to that migration, but I do think that k8s cluster for services / temporary storage / parallel jobs and separate unraid box that runs NFS (and doesn't do much else) is going to be a great setup for a home lab.
I dunno why, but the k8s in my workplace keeps breaking in painful ways. It also has an endless supply of breaking points that makes life painful for anybody that depends on what runs in it, but aren't detected by the people that manage it.
Honestly, that second part is an exclusivity there, but I have never seen people "just keeping adding things to it" on practice.
aka, you're front loading the complexity.
You can even think of it as paying insurance premiums upfront. You get to "make a claim" if the requirements do grow into the sort of need that suit such a cluster/complex setup.
It's a bit like factorio with the extra dopamine hit of getting to unbox stuff.
You don't need k8s for all of that, but there's not a simpler solution than k8s that handles as much.
Life is full of pain. Deal with it.
Some people seem to have no concern with the needs and timetables of the would be customers but instead burn through cash building fancy nonsense.
It's like going in to a car mechanic for tires and then finding out it took 3 weeks because the guy wanted to put on low rider hydraulics and spinner hubcaps for his personal enrichment.
The worst part is it's inherently ambiguous to the next people. They don't know if the reason something is there is because it's needed or because it's just shiny bling.
It's so insane. Like hiring a long haul truck to pick up a sandwich
I've got this allergic reaction to bullshit and fetishize successful products and customer satisfaction. I think we've both changed; I'm different than I was 20 years ago and so is web development.
Tight applications with minimal tools that can be pivoted and changed swiftly which require competence and finesse to administer where you don't create developer debt, these are out of fashion.
All profitable hacker spaces professionalize as romantic magic becomes a liability.
I'm a middle aged divorced man, not divorced from a person, but from a profession and I've been trying to date around with new loves.
Having seen some of these half-rolled, first-time-understood k8s deployments, and the multi-year projects to unravel the mess that was created, overflowing with anti-patterns and other incorrect ways of doing things, I think I would prefer a narrower scope of true experienced professionals (or at least some experienced pros that can help guide the ship for their mentees) working on and designing k8s infra.
And for those that don't need it (the vast majority of startups, small businesses, regular-sized businesses, etc), just stick to the easier-to-use paradigms out there.
This is a case where “things will be much easier when we scale to a massive number of clients” turned out to be true.
Everyone is building like they are the next Facebook or Google. To be honest, if you get to that point, you will have the money to rebuild the environment. But, a startup should go with simple. I miss the days when RAILS was king just for this reason.
The added complexity is overkill. Just keep it simple. Simple to deploy, simple to maintain, simple to test, etc. Sounds silly, but in the long run, it works.
Downside is its a one to one system. But I just use downsized servers.
vulnerability scanning of your images.
Fargate
RDS
Yeah, that won't scale to a million QPS, or even 10 QPS. But way more businesses fail because they never achieve 100 Queries Per Day, instead of failing because they fell over at 10 or 1,000 or 1,000,000 QPS.
I mean, hell, Twitter (back in the day) was famous for The Fail Whale.
Getting enough traffic is harder and more important than your "web scale architecture" for your startup. Making actual cash money off your traffic is harder and more important than your "web scale architecture" (ideally by selling them something they want, but making cash money through advertising or by impressing VCs with stories of growth and future value counts too).
There is precisely _zero_ chance that if you ever get within 2 or 3 orders of magnitude of "a million QPS" - that the code you and your cofounder wrote won't have been completely thrown away and rewritten by the 20 or 100 person engineering department that is now supporting your "1000 QPS" business.
But still, no matter what, the odd customer demands they need all these complexities turned on for no discernible reason.
IMO it’s a far better approach with any platform to deploy the minimum and turn things on if you need to as you develop.
Incidentally, I’ve been exposed to “traditional” cloud platforms (Azure, GCP, AWS) through work and tried a few times to use them for personal projects in recent years and get bewildered by the number of toggles in the interface and strange (to me) paradigms. I recently tried Cloudflare Workers as a test of an idea and was surprised how simple it was.
I thought the same thing until recently. Apparently there's a "Docker Swarm version 2" around, and it was the original (version 1) Docker Swarm that was deprecated:
https://docs.docker.com/engine/swarm/
Haven't personally tried out the version 2 Docker Swarm yet, but it might be worth a look at. :)> Docker secrets are only available to swarm services, not to standalone containers. To use this feature, *consider adapting your container to run as a service. Stateful containers can typically run with a scale of 1 without changing the container code.*
(Emphasis mine. From https://docs.docker.com/engine/swarm/secrets/ )
The additional benefit is devs can run all the same stuff on a Linux laptop (or Linux VM on some other platform) - and everyone can have their own VM in the cloud if they like to demo or test stuff using all the same setup. Bootstrapping a new system is checking in their ssh key and running a shell script.
Easy to debug, not complex or expensive, and we could vertically scale it all quite a ways before needing to scale horizontally. It's not for everyone, but seed stage and earlier - totally appropriate imo.
Both are Apache 2 and the Flatcar folks are excellent to work with
https://github.com/flatcar/Flatcar#readme
https://github.com/bottlerocket-os#bottlerocket
Running and configuring VMs isn't hard to do correctly, it just takes discipline to never "hack it in the moment" - or if you do, can that change in your config system.
Yup, and I'm glad your experience has been different from mine but mine has been that tired and stressed people are anything but disciplined, so nipping a few "I'll just apt-get ..." in the bud goes a long way. So does Reverse Uptime (or its friend, Chaos Engineering)
If it interests you, both major git hosts (and possibly all of them) have and endpoint to map a username to their already registered ssh keys: https://github.com/mdaniel.keys https://gitlab.com/mdaniel.keys
It's one level of indirection away from "check in a public key" in that the user can rotate their own keys without needing git churn
Also, and I recognize this is departing quite a bit from what you were describing, ssh key leases are absolutely awesome because it addresses the offboarding scenario much better than having to reconcile evicting those same keys: https://github.com/hashicorp/vault/blob/v1.12.11/website/con... and while digging up that link I also discovered that Vault will allegedly do single-use passwords, too <https://github.com/hashicorp/vault/blob/v1.12.11/website/con...>, but since I am firmly in the "PasswordLogin no" camp, caveat emptor with that one
I have had great success with a very simple kube deployment:
- GKE (EKS works well but requires adding an autoscaler tool)
- Grafana + Loki + Prometheus for logs + metrics
- cert-manager for SSL
- nginx-ingress for routing
- external-dns for autosetup DNS
I manage these with helm. I might, one day, get around to using the Prometheus Operator thing, but it doesn't seem to do anything for me except add a layer of hassle.
New deployments of my software roll out nicely. If I need to scale, cut a branch for testing, I roll into a new namespace easily, with TLS autosetup, DNS autosetup, logging to GCP bucket... no problem.
I've done the "roll out an easy node and run" thing before, and I regret it, badly, because the back half of the project was wrangling all these stupid little operational things that are a helm install away on k8s.
So if you're doing a startup: roll out a nice simple k8s deployment, don't muck it up with controllers, operators, service meshes, auto cicds, gitops, etc. *KISS*.
If you're trying to spin a number of small products: just use the same cluster with different DNS.
(note: if this seems particularly appealing to you, reach out, I'm happy to talk. This is a very straightforward toolset that has lasted me years and years, and I don't anticipate having to change it much for a while)
One big advantage of the operator is that its custom resources are practically kind of standard by now. This means helm charts for a lot of software ship those and integrating that piece of software into your monitoring is a matter of setting a few flags to true. The go to solution for a k8s monitoring setup is https://github.com/prometheus-community/helm-charts/tree/mai...
But yeah, if I only wanted a thing, Ebs works.
rsyslog + knowing what the fuck you are doing is much better.
Boring, but useful.
In all seriousness, that is "no load". I know it fits 99% of all startups, and many larger companies too, but that's kind of the point.
I wouldn't do it differently though, I think it's a perfectly fine architecture :)
But that will cost more than $10/month.
I've not seen it with Go because I haven't worked with Go in a production capacity; but I've seen C# handle thousands of RPS per node.
Friends don't let friends use ruby|python|perl|php|...
Thankfully. ;)
Yes. This is the basis of privilege separation and differential rollouts. If you collapse all this down into a single server or even lambda you lose that. Once your service sees load you will want this badly.
> SQS and various background jobs backed by Lambda
Yes. This is the basis of serverless. The failure of one server is no longer a material concern to your operation. Well done you.
> Logs scattered across CloudWatch
Okay. I can't lie. CloudWatch is dogturds. There is no part of the service that is redeemable. I created a DyanmoDB table and created a library which puts log lines collected into "task records" into the table paritioned by lambda name and sorted by record time. Each lambda can configure the logging environment or use default which include a log entry expiration time. Then I created a command line utility which can query and or "tail" this table.
This work took me 3 days. It's paid off 1000x fold since I did it. You do sometimes have to roll your own out here. CloudWatch is strictly about logging cold start times now.
> Could this have been simplified to a single NodeJS container or Python Flask/FastAPI app with Redis for background tasks? Absolutely.
Could this have been simplified into something far more fragile than what is described? Absolutely. Why you'd want this is entirely beyond me.
Your dynamodb solution isn't foolproof. It has throughput limited to the partition granularity -> in your case the lambda name. It's also relatively expensive and fairly slow to query in bulk (DDB is designed for OLTP).
I don't have direct experience here, but I expect slapping grafana on top of any disk basked source is likely to be cheaper, faster, and have better ergonomics. Once your logging is too much for a disk to handle (this will be later than you would've outgrown ddb, but before you would've outgrown cloudwatch) then you can bring something fancy in.
The event volume is not particularly large as we tend to process things in batch and rarely on the edge of an event. I also wouldn't, for example, log API requests using this mechanism. We're nowhere near this being an issue as 20-30 lambdas is not a particular problem for us. Choose a good naming convention and build your own deployment infrastructure and it's no sweat.
> relatively expensive
Large object compression and/or offload to s3 is baked into our dynamodb interface library. Not that this matters as almost all log records end up being less than 4kb anyways.
> slow to query in bulk
Which is why time is part of the key. You're not often looking back more than an hour. There's bulk export back onto campus servers if you wanted that anyways. TTL is default 1 day. Running a "tail" is absurdly cheap, much cheaper than CloudWatch's laughable rate for their similar feature, a miss is 1/2 a read unit, and a hit is almost never more than 2.
> slapping grafana
I didn't need "observability." I need current state and recent deltas. This is particularly true when any changes are made. Otherwise my logs are pure annoyance and don't generally provide value. We optimized for the exceptionally narrow case we felt the cloud underserved in and left it at that.
Elsewhere in thread you say:
> The event volume is not particularly large as we tend to process things in batch and rarely on the edge of an event.
So the service is not actually under load, and it runs in batches so (temporary) failure is not actually a concern.
> This work took me 3 days. It's paid off 1000x fold since I did it.
Since Lambda was introduced less than 10 years ago, what you're saying here is that it'd be full time job for you for the past 10 years to maintain this (3000 days instead of three) if you have not gone the serverless way, which I find doubtful.
> Could this have been simplified into something far more fragile than what is described? Absolutely.
Considering the hyperboles in the rest of your comment, this sounds more like snark than a considered opinion.
What relation does any of those have with load?
(And also, why are people so kin on doing privilege separation by giving full privilege to a 3rd party and asking it to limit what each piece of code can do?)
In reality, there is a strong bias in favor of complex cloud infrastructure:
"We are a modern, native cloud company"
"More people mean (startup/manager/...) is more important"
"Needing an architect for the cloud first CRUD app means higher bills for customers"
"Resume driven development"
"Hype driven development"
... in a real sense, nearly everyone involved benefits from complex cloud infrastructure, where from a technical POV MySQL and PHP/Python/Ruby/Java are the correct choice.
One of the many reasons more senior developers who care for their craft burn out in this field.
But an application built in the high pressure environment of a startup also has the risk of becoming unmanageable, one or two years in. And to the extent you already have familiar tools to manage this complexity, I vote for using them. If you can divide and conquer your application complexity into a few different services, and you are already experienced in an appropriate application framework, that may not be such a bad choice. It helps focus on just one part of the application, and have multiple people work on the separate parts without stepping on each other.
I personally don't think that should include k8s. But ECS/Fargate with a simple build pipeline, all for that. "Complex" is the operative word in the article's title.
And at that point you've assembled a stack just as complex as doing it all inside a single k8s cluster.
Also, if you're fair... not all those AWS acronyms you're listing would be displaced by the single k8s cluster. (Maybe you weren't arguing to swap out complexity, rather that the complexity floodgates were open already anyway?)
Very common in fact since many k8s clusters are air-gapped except for a single inbound edge node.
1. New technology is bad and has no merit other than for resume.
2. Use old technology that I am comfortable with.
3. Insist that everyone should use old technology.
hahahahahahahahahaha. Yes, back in the days when all you could do on a website is read the text.
Old technology was 1000% not simpler. What an insane & absolute statement to make in an enormous field just because you can't make a solid argument.
Focus on product market fit (PMF) and keep things as straightforward as possible.
Create a monolith, duplicate code, use a single RDBMS, adopt proven tech instead of the “hot new framework”, etc.
The more simple the code, the easier it is to migrate/scale later on.
Unnecessary complexity is the epitome of solving a problem that doesn’t exist.
Over-time this “clean” abstraction adopts a bunch of optional parameters based on the upstream API routes, leaving you with an omni-function that is more convoluted, and thus harder to change, than if the API routes weren’t overly optimized from the get-go.
As a personal rule, I’ll let myself copy something 3 times before taking a step back and figuring out a “better” way.
Since it's B2B we don't need zero downtime, updates at midnight are all right.
A day before rollout they go through the staging server and the test environment, so no surprises the next morning.
Before updates, the backups kick in, so if we need to recover from a bad update we can roll back.
Sounds all 2000 and not very fancy but boring and profitable cuts for us
My ci/cd is doing a system test because everything is in containers. I can do full e2e tests and automatic rollouts without a downtime.
What i can do, can everyone else do when i'm on holiday.
How fast are you back if your server burns down tomorrow? How often have you tested that?
Are your devs waiting regularly on things?
Yes, we collect server metrics - that's pretty old-school
> How fast are you back if your server burns down tomorrow? How often have you tested that?
25 Minutes - we test it once a year and we have third partys to check it. It's called an audit. They also check other cyber security related stuff.
> My ci/cd is doing a system test because everything is in containers. I can do full e2e tests and automatic rollouts without a downtime.
We have a staging system for this.
> What i can do, can everyone else do when i'm on holiday.
We also have documentation; is this really a big thing?
> Are your devs waiting regularly on things?
Code Reviews, these take time
---
Are these real problems organizations have?
Huh. I never said to roll one's own [hardware] infrastructure, although it even makes sense if having a GPU cluster.
The result: It's still up after 5 years. I never looked back after I created the project. I do remember the endless other projects I did that have simply died now because I don't have time to maintain a server. And a server almost always end up crashing somehow.
Another thing, Pieter Levels has successful small apps that relies more on centralized audiences than infrastructure. He makes cool money but it's nowhere near startup-expected levels of money/cash/valuations. He is successful in the indie game but it'll be a mistake to extrapolate that to the VC/Silicon Valley startup game.
You can even run the whole thing locally.
We actually just did a Show HN about it:
https://news.ycombinator.com/item?id=41502094
Faulty input killing your logic - I saw this plenty, would Lambda really help here?