Ask HN: Solo-preneurs, how do you DevOps to save time?
When you're 1 man show (or tiny team) working on some web oriented application beyond POC level and beyond heroku (for whatever reason) — managing your CI, deployments/rollbacks, DBs etc. looks like a nightmare to me.
Just properly setting up and maintaining a small k8s cluster looks almost like a fulltime job.
I wonder how do your CI, deployment workflows look like?
Any particular tools, practices, guidelines, architecture patterns to reduce pain & timewaste?
How do you CI => deploy/rollback stateless instances? How do you manage your DBs and backups? Etc.
327 comments
[ 5.6 ms ] story [ 289 ms ] threadDBs run side by side and are simply backed up regularly with whatever backup solution the VPS offers.
That's it.
Out of school I put way more effort in building my infrastructure but in reality a good VPS, something like Cloudflare and some app based caching can run millions of daily views over 20+ different apps without issues.
Edit:// it's mostly rails apps I host btw. For my wordpress installs i use dedicated VPSes because those scale horrible.
I read such messages (and also agree to it mostly) and then I encounter my peers wanting active-active clusters with K8s on top or something and I feel I like a dinosaur.
Refreshing to see KISS still being used. Also gitea. What an awesome piece of open source software.
> git init --bare
On your remote server is all you need to spawn a repo to push to, usually hook examples already included.
The hook can then directly checkout to my live directory, run migrations and restart the server without weird remote automations after every (or specified) push.
Gitea is just a little extra (setup once, touch never thingy) for quick edits and checks (it's a wonderful lightweight piece of software too)
Hatchbox manages the deployment, backups, rollback etc.
Envoyer and Forge for PHP stuff.
Render/Heroku for node.js stuff. Netlify for static stuff.
For the more complex bits I have, I use Pulumi to manage everything through IaC. It's still complex but at least it's robust and I can sense-chech changes to infra through a PR to myself.
As for DBs / engines and such, I try to go for hosted services as long as they're not stupidly priced. I'm currently using Elastic cloud. If I need a relational DB I'll probably deploy Postgres on AWS and install some scripts to back up the data to an S3 bucket every day or so.
I chose NOT to use k8s or Docker because, like you said, it's basically a fulltime job to maintain. I don't think it's needed at all unless you're a large organization.
So basically: a mixture of roll-your-own, streamlining platforms, and hosted services. Do whatever works and is easiest, and don't worry so much about having ALL the goodies.
Also, I review how much time each product / feature consumed each year and then decide what to deprecate. Each April I then have my own little party where I ceremoniously turn off last year's time wasters.
I'd say it all boils down to "know your customer lifetime value"
It sounds like you have a great set of intuitions, and you're right, all that infrastructure is a nightmare to set up and manage.
Step 1: Question every requirement
Step 2: Drop everything that is not critical
Step 3: Profit!
Git hooks and Makefiles are great!
I have an HP workstation I bought from eBay under my desk that has 32GB of RAM, 2x 12 core CPUs and a static IP address. It's probably 8 years old, but fast enough to serve whatever we need for a long while.
The machine (Old Blue), hosts our Git repos, web app, database and integration service (git hook that calls, 'make publish').
We're not serving Google scale traffic, so we don't need Google scale infrastructure.
Keep it simple whenever possible and don't let the modern stack complexity creep in until you absolutely need it.
Even going to the cloud, when you've done it 10 times and know how, is way more work than you need when just starting out.
Take on those costs and complexities only when your traffic requires it, and you may just find out that you never have to pay rent for compute.
In fact, it's super easy to back up as it's in a SQLite database (for now).
I used to rsync them, but when traffic picked up, I started to fear an inconsistent copy (rsync does verify at the end there are no new changes, and restarts - but i don’t know the exact strategy so I am wear if relying on it).
I switched to doing .backup and rsync that instead - but am considering switching to litestream instead.
I have multiple projects, none of them making money currently, so $100/month for each basic app is not good.
I switched to using Dokku on Vultr, $12 a month. You can easily create Postgres databases and link them to Docker apps. I haven't bothered to setup CD yet but it looks like it should be simple, for now I just push to it when I want to deploy. Liking this setup so far.
For frontend I use Netlify, I use their redirects to proxy the backend.
k8s, ci, etc are all really useful and solve a lot of hard problems, but god are they a bitch to set up and then monitor, fix when something weird happens, secure from nasty people outside, patch when some update becomes necessary, resist the urge to “improve”, etc. This doesn’t include the time it takes to work these tools into your project, like k8s for example really requires you grok its model of how stuff it runs is supposed to look. There’s a reason entire companies exist that simply offer managed versions of these services!
If you’re using fewer than like 10 boxes and don’t work with many people, you can get away with spooky arcane oldhat sysadmin deploy techniques (scp release to prod hosts, run deploy script, service or cron to keep running) and honestly it’s pretty fun to just run a command and poof you’re deployed. Then you can just wait until it starts making sense to use fancier tooling (more load, more sites/DCs, more services, more people/teams, etc). Monitoring is always good to start with, for other stuff a good litmus test is that price points for hosted k8s & ops stuff will start to make sense, you’ll be paying in time in any case. Who knows, maybe when that happens something new will pop up that makes even more sense. Until then, reducing the amount of stuff you need to worry about is always a good strategy.
If you want an intermediate step you can just make docker images of your application and deploy and run those manually (again with shell scripts).
You can get a long way that way without having to deal with kubernetes.
If you have multiple people on the team that have access to servers, this will shoot you in the foot a year or two later. Good luck replicating that server setup when no one has any clue whatsoever about all the small tweaks that were performed ad-hoc over the years.
It is easy when the server is regularly backuped.
I never realised I was using spooky arcane oldhat stuff! I feel wizardly now.
My projects (for small clients and myself) basically use this.
- A "build.sh" script that does a local build of back end and front end
- A "deploy.sh" script that scp's everything to the server (either a digital ocean VPS or an EC2 instance), runs npm install, runs database migrations and restarts with pm2
So running my entire CI pipeline in terminal is: ./build.sh && ./deploy.sh
https://ansistrano.com/
It's pretty fantastic.
I've been writing some Ansible playbooks recently for the first time in years, and came upon geerlingguy's work. That guy is a powerhouse when it comes to writing Ansible roles/modules!
Each schema migration should also come with its own rollback script.
The downside is that you might need three migrations for some operation, but at least you won't break stuff.
The assumption that you can do a schema migration while deploying new code is only valid when you have very limited database sizes. I've seen Flyway migrations break so many times, because developers assumed it was fine to just do complicated migrations on a 200GB database. Or a Django database migration just lock up everything for hours because no one cared to think about the difference between migrating 100MB and 100GB. And I've never seen anyone seriously considering rolling back a Flyway migration.
----------
Static website:
1. Setup NginX using a symlink to the current version
2. Copy the new files to a separate folder
3. Change the symlink to point to the new version
----------
REST service (behind NgninX reverse proxy):
1. Current version runs on port X.
2. Deploy new version and run it on port Y.
3. Update NginX config to use port Y and reload
4. If you need rollback, just reverse step 3.
This can be done using scripts or Ansible too if necessary.
This also solves the roll-back problem mentioned elsewhere. Just checkout the previous version (as long as you don't make any side effects like DB updates that are incompatible).
I admit, this requires relatively simple software.
Then you could have any necessary restart or migration or whatever run in a post-receive hook.
I realise that's starting to get back in to tooling and config etc., but it's not that complex, and if you're already using git anyway...
> and probably very stupid) method is an Expect script that lets me supply a password to sudo.
The old school sysadmin way of doing this would be to have a dedicated deployment user (ssh pubkey auth only - possibly restricting only specific commands), with a sudoers[.d] configuration to allow that user to run an explicit group commands without a password (NOPASSWD)
Alternatively, it's pretty easy to use git-archive (optionally, with an exclusions list to keep out files you don't want to serve publicly) to do something similar without ever cloning the remote repo. You can fetch a zip or tar.gz with a given commit's files, straight from a remote repo.
Also you probably don't want your web under /var if its a partition and you fill it with lets say user content then you will lock up your server.
The sites are either on Linode or SSDNodes. Has been working for me for decades and don't need to change it for these hobby things. After all, the log files prove that these things get more traffic from exploitation probes than real people ;-)
Admittedly, these are mostly "local" drive-to-drive transfers over USB rather than server-to-server transfers over house-wide gigabit. But consider trying the transfer without compression.
I'll definitely have to read up on your other flags though, if your rsync works from just one machine rather than both that might solve a minor problem case I have where the destination machine lacks rsync...
https://unix.stackexchange.com/questions/188737/does-compres...
I should review those options, definitely.
> -C, --cvs-exclude auto-ignore files in the same way CVS does
Just goes to show you how long I've been using this stuff :-)
The additional options may not be relevant now that I am just deploying creating a new target directory each time.
You would be surprised at how many old enterprises still do this. Sure, it's ugly, but it can be simpler than k8s, and anyone with scripting chops can understand it.
I cannot imagine not setting up at least a basic CI. Even if you remove all unit/integration test alarms, a CI is still very useful. The 20 minutes of build -> upload -> deploy script -> check prod vs a 1/2 day to set up a CI and forget it.
I'm not sure you save much time but you save a lot of headache
That being said, it would take a lot to convince me a standard CI/CD pipeline is not worth the investment. My only experience with non CI/CD deployment process was a very manual process at a startup, then when I saw my first CI/CD it felt like magic.
I am not sure how it would work on a larger scale, but for my use is perfect.
Vercel first paid plan is $20/month and FaunaDB $25/month, so my guess is a business with some traction would have to pay at least these $45/month.
I didn’t do CI/CD. I just reviewed my own code the next day, improved it and put it live. As the only dev I knew more or less how my code worked and what risk factors to prepare for in each deploy, and serious, business-impacting production issues were rare.
A simple stack of Meteor on Galaxy + Mongo on Atlas meant all my tools integrated well together and a whole lot of devops things were easy enough that I rarely had to think about them after they were set up once.
I can think of many cases where this would not work but it worked well for my context.
I've been running a Python-based, highly custom web store solo since 2007, and it supports multiple people doing fulfillment. I host on Opalstack so as to outsourcing patching, email config, database maintainence, etc. I run directly in a Git repo (actually Hg, it's that old) and releases are "git pull && ./restart.sh". Rollbacks are "git checkout ...".
I've had to migrate/rebuild the VM about every 5 years. Tech changes enough in that time that no automation will still work unmodified. So I just keep good notes about what I did last time, and figure out what the new equivalents are when I finally have to do it again (updating the notes, of course). Database and Conda are easy to port. It's usually DNS and email integrations that are a pain.
As others have said, KISS is key. Industry DevOps is for a work setting with a decent-sized team, where you can afford the overhead of maintaining it all in order to make the overall team more efficient.
I think if you’re a solo founder you have to accept on day one you’re at a time disadvantage when it comes to the actual hours you’ll spend on moving the business forward vs admin, ops, legal stuff etc etc and so you have to really fight smart!
I.e K8S? Definitely don’t need it. Complex CI/Deployment workflow? Nope, just the minimal shell scripts to do clean deploys and rollbacks. Pay for a HA RDS so you never need to worry about DB replication etc.
If you just work with the minimal set of tools you’re comfortable with and relentlessly keep things simple it’s possible. I’ve built high availability stuff as a solo founder and still found (at least some...) time to work on sales and the business.
Unless you build a side project to learn a new technology, you should use something "boring" that you know well.
Here's my personal choices in preferred order:
1. Static web sites - Just upload to S3 behind a CDN and forget about it
2. Server rendered website - PHP. It may not be something you brag about, but in my experience it just "works" and has excellent uptime.
Note: If I had to recommend something to a larger organization, I would probably suggest Next.js to attract developers. PHP is often associated with "old school" tech
3. REST service - Spring Boot. I have so much experience with it and there are so many Spring projects that supports a huge range of technologies from monitoring to data
4. Scripts - Python
Exactly my point that I was making here in this thread: https://news.ycombinator.com/item?id=28840902
An example would be AWS' RDS
I personally manage the CI/CD infra as well as the data infrastructure for my startup.
Here is my setup:
1. Self-hosted Drone.io instance connected to github(the instance is behind a firewall and can only be accessed via a self-hosted VPN. Github IPs are whitelisted so that auto build/deployment can be initiated via github hooks)
I have drone.io config that helps me manange the versioning, deployment to stage/production, run tests, do rollbacks etc.
for eg: merging to master on github auto builds the master branch on drone and deploys to staging along with pulling the latest db backup from S3 and adding to stage for testing
2. Self-hosted production and staging server that hosts my web/django app. It has a local postgres db and is encrypted and backed up on S3 every hour.
3. Self-hosted Elasticsearch cluster using Ansible playbooks with daily snapshot backups to S3.
beyond this, I self-host 3 VPNs to protect my entire infrastructure. read my old HN comment about it: https://news.ycombinator.com/item?id=28671753
All this is one time infrastructure setup and it all has been running smoothly for more than a year without any hiccups.
I would be happy to help you setup a similar setup if you want. hit me up at vikash@quantale.io
Did you have any instance where you needed to restore the DB / Have you done restoration tests to ensure this is a feasible solution? Also, since you are using local postgres DB, do you not run into issues like transaction id wraparound ?
Also, how many days of backup you keep on S3? Considering you create 24 backups a day.
Whenever a build is deployed to stage server, it automatically pulls the latest version backup in last 1 hour and restores it on the stage for testing with the latest data. I have config to do the same restore to the prod if needed.
I have been using the same setup for over 2 years to support an internal web app of a company.
> Also, since you are using local postgres DB, do you not run into issues like transaction id wraparound ?
This problem only arises when there is heavy load on the database, specifically high WRITE/UPDATE/DELETE operations. Quantale uses postgres to store only user info and all the other data is stored in Elasticsearch(twitter, reddit, news, SEC, Financial data on over 15000+ stocks). This is intentional because with the scalable server resources, my postgres setup can handle a large number of users before I see that problem. I can always scale up later when needed rather than over-engineering it right now.
>Also, how many days of backup you keep on S3? Considering you create 24 backups a day.
I have 7 days backups on S3 with versioning switched on for 60 days. So at anytime I can access the version backed up in last 60 days.
Safe to say you have a verified way of backup and restore (with worst case data loss of an 1 hour).
I am not exactly an expert on what stage does Transaction ID Wraparound comes in but still know that as one of the key criticisms of PostgreSQL so wanted to check.
Also, 7 days backup with versioning switched is a smart tactic! Thanks once again for this.
What I have now is terraform on Github which handles Networking (VPC, subnets), ECS/ECR, ALB for my backend and S3/CloudFront for frontend.
Everything is on AWS because I'm familiar with it, so it's faster. And I compensate for costs with AWS Activate + Free Tier. This combination is usually enough to understand whether the project will get traction or not.
Actually spending little time on IaC was one of the best recent investments. But I guess it holds more value when you have a couple of projects, still exploring and these tasks get repetitive
Later, I realized that Heroku + Netlify are good enough. Switching was easy. The hardest part was admitting to myself that I had wasted so much time on AWS. I felt relieved after I deleted the Terraform and Packer code.
I was thinking Digital Ocean. Or Heroku for the current project, but ended up with a design with quite a few serverless pieces using lambda. Which has a great free tier and is very useful for my use-case.
Would not say that AWS is too much complexity though.. When you go over things at least once and understand how thing fall together
The problems of a solo dev are very different than a dev on a team. Knowledge silos don't exist. Distributed expertise doesn't exist. There's no one to mentor, no shared vision to maintain, no intertia to combat.
I consult on big complicated team projects. I also manage multiple solo projects.
On solo projects, deployment is a script I run from my dev machine. I'm the only person who deploys; anything else would be solving a problem I don't have.
The only "CI" is running the tests before I run the deployment script. I'm the only one who needs to see the outcome of the tests. Anything more would be solving a problem I don't have.
Architecture is whatever makes the most sense to me personally -- which is often miles away from what I would recommend to a client, who needs something any new hire will recognize and be able to work with.
I pay a service to manage backups so I can walk away from a solo project for months and know it's ticking away.
The point is: solve problems you actually have. Don't try to run a "professional" operation by doing what big teams do all by yourself. Big teams have different problems.
Any recommendation of such service?
[1] https://github.com/spatie/laravel-backup
This bit seems to hold true across any size team for me. If I have notes I can spiffy them up and make cheat sheets out of them and bootstrap other developers faster. Or if it is just me, my feeble brain will forget odd details that I needed for something.
[1] https://sambleckley.com/writing/lab-notebooks.html
- Easy Postgres clusters (but not managed) - Heroku like deploys (but less magic) - Easily geodistributed (deploy near your users) - reasonable pricing to get started (and not too bad beyond that).
GiHub actions-> docker build->docker push to github package repo
then on vm a 2 line script to docker pull && docker-compose up -d
Database all by hand.
Extremely easy i just push, wait build and restart
Here are the list of DevOpss flow I've done:
* One project is a website. I have a pipeline, triggered when something is merged to master, that builds Docker images, pushes them to a Dokku instance and deploy them.
* One project is a JS library. I have a pipeline, triggered when a git tag is created, that packages the library and uploads it to npm. I use Netlify to host the documentation website of it.
* For this same project, I'm planning to have a pipeline triggered for each PR to run the linter and tests.
I work as a DevOps/MLOps, and I'm quite comfortable with CI tools in general. I hate doing things manually and I love automating, so it comes quite naturally for me to do it. Once it's set up, it rarely needs to be changed.
Good luck!
Yes, you'll pay more for a Heroku Postgres instance than you would for a VPS on Digital Ocean with Postgres running on it, likewise you'll pay more for Heroku Dynos than another VPS to run your application server. On the other side though, your backup, rollback, deploy, and scaling strategies can all be "Heroku does that", and you can focus on the much more valuable job of building a business.
Other competitors like render.com are starting to look like they'll be competitive on features and are way cheaper too.
I see this all the time. Entrepreneurs trying to solve the people problem by avoidance. The problem with the super efficient player is twofold. First, it's not really scaleable. Second, what if the person gets sick or is in vacation?
BUT: In the last year I've had some rather serious downtime issues that were 100% Heroku's fault. (Poor encapsulation of Postgres configuration via the connection string, and they mistakenly claimed that I violated their TOS when I didn't. They apologized and fixed the situation.)
I'd really like to see some good alternatives to Heroku. Not just because of their screwups, but because the languages that I want to work on have limited support in Heroku.
Edit: Yes, I know there are community supported ways to do this, But I want something that is as slick as the NodeJS ecosystem on Heroku.
They use self-signed certificates on Heroku Postgres servers without a certificate authority present in the chain. No chance to establish reliable peer verification. A shame, I would love to use Heroku more often without compromising security.
That said, I would also recommend Heroku to many.
The context here is also that of a single entrepreneur building a business. At that point in the lifecycle of a product downtime really isn't that big of a deal, customers tend to be pretty understanding so long as you recover.
1. Their server manager process got in an unresponsive state. It wouldn't update my service. It stopped reporting logs from the service that was running. I went to the AWS Loft and consulted with the AWS Support technician. They had me log into the machine with SSH and troubleshoot. I was using Elastic Beanstalk specifically so I would not have to log into machines.
2. The console frequently showed servers as running when they weren't. And vice-versa. I wasted a lot of time trying to figure out why my server didn't start when it actually did start and why my server isn't listening on its port when the console incorrectly showed that it started. Logs were also delayed randomly.
3. The final straw happened on a Saturday. Elastic Beanstalk suddenly started returning errors. I purchased an AWS Support subscription and filed a ticket. It turns out Elastic Beanstalk Team had deployed on a Saturday and then rolled back. Unfortunately, anyone who did API calls through the new API version got new values added to their Elastic Beanstalk API backing database. The rolled-back the server errored when it saw the new values. They refused to just fix the bad data in their database caused by their bad deployment. Instead, they asked me (their customer) to download special management tools and perform manual operations to fix it. And I had to pay them to find out these instructions.
Now I use Heroku.
Looks interesting
> Create a backend
But you'd probably host your SPA elsewhere, eg Vercel, Netlify, GitHub pages
Would be curious how it compares to some of these.