Ask HN: Companies of one, how do you ensure business continuity?
Reading the thread about companies of one's tech stack[1], I wonder how you ensure business continuity (in case of holydays, family event, sickness, ... ), especially when you host an application.
I'm currently developing a service alone, but one of the problem I see going forward is how can I ensure business continuity if when I'm not available.
Very interested to know more about your experiences!
[1]https://news.ycombinator.com/item?id=32960033
25 comments
[ 3.2 ms ] story [ 66.6 ms ] threadAs soon as it makes sense, you should start hosting somewhere in the cloud so you can offload at least some of the potential problems to SREs. I also found that moving your database to the cloud can be a good trade-off when running solo.
Running a popular service solo is strictly for masochists :)
If there is too much to handle, it means there is some cash flowing in and might be time to get a first employee (or outsource some tasks).
They're definitely still worthwhile, but that needs to be weighed against how comfortable you are setting up/maintaining these to begin with.
If they don't divert too much energy because you're already well versed in them, they're probably a win.
But needing to shell into a couple of EC2 boxes & manually deploying/upgrading stuff might be a safer bet for some people.
You don't want to answer a P1 ticket at 3am & be getting some esoteric Kubernetes error and have no-one to ask. I think it's tough to learn & adopt new ops tools as a team of one if they're things that are on the critical path.
However, I disagree with the small benefits of DevOps for a solo dev: if done properly, you start to have complete 8+ hours night sleep!
Simplify business + product design and make arch boring
Prioritize getting more revenue so you can hire
- Keep everything as simple as it can sensibly be. Obviously this will depend on your application but if you can, you're better off having a simple docker compose stack than a complex k8s cluster (as an example), if you don't really need it. This will make it quicker to fix and get back to your vacation when you need to.
- If you can afford it, pay for cloud hosting and other services (like backup services, Route53 domains with certificates etc). Their SRE efforts are going to be infinitely more capable and available than yours are.
- Use password managers and secure them well. Make it so that you could lose your laptop and be up and running again within 30 minutes of buying a new one.
- Keep your development process consistent and documented. I develop everything inside a Docker container and use the same application/service template for everything. Again, this makes it much faster to troubleshoot and bug fix on another machine if required. I do all my development on an EC2 instance using VSCode Remote SSH + Containers so I can connect to the same environment from any machine. This also means if I need to fix something from somewhere with spotty internet, I only need SSH access and I have access to 10Gb internet etc, so uploading new images and patches etc is not affected by my location.
All these things probably cost me a few thousand dollars a year (tax deductible) but to me they are worth it for the peace of mind and the few times I have had to fix something in a pinch, it has more than paid for itself.
Having a spare laptop and phone is cheap - it does not have to be latest and greatest.
Keep a solid stack of cash. 6-9 months worth of potato diet (consider fixed costs). Never let bills run out on you. Never trust dynamic pricing.
Make systems work for you 24/7, keep measuring, have alerts set, incl. recurring jobs via calendar.
Keep a solid log of changes and most recordable things.
Offload everything related to accounting and focus on one thing, the tech.
Keep your customer's expectations in check. They should know you're not a multinational VC funded grunt cave. Keep it human, keep it real. So that when stuff does go bad, you are on the right foot to answer for it, and the narrative is based around compassion and not SLA's.
Sometimes you lose motivation for 6 months... and that business should be able to tick along just fine.
In many ways a business of one needs simplicity and over-engineering. Monitoring, alerts, automation... but all really simple. Nothing clever, nothing difficult, composed of entirely standard parts, tech, systems. I say over-engineering as you don't want to need a human (you) to be doing anything for this thing to function. Your job is literally to put yourself out of your job... and for the business to cruise along. Automate everything... and if you cannot do so, make it so simple that toil is virtually nothing.
I have a small search engine for chemical properties[0] running with a boring go stack, with just a stupid SQLite database (but a custom in memory index for some particular "chemical" search part). It has been humming around for the past 10 years without downtime.
The stack is incredibly simple and monolithic. A complete redeploy is made by a simple push on master, simple CI/CD with Woodpecker[1] building Docker images and triggering a refresh of the system with docker compose.
Every single step can be down manually if a given step or component is failing, but in the "everyday" scenario, everything is automated.
So, keep it as simple as possible with minimal "interconnections" with other services. If possible do not depend on any external services.
[0]: https://www.chemeo.com
[1]: https://woodpecker-ci.org/
For me it's all about keeping things simple, automating everything that can be automated and keeping structure in my day. Simple things like always handle all operational stuff as soon as it pops up so you'll never get behind on anything. Once you start getting behind on administration or e-mails it becomes progressively harder to catch up.
And yes, this means that if you're going on holiday your laptop goes with you.
Also, I prioritize eating, exercise and sleeping. If it means I get to spend less billable hours doing an integration project for a customer then so be it. It's almost impossible to overstate how important it is to keep physically and thereby mentally fit when running a business on your own.
- What if the employee quits
- What if you're unavailable as before, and the employee gets stuck
- What about the plethora of tasks you cannot expect the employee to do
All the while adding complexity of hiring and managing someone
Technical user support & service uptime: can simplify by design, docs, self-healing. For a focused service, remaining timely human support can go to a new hire. If they can even handle 50% of remaining load here, and defer responses to next day/few days, awesome. If they leave, can get another. Bonus points if they can help further reduce the sting here over time by writing more docs or writing more code.
Sales: Unfortunately, someone good at technical support is often not good for sales support, and a minimum, not at the level of the founder who has seen so much more. So self-serve sales automation helps, and eventually another hire here too. Luckily, sales is generally a lower cadence, and tech support can depressure via an acknowledgement of reciept etc till actual reach out.
Besides that, I've kept things simple and built my own tools around deployment - so making quick changes/fixes are easy for me as a team of one.
I'm refreshing the docs site [2] right now, which I hope will reduce the number of emails I get and make it easier for customers to get the answers they need (even to questions they didn't know they had) themselves.
[1] https://PlotAPI.com
[2] https://PlotAPI.com/docs
1) build for limited/easy observational/maintenance. Adding observers and maintaining systems is time you could spend elsewhere. 2) favor self-healing. I generally do this by using severless services -- code won't suddenly execute differently in a new lambda/function or container, so if the data center has an issue, my issue is resolved when there's is. 3) keep your running stack simple. The less services you leverage, the less you are impacted by issues. It may make more sense to use the database for locking instead of redis because that's one less thing to learn. 4) reproducibility of your environment. DNS is almost the only thing I need to set up manually (because the effort to automate making SOA/NS links). But after that, I can rebuild my entire code stacks and make a new dev environment in minutes. I then can load up the recent DB backup. Having this also makes it easier to deploy in multiple regions, if you get to that level of redundancy needed.
I guess my advise is mostly keep it simple so your less likely to have an issue.
For example, the HashBackup website is for just for documentation, and if it fails, having documentation offline is not a huge deal. If it had user signups, accounts, and all that, it still would not be a big deal to be down as long as the installed software continued to work, at least for a few days. And backup software is not "mission critical" to the same extent that hosting a company's tech support website would be, for example. All of these have been very conscious choices.
For example, integrating a 3rd party API? How reliable is that API, what happens if it goes down, is there an alternative API to have on backup, etc? I reinvent the wheel sometimes, if its not too hard, just to decrease external dependencies.
Thinking of allowing user generated content such as uploading images, or writing comments/posts? Prepare to spend a good deal of your time moderating that content and managing spam. I go out of my way now to avoid having UGC on my sites, even when it would enhance the product.
Going to use the latest bleeding edge technology? What happens in 3 years when it's obsolete or the next upgrade has breaking changes? I try to use dumb boring old LAMP with vanilla JS whenever possible. I have several 10+ year old websites using this boring tech stack that are still chugging along that I literally haven't looked at in years. Whereas I have an old angular 1.0 website for which I have forgotten the build step because I failed to take notes and I doubt "npm install" would work anymore for its ancient dependencies, so its stuck in dinosaur land unless I want to do a total rewrite. Keep it dumb.