I haven't tried it, but Piwigo[0] looks promising for photo albums & management. That or Ente[1] although Ente doesn't have a self-hosting option like Piwigo.
If you really want true self hosting you would run it off your own on-prem machine and use your ISP to push & pull content. Putting things on a VPS is not really 'self' hosting as you're entrusting a third party to not get their datacenter burned down, or the hard-drives corrupted, etc
That said, the only caveat to hosting in your own house is it could suffer a fire, and your data is wiped, so having /BOTH/ a VPS and an in-house on-prem solution means you're not putting all your eggs in one basket and you have a contingency plan in place, which one day may be worth it. It buys you peace of mind because of the redundancy.
> That said, the only caveat to hosting in your own house is it could suffer a fire, and your data is wiped
Well, there are other reasons to prefer using external hosting. Home connections are typically port‐filtered, have dynamic IP addresses, and have a low IP reputation, and your ISP selection is very limited. Whereas if using a VPS there are so many options that it’s easy to shop around.
But you can still self‐host while getting the benefits of a VPS. Just forward ports from the VPS over a WireGuard tunnel to your real machine. Then all the actual infrastructure is on hardware you control, and the cloud provider has no access to your TLS private keys.
Yes, and you can even do this quite cheaply. Oracle cloud free tier has a nice traffic allowance: https://paul.totterman.name/posts/free-clouds/ . Add tailscale/cloudflare tunnel/plain wireguard for connecting your home server to the cloud instance.
IANAL but I believe another reason to true self host, at least in the US, is that rules for things inside your house have extra protection. Sure they can still get a warrant, but this is a totally different level than what they need to get the same data off of a VPS.
Do you really have any search and seizure protections on a VPS?
> Do you really have any search and seizure protections on a VPS?
I'm aware of this, which is why I do full disk encryption of any VPS instance I operate. See the Third Party Doctrine[0] which applies to the US only AFAIK.
I am comfortable re-building my self hosting setup from scratch/backup. I enjoy the sense of agency being able to fix something myself vs wait for a cloud service to return. As I rely on my self hosted setups more, I also build in the appropriate amount of high availability features required. You will learn a TON of skills that are sideways related to software engineering. It's very empowering to be nearly entirely self sufficient with your profession. I can write/test/deploy software (ie pay the bills) and never have some critical service or infrastructure carpet pulled out from underneath you(ie dockerhub,github) and prevent you from doing your work.
This is such a niche attitude/market but it has been incredible to see the surge of self-hosted applications/services over the last 5 years.
It is also relatively easy these days with modern ci/cd tools to have a "portable" enough stack that in the event of an emergency you could purchase a few linode instances and be migrated to a vps environment in an afternoon.
The author mentions but doesn't address the Picasa problem, which incidentally is the one I care most about.
What do I do when all the useful software is cloud based and requires me to store my data with the service provider in order to use it? Self hosting is not a solution.
Self-hosting is not always the answer for a lot of people.
Self-hosting are not easy for laypeople (someone who are not familiar with it) to try to get their feet wet with it. For myself, I am on the level of beginner and I do struggle to stay on self-hosting path. When I set it up, I learn there is more steps that I have to do because the documentations and guides did not bother to explain those step and expect me to research more to find the information about it.
My biggest beef with self-hosting is that they expect us to set up the SSL/TLS certificate without explaining the step to set it up. Some guides does have section about it but never provide the details about creating CA for my self-hosting needs. I turn to Google/DDG to find information about it and they are all over the place or leading into dead-end.
There are few others thing I have gripes with self-hosting. I like self-hosting and they are pleasing for me as I don't need to rely on third party solution. The gripes I have is the documentations that are over the place or sparse information about it.
It sounds like part of the difficulty has to do with the general poor quality of online tutorials. There is a need for properly written guide books and magazines, but unfortunately, it seems like there is no way to pay for people to write them.
> My biggest beef with self-hosting is that they expect us to set up the SSL/TLS certificate without explaining the step to set it up. Some guides does have section about it but never provide the details about creating CA for my self-hosting needs. I turn to Google/DDG to find information about it and they are all over the place or leading into dead-end.
If you have your own domain pointed at your server, the Let's Encrypt certbot can automatically pull in a certificate and configure your apache/nginx webserver (alternative webserver caddy has this feature built in as far as I know).
If you don't have your own domain, don't go with self-signed certificates. Get a free https://desec.io/ subdomain, and they have their own certbot plugin to generate automatic certificates.
A DNS misconfiguration can cause your Let's Encrypt to do weird things on a configuration that was (and still is) perfectly correct.
That was how I hit it. I eventually figured out what people screwed up in DNS. But certificates still didn't clear. So I spent an extra couple hours staring at DNS trying to figure out what I missed when the issue was that we bumped into the rate limit at Let's Encrypt (which is REALLY low--I think 5 failures is enough to trip it) while the DNS was bad and the only thing we could do was sit around for a week with dead certificates.
I've hit the problem you describe, and I feel your pain. I also respect LetsEncrypt's choice to rate limit failures. I renew a couple dozen domains at a time, so one error can quickly cascade into being blocked. IIRC the block timeout starts at 24 hrs and goes up from there if you keep trying -- this is easy to do if you don't see the raw response error message!
After being bitten by this a couple times, I added a dry-run step to my autorenewal script. If the dry-run exits with success and generates a good new cert for the domain, I repeat by pointing to the LE prod server. This works every time (so far, but for years now).
I'm suggesting that any LetsEncrypt certificate automation system (or docs) targeted at relatively low-sophistication users (i.e. not you or me) should include this sort of dry-run check so that the user doesn't paint themselves into a corner with a somewhat persnickety, but essential, service.
Also of course, it should attempt to renew after 60 days, so that if things go badly wrong, there are a few block-timeout retries available before the 90 day expiration.
If you use Caddy, you'll almost never run into rate limits from Let's Encrypt, because Caddy rate limits itself, and will fallback to ZeroSSL instead of Let's Encrypt, and even fallback to LE's staging for additional retries against LE before trying the live one again if it works with staging. See https://caddyserver.com/docs/automatic-https#errors
I think the whole "self hosting isn't easy" meme gets repeated so much that people just take it as given now and default to managed software. Or, someone might argue "Well, my grandmother who knows nothing about tech cannot self-host, so it's not viable!" ignoring there is a huge spectrum of competence between grandma and a seasoned Linux sysadmin. People aren't morons, and there's enough info out there on how to do it. I agree it's not organized very well, but it's not like setting up a web server is dark wizardry.
With all the tools out there and easy access to VPS services and even bare metal for your basement, there's never been a better time to self host. And not just web servers, but E-mail, git, photos and media, and so on, it's very accessible.
The complaint is fair though. Trying to find a complete or the "correct" guide to something is very difficult even when you already know roughly what you are doing.
I took me ages to work out how to setup postfix properly from about 10 slightly different "guides". The Postfix book wasn't even that helpful. There are also lots of very out-of-date guides that might have been OK for 2015 but not anymore. They don't get deleted because "link juice"
It is sad but true but you get one little bit wrong and you potentially leave a door wide-open.
Postfix is a special kind of hell though, in that getting a good setup requires wading though decades of legacy stuff and patching together a bunch of non-default stuff to get, for instance, dkim signing and stuff working right. I've done this before myself, and agree it was super annoying and not fun, but I also think it is potentially the biggest outlier in self-hosting difficulty I've encountered.
Lots of services are barely more than - apt install, systemctl enable --now, ufw allow 8080 (if you even firewall within your network).
I actually found Postfix fairly easy to configure once you have a solid understanding of Email (which took me a good while at first). Dovecot on the other hand...
I agree it's overblown. It's amazing how robust of a setup (more than sufficient for residential use!) you can get with little effort given how easy things are nowadays.
I've been self-hosting a lot of load-bearing household stuff (I have stuff on the "wife-critical" path: if it goes down, "the internet goes down" and I get a text from her) for almost 10 years and I've only had 2 incidents of particular reputational-risk note:
1) a routine reboot of the main server triggered a BTRFS bug that blocked mounting it again. This took an evening and a reboot into an arch linux ISO to fix (arch had a new-enough version of the btrfs tools that had the ability to fsck/repair the fs).
2) my proxmox setup was initially installed with zfs and zfs-on-root. This exploded and the "on root" part stopped working one day. This was the most annoying thing to fix so far because I ended up dumping any interesting data to an external HDD and just re-paving the server, this time reinstalling with just ext4 and lvm (which is admittedly a setup I'm much more comfortable debugging). No issues since then.
Both these events are from over 3 years ago, so it's been smooth sailing in recent times.
I'm skeptical that your layperson would be able to keep self-hosted applications secure constantly. Hell, huge corporations have a difficult time with it.
I have this issue too. When I tried to set up self-hosting, I assumed that there are steps that requires me to expose it to the internet. Turn out that it already exposed and didn't (or barely) provided the information of how to close it off securely and keep it private network only. When I tried to find information about it, there was always guides that are not consistent with it. Some will say I have to go in php.ini to do this, then go to SQlite to do that, then go to other files do there, then adding 20 steps to keep it secured. I'm just wondering why there are not any centralized options to do this. I just want a option that I can tick in the software and left it off as that.
I understand those documentations are not for laypeople for me. However it is annoying when people out there kept pushing the self-hosting for beginners narrative without providing the necessary tools for laypeople to keep themselves secured and reliable.
>I understand those documentations are not for laypeople for me. However it is annoying when people out there kept pushing the self-hosting for beginners narrative without providing the necessary tools for laypeople to keep themselves secured and reliable.
And that, in a nutshell, is the problem.
A few clicks, a configuration form and integrated tools to set up external dependencies (i.e., LetsEncrypt certs), et voila! You're running a self-hosted application.
AFAICT, this is more about developers not creating the packaging/configuration/management tools necessary for effective use by non-technical users.
Sure, I can write a sql query to modify the schema of an applications' database, but my highly educated and intelligent physician brother would just throw up his hands in disgust.
Make self hosting easy and people will use it. And Docker-compose isn't "easy" for a lay person.
Majority of the documentations I came across usually have the mantra of "Do this and you are golden". I know it is not dark wizardary, it just the documentations are aiming for someone who have the experience and the technical knowledge of this. Whereas there are people who are pushing "self-hosting is the answer! Even your tech-inept grandma can do it!" without providing documentations for inexperienced people like me. Annoyingly that some guides have parts that have a links to other guides that barely provide information about this. It is like "I know how to set it up but I am not gotta tell you how to do it, so here the link that might help" and it didn't help at all.
When I have begun to install and manage servers, more than 20 years ago, I did not have any kind of prior experience and I did not have anyone whom I could ask.
So I have just read the handbook, but I have read it completely, which needs more than a day.
It is likely that there are also other operating systems and Linux distributions that have good documentation, but I can testify only about those that I have used in the beginning, the FreeBSD handbook and then the Gentoo Linux handbook.
Both handbooks were good enough to convert anyone into a system administrator.
Unfortunately, both handbooks are not as good in 2022 as they were e.g. in 2002, because they have not always been updated after every change, or the updates have not been as detailed as the original parts of the handbooks.
Even so, both handbooks remain reasonably good today.
Especially the FreeBSD handbook is good for someone who lacks experience, because FreeBSD is much more self-contained, i.e. there are a lot of choices that have already been made for you and you do not have to worry about them.
So for someone who is inexperienced, I believe that the fastest way to managing a server remains to read the complete FreeBSD handbook and install and configure a server based on that.
There are programs which are available only on Linux, but the administration of a Linux server requires much more work than for a FreeBSD server (even if much less than for a Windows server), so for a beginner I think that FreeBSD with its more complete documentation and less possible choices is easier to try.
To do this right you should also think of backups, updates, and monitoring. Self-hosting is true freedom but doing it right for things like email is akin to running a small business. On the positive side docker makes many things a breeze.
I tried with Docker before and it is not a breeze as you think it is. I tried to use Docker for Calibre-Web and it is a pain to make it work. Because Calibre-Web required to access their database in the filesystem outside of Docker. Docker provided minimal (more of lacking) information of how to expose the filesystem for Calibre-Web to use their database. Calibre-Web cannot create their own database, it relies on Calibre, standalone app, to generate the library that it need to have access to. It took me ages to finally to find a way to expose the filesystem and only provide permission to access that particular library.
I am surprised by this shortcoming of Calibre image. I guess the trade is learning how to install calibre vs. learning how to deal with docker. I'd also agree that even if you use docker and installation is easy - for any self hosted apps you are using for a long enough time you end up learning enough about them to be able to install without docker (and avoid managing docker in addition to everything else).
Calibre does have their "book sharing" solution that are built in the software. However it is more like content server. Calibre-Web is a third-party solution that are not affiliated with Kovid Goyal (creator/main developer of calibre). And Calibre-Web is basically browser version of Calibre without requiring other people to use Calibre to access Calibre-Web. So Kovid did not create this calibre-web image from what I understand.
This was my single biggest hurdle when I was trying to set up a personal VPN to remote manage things- I didn't really understand what I was supposed to do, or why things just never quite worked.
That's a good post on the topic, thanks. Like a lot of others I'm a hybrid-self-hoster. I do rely on some third-party, third-party-hosted or other cloud services, but I also spend a lot of time bringing things back home when I can.
It's tricky to be in that hybrid-box since the conversation in this area is very dichotomous--cloud things OR my own thing--but overall I like keeping my options open and swimming with the herd ;-) in making sensible use of cloud services when it seems appropriate.
I think the granularity of control is just as important as where the app is hosted imo. Its perfectly valid to make a fair compromise on ease of management vs. being able to vendor your own versions. And especially with how great Tailscale/Wireguard networking is nowadays, you really can make that line blur between your own network + a cloud provider.
Should someone interested in self-hosting do it from a literal PC in your basement, configured as a server?
Or is self-hosting on AWS / DreamHost / whatever good enough?
I ask because I like self-hosting a lot, especially when market solutions don’t really do what I need them to.
But security, man, that worries me. I can’t tell you what a three-way handshake truly is, or what a signed certificate really means: so self-hosting my own email / web server / etc. from my basement gives me a fear that someone, somewhere will take advantage of a vulnerability in some system component that I’ve never even heard of.
Even better do you really need "self hosting" many people will be good enough with external drive.
You can also setup something like Synology which is good enough for layman and if you keep it in your local network it is basically easier than configuring some old PC.
> Should someone interested in self-hosting do it from a literal PC in your basement, configured as a server?
It's a good place to start/test. But don't open your firewall: do all of your testing on your internal network. You really don't want to open your network to the kind of problems that can occur while you're learning.
When you're ready to really host things then you should rent a cheap shared instance, or maybe a low-priced dedicated server. You can pick up something decent for $10/mo. That's not much if you're skilled enough (eg, employable enough) to learn how to self-host.
For your internal network you can use a pi-hole to set up all of your DNS entries so you can even visit "http://example.com" and have it point to an IP on your LAN.
For some things your local network is enough, like personal pictures and other private files. E-Mails I would suggest to host in a datacenter. Not necessarily in AWS but a local company offering hosting.
For those who feel unable top securely self host I'd suggest looking into smaller providers of hosted E-Mail solutions. A large number of federated services is better than everyone being on Google Workspace or MS360.
I self-host entirely on a Dreamhost VPS, precisely because of the issues you mention. I'm fairly experienced with many of the more technical aspects, but Dreamhost is more diligent than I am, and they stay abreast of issues I'm unaware of. So I handle the app layer (Nextcloud, FreshRSS, Fossil, etc.) and they handle the OS, web server (Apache, PHP, etc.), and certs (through Lets Encrypt). This balance has worked really well for me. No affiliation, just a customer since 2004.
I would not encourage someone who completely lacks experience in server/network management to do self-hosting, as it is easy to make mistakes.
Nevertheless, if someone is willing to dedicate some time for study and experimentation in the beginning, this is not an insurmountable problem.
I have been using self-hosting on "a literal PC in my basement" for about 20 years, without any problems whatsoever, and with negligible costs (the main cost being that I have a set of public IPv4 addresses and a fixed IPv4 address on my router connected to the ISP, which implied a more expensive monthly fee for the ISP).
After the first few months, during which I have made frequent changes in the configuration, while I understood better and better how it should work, the time wasted with server management during the next years has been negligible, i.e. just a few hours per year, used mainly for software or hardware upgrades.
Configuring and managing services just for personal needs or for the needs of a small number of users, e.g. a family, is much simpler than in an enterprise setting.
For reliability, it is good to have a second spare computer and a second image of the root SSD/HDD used on your server, to be able to replace the active server in case of failure. As others have already mentioned, periodic backups should be done and they should preferably be stored in a different location.
While I believe that self-hosting is not difficult, unless someone has already done such management work as a professional, it is necessary to learn many things.
For security, the first thing needed is to understand well what a firewall does, which are the firewall rules needed by whatever services you want to host and how to configure and monitor whatever firewall program you choose.
For this, some knowledge about how the main IP protocols for networking work is necessary.
The management of keys and certificates is also important, as you have mentioned, but what you need to learn for this is much less than what you need to learn about networking protocols, in order to both make a correct server configuration in the beginning and to diagnose any problems that might appear later (usually because someone at your ISP makes some changes in their configuration, which break yours, but nobody who answers the support call has any idea that they have changed anything, so you should better be able to identify yourself what they might have done, if you want a quick solution).
What self hosting stories don't seem to focus enough on is backup and encryption, as these are the main issues with server-in-your-house hosting. Even disregarding fire/water damage it's not uncommon to have hard drives die outright, which is a problem if you didn't think to (or had the money to) set up zfs for data redundancy purposes.
I agree coming up with a good backup strategy is an essential ingredient to long-term-sustainable self-hosting.
Speaking for myself, I don't have the goal of 100% detaching myself from "the grid", so to speak. I still want to pay an ISP to act as a gateway to the internet, and want to pay the local electric company to power my house.
To me, "backups" are a commodity service, like internet service and electricity.
Dumb file servers are offered by any number of places for a price lower than the cost of in-housing that service, and with a negligible switching cost at for my workload.
I'm personally OK with having one relatively shitty local mirror, and a background task that rsync's to backblaze. If BB makes noises about going under, I can migrate aws s3, rsync.net, digital ocean, whatever entity wants to charge me the least for my workload.
I don't think NAS's or ZFS are strict requirements, although playing with them can be fun.
This is an important call to action, in a world where your user experience of an application is determined by a Product Manager who may be stat-maxxing a graph, I hope that we can see a resurgence of self-hosted apps.
Selfishly speaking, I work at Railway and our community maintains a list of self-hosted apps (we call them starters) that people can deploy to our platform. You can checkout the list of apps here: https://railway.app/starters and we even accept submissions via our GitHub repo: https://github.com/railwayapp/starters (Just reply to me here and we can get it reviewed for ya.)
I'll have to take care of backups, security, availability, updates, etc. I prefer to use a managed solution.
If you don't want to lose data on being banned, just do your own backups, which are by themselves much less time consuming to handle than full-blown self-hosting.
I'm fine with the occasional service being axed, I'll just migrate to another one. Often, somebody writes a migration script and open sources it, making that even easier.
It is good though to promote and vote with your wallet for services that give you good and dependable support.
> I'll have to take care of backups, security, availability, updates, etc. I prefer to use a managed solution.
A hosting cooperative is often a good compromise. You get to mutualize services and maintenance with other people who have the same needs.
On a spectrum from selfhosting to cloud computing, hosting cooperatives lie in the middle when users have agency but don't have to take care of everything by themselves.
Self hosting also implies building (or using) your own self hosted product. That's a significant requirement, particularly if you want social features.
I'm going through this dilemma with books. Goodreads lost my account of nine years. I've managed to recover most of the data from a backup and set up my own blog. I'm self hosting! But my blog is very spare and is not backed by a database of books, book covers, etc. Also it has no social features, no easy way to see other people's reviews or find related books or... I could imagine building all those things but that's like building a whole product! I could also imagine some self hosted book product I could just use (analagous to Picasa in the story) but it doesn't happen to exist.
Meanwhile there's a pretty great product for books in Goodreads, other than the crippling disaster of losing a user's account. Also some good cloud competitors like The StoryGraph. So maybe I should just use their product and hope my data is safe.
PS: I was at Google when Picasa was acquired. My memory is that the plan was always to focus on the hosted version. Maintaining a desktop standalone product was very much not in the Google business model.
Maybe I didn't explain myself well. Yes, I could get a data dump from many sources. It is a lot of work to turn that dump into a product that I self host.
You don't have to write that stuff. There is a fairly well-known project licensed under AGPL3, that's fine for self-hosting if perhaps not commercial use. Just search around.
We're not quite publicly launched yet, but I've been working on making self-hosting easier for several years now. People often ask "why would I self-host?" and it's hard to pin down one answer - instead the answer depends on your values - but there is an answer. This post is excellent because it's not "do it for security" or "do it to see fewer ads" or "do it to fight big tech" or "don't give photos of your infant to Facebook". It's all of those reasons, but it's also more broadly (and deeper in the kool-aid), because it helps fix the internet itself.
> This engineering talent is supposed to be solving world’s problems but instead they are ensuring how everyone wastes their time
Agreed! If software was sold for its utility instead of its addictive properties - this might start to change. Self-hosted / open-source software does need plenty of "hosted" accoutrements though: backups, remote access, etc. Shameless self-promo: we're trying to solve this over at https://kubesail.com
Dismayed with the brittleness of Pinboard and the bloat of most alternatives I turned to self-hosting an excellent bookmark server called linkding[0] on a Raspberry Pi. Very happy with the result.
I've been pretty happy with my local Unraid server. I have a few things running on it, including Plex for my music library and Nextcloud for notes, file storage, and automatic photo uploads from my phone.
The software and Nextcloud data are all on an SSD, but the Nextcloud data gets a nightly backup to a mechanical hard drive. The music doesn't have any backup, but I could always re-rip the CDs if I had to.
I run a few services from my home but still have to rely on aws/fly.io for some portions of my infrastructure.
I really want is to learn how to rent rack space from a colocation. The documentation available does not make it easy to learn. Can I just buy an old 1U blade, throw xen on it and show up at my nearest colo? What do I need to preconfigure to ensure I have remote access without giving remote access to the colo as well? Do I get physical access to the data center?
Wish I could find some guides on this topic. 95% of blog post tutorials are just ads for the latest trendy cloud startup/language framework.
I did this once. Don't overthink it too much - yes, it is as simple as finding a rack with sufficient space, power and network, plugging it in and going. You'll most likely get a public IP and have no access to your neighbors, so they won't really care what you do with it as long as it's not illegal or against the Terms of Service for your host. So yeah, if you want to do it, just do it. Get an OS you know, install an SSH server or Remote Desktop, and rack it up. If you can get to it on your LAN, you'll be able to get to it on the public Internet. Also, quickly learn about good auth and firewalls and fail2ban.
That all said (and said with the clarity of age and knowing I was a stubborn kid who did things "because I could"), the experience of spinning up a VPS today on Linode or Digital Ocean is effectively the same, infinitely cheaper, and a lot more fun than racking a server somewhere. I can script up a fleet of servers from my bed at 1am just because, and can't tell the difference between SSH'ing to them versus that one box I did 15 years ago. If you want to do it, go nuts and have fun, but you really aren't really missing much over conventional VPSes these days.
I gotta disagree with you though on cost. You can get a beefy refurbished dual Xeon blade for a couple hundred bucks. Rack space where I live is like $50/month for 1U and gets much cheaper/machine as you scale up. $50 on aws will get me maybe 1 medium ec2 instance and an s3 bucket. With a used blade I get 20x the compute for the same price.
You're overestimating AWS/cloud costs by a decent amount.
t3a.medium is under $30/month and that price only goes down when you reserve for the year. Save even more if you can run your service on ARM/Gravitron.
A VPS service like Linode will have even better pricing than AWS.
Driving over to a data center isn't free (time and cost), either. Those used Xeon blades are cheap for good reason – the companies that originally owned them consider them EOL. There's "no such thing" as dealing with hardware failure (except for occasional stop/starts) in the cloud.
You’re right about the time not being free and certainly hardware failure. Although HW is way more reliable than people let on.
But aws is extremely expensive per unit of raw compute performance. Obviously I concede that the stability and availability is unmatched compared to DIY solutions.
For my specific workload I need lots of memory and cpu cores. (Lots meaning my single 32c/64t 32Gb Xeon tower in my home). I almost fully utilize my resources and STIL need to pay aws for storage/tunneling/DNS/etc.
Consuming equivalent compute in aws would be hundreds of dollars per month.
You're quite welcome. I'm not trying to dissuade you, just provide a point of view I've got from having felt the same way.
I'm definitely not comparing to AWS, because yeah those can get super expensive, super fast. What you're paying for with AWS is Amazon-tier stability (whatever that's worth these days), but the difference in uptime between them and a Linode is more than fine for my needs.
With your $50/mo, make sure that includes power - Xeons eat watts. Also, be sure to compare apples-to-apples on bandwidth. By comparison, Linode's dedicated CPU plan (not shared, closer to bare metal, but still not) starts at $30 for 2 CPUs and 4GB of RAM and 4TB of transfer, and they'll take care of keeping you on the latest hardware. Again, I don't want to dissuade you, because colo is fun and it's cool to think about your own box out in the world. If anything, I'm envious of how easy it is nowadays compared to when I drove 3 hours to South Bend, Indiana to colo a box of my own, or the first time I needed to engage remote hands because the box got in an irreparable state.
If you have a cabinet, and neighbors are caged to prevent your access to those, then you may get physical access. Call a small provider near you and ask.
Many rack space rentals will not permit you to just install whatever PC you fancy because it is potentially a risk to the neighbours in terms of fire or bad hardware, most will happily quote you to buy one their approved ones!
It is pretty easy to get a rack space provider where the provider cannot access the machine but this can be good or bad. In some cases, I would rather they could shutdown the host if, say, the RAM is broken and replace it but if you would prefer to do this yourself, that is fine.
In most cases, you will be given a public IP address directly mapping to your machine via a router/nat lookup so whatever services you open on your machine are open on that public IP address so pretty easy to setup RDP/ssh/whatever.
Probably the biggest issue though is the extra work or hassle if something goes wrong. I remember at a previous company where some guy would frequently have to drive for 30 minutes each way to go to a data centre to perform certain updates that couldn't be done remotely.
> Many rack space rentals will not permit you to just install whatever PC you fancy because it is potentially a risk to the neighbours in terms of fire or bad hardware, most will happily quote you to buy one their approved ones!
I have _never_ experienced this. The only restrictions I've seen on colo contracts I've gone after were related to UPSes and things with large batteries in them. So a big stack of laptops would be a no, but if I wanted to put Atari ST's or Dell PowerEdges or white box builds or bitcoin miners it doesn't matter. I guess I've always done things at at least a half or full cab, never single Us at a time.
* you’d have to sign up with a colo provider first. since data centers in physical buildings, this just depends on where you live
* when you sign up with them they provide you with info like ip addresses or how to connect to their network (they might have dhcp, or you might have to configure static ips). usually there is a initial setup fee, around 1 month of rent.
* if you just rent a a 1U space you usually can get physical access to it while accompanied by someone working for the data center. usually this is during business hours, but each data center will have its own rules. if you rent larger units, such as a full rack (42U) or half a rack you usually get a key card and can access it 24/7 (this usually involves a phone call for them to remotely open a lock)
With the ones I have used you just click around on the homepage selecting what you want on the server and then pay. Some sell second hand repurposed servers on auction that they will set up for you. A while later you get an SSH login on the server and that's it, your server is running somewhere in a basement/bunker/old mine and you can go visit it if you want but in general you can do everything remote. There is even stuff that can let you see the bootup in bios from remote (Called KVM I believe). Some help you set up backups on the server and help you with setting up programs on the server but then it starts to get expensive.
You can also just rent a space to place your own server but I haven't tried that.
Colocation provider will bring the circuits to provide best-path connectivity based on packet destination. There shouldn't be an additional charge for this. They are incentivized to manage their bandwidth so data transfers fast, as they are likely charged wholesale for fiber availability.
You will likely be charged 95th percentile mbps based on your usage. (Again, "pipe space required" to your needs.) Basically, whenever you're busiest -- 4pm-9pm are popular times for us in the USA.
Some customers limit their bandwidth themselves (like, only allow max 12mbps file downloads, etc.) especially when they have the hardware to support huge bandwidth. Or your colocation provider can perhaps limit max connection to 100mbs or 1gbps if you want.
Power is usually leased in amps. If you go over amps the circuit will break -- at worst case scenario. But typically they get in touch with you and tell you to upgrade.
Also, they do want to know vaguely what your service is. Because you'll likely lease their IPs, they will question you if you do a lot of email (caution for spam), or run a Tor exit node (legal hassles for them in many cases).
I've never worked with a colo vendor that once you contacted them didn't have exhaustive support for "how to we get to the point where we can start billing you", usually including an actual human that you can ask questions.
> It gives you the peace of mind by keeping you in control of your data.
I like the sentiment and the points made, but the author uses this amorphous concept of "your data" throughout and I feel like it simplifies things a lot and conflates many different issues.
Most people shouldn't focus on self-hosting literally all the data related to them. This is a sort of perfectionist mental compulsion many of us on HN are familiar with. You have to decide what data you actually really don't want to live without in the rare event you lose access to it, and prioritize that. For most people, this data is not very complex: family photos and videos, an album by an obscure artist, a game you like to play every few years or hope to show your children.
If you are an activist, or someone creating dissident media, or something like that, you should already be wary of the cloud -- the incentives already drive you to use tools that are secure and self-host when needed.
If you truly don't like the ways the big tech companies are doing things, you should find ways to organize with others and demand change; otherwise you are just modifying your personal habits and thinking you are sticking it to the Man with a one-person boycott.
Self hosting is hard. You need to take care of security, backups, software updates, software installation and so on.
Even on something like a QNAP (which can be compared to managed hosting) this can be hard. Flip the wrong switch and you expose something to the world. Missed a security update: your device is now vulnerable.
While I host a lot of things myself I can understand self hosting is not for everyone.
I tried it but there are so many traps you can fall in, like security settings as mentioned by you.
When i had my server online back then, it was hacked 1 week later :D
I hear a lot of stories like this. I've been self-hosting for a few years out of my home. I have a symmetrical gigabit fiber connection. My IP changes very frequently (DDNS and a low TTL solves that problem for my use cases).
_anyway_
I haven't been hacked.. yet. /me knocks on wood
The precautions I take are basic:
- Use unique and secure credentials on each service I expose.
- I only expose ports 80 and 443 to the public. 80 HTTP redirects to HTTPS/443
- I keep my software updated (docker-compose pull)
- Nightly backups to cloud storage and local disk
- I "airgap" my home network from my hosting network. There is no shared hardware between them including firewalss/routers, switches, etc.
I figure cloud services and SaaS get hacked anyway. I can't enumerate the breaches my data has been a part of. If my self-hosted stuff gets hacked at least I can do the forensics and actually see what happened and what was accessed. With a 3rd party all I can hope for is what their PR department lets out.
Not necessarily. For my use case it’s one extra 4 port gigabit switch and a single pc that runs everything containerized including the NAS, firewall, and apps.
I'm interested in how you set up your home and hosting networks without any shared hardware. I've been running my own websites from home for awhile on their own machines, but never considered they could be on a completely separate network all the way up to the modem.
My ISP provides me with PPPoE into my house. I have that Ethernet going into a small switch which both networks connect to via a firewall. Each network establishes its own PPPoE session and receives its own (dynamic) IP address.
The first hack I noticed was that someone had set a password on my redis server because the default was no password and I had accidentally exposed it to the wider internet. This was exposed for 6 months before this happened. Who knows what else was accessed without me knowing.
It's pretty silly how many services are public by default when ideally they should only listen on a unix domain socket (or nothing) until you configure something else.
I'd love to see a blog post that says, this is how to setup X (I dunno.. mediawiki, owncloud, whatever).. and then go fully in-depth into _everything_ surrounding it.. security, backups, logging, alerting, monitoring, backup testing/restoration etc.. a blog post that really covers everything for a well-protected 21st century hosted application that won't leave the owner in tears after a year!
There's honestly so many posts that make it look so easy, but without everything else that would normally make it a job position in a company :)
I think the hard part is that would be largely dependent on specific implementation, which itself is very opinionated. I could write a post on how I run, maintain, and secure Docker Container X on Ubuntu Y using vSphere with Synology and get 100 comments on why CentOS is better and I'm wasting time/money with vSphere over Proxmox, etc. Cloud doesn't have quite this problem. Once you've chosen a cloud provider, you have significantly fewer options in each category, minimizing this option-overload.
It should start with how to make your system upgradeable too. I've server that started on Ubuntu 16 and made a helluva mess upgrading to 18. Due to php changes i've had to use ondrej's packages for later php... but that will break on a (very overdue) upgrade to 20...
All these script kiddie tutorials are terrible at showing how to maintain a server for years.
This is where docker really shines. Unless you’re a php developer or have a lot of experience with it, gluing it all together is best left to some clever person maintaining an upstream docker image.
Docker and I are not friends. The quickest way I found to fill up my limited VPS hard drive was to install Docker. All the work arounds to limit it failed. Then there's the whole lack of concrete control over iptables, where a tiny mistake can open you up to all sorts of horrors. So, it's great that Docker works for many, but I get the exact opposite of warm and fuzzy for it however.
Perhaps Nix could help? It's great for configuration and reproducible builds without the overhead of containers (which in practice aren't often reproducible).
Docker is not a good solution.
Many security focused systems (important for self-hosting) are on FreeBSD. FreeBSD doesn't allow docker, because of it's major security vulnerabilities.
Docker is great for getting toy projects to work somewhere as a last resort, if the dependencies are strange and you need a convoluted (read: badly thrown together) environment to set up the app.
A well made application should not need docker to run.
"Flip the wrong switch and you expose something to the world."
One strategy for dealing with accidental misconfigurations is to employ a "network slug"[1]:
"A Network Slug, or "Slug", is a transparent layer 2 firewall running on a device with only two interfaces. ... The purpose of a Slug is to reinforce a security policy or to block uninentional leaks of information."
I think all those are anti-features on a network slug.
As I understand it, the device is intentionally simple because it is there to ensure some misconfiguration cannot expose some port that should not be exposed.
I have implemented firewalls similar to this in the past. They typically had three network interfaces. Two of them were configured as bridges and then I use ebtables/iptables to filter traffic flowing through. These two interfaces would have no IP address and would not be visible on a traceroute, etc.
The third interface would only be connected to a separate admin network. Or it might not even be plugged in. In the latter case, the admin needing to change anything on the device would have to be physically present and bring a "crossover" ethernet cable and plug their laptop directly into the third NIC of the firewall. From there, they would be able to ssh into the firewall and change config.
A network slug does not have an IP address. You cannot connect to it over the network. I'm not sure you understand what the device is and what it does.
Let me give you an example - I have a "port 22 slug" and what it does is block all traffic of all kinds except for TCP22. That's it. It does nothing else and it does it transparently without having an IP address of its own. If I wanted to reconfigure it, I would connect with a serial console.
I have never head this idea described in text before. However, I have made firewalls this way for decades.
They were typically for stuff that ran in a datacenter so it would be a 1U server with three NICs.
I would really like to make such devices for home or office use. What would be a good device to use for this? Unfortunately, RaspberryPIs do not come with 2 or 3 NICs.
Any recommended alternatives?
I would have a look at the openwrt project’s database of supported devices. You can filter for devices with 3 nics (though not sure it supports filtering for “3 or more”).
use VMs. qemu/kvm. the Tor-based Whonix OS takes the approach of one VM running a Tor proxy and another VM running your application software. the latter VM only has access to that proxy, and no other network interface. it’s effectively the same approach as i understand a slug to be, but with the hardware virtualized instead of physical (or course you don’t have to use Tor — you can define whatever interface you want: a VPN, a firewall, etc).
Easiest solution is to just host stuff on a local network without access to the wider internet. E.g. running on an old laptop/raspberry pi/server in your basement.
Sure, that means you can no longer access your self-hosted stuff when you're out of the house, but the tradeoff is peace of mind about your data leaking or worse.
What I mean is just the general statement that can be applied to any company or server.
Even if they release all of their code as free and open source (not entirely true, as I understand it - their public servers that handle connecting your devices is not), there is no gaurantee that they are actually using that specific, unmodified code on their servers.
They could add or remove whatever they like to the software before deploying it on their servers without you knowing.
You have to trust whatever they may write on their blogs. Some people may not care, but the winds these days are shifting away from trusting companies with their private data.
That helps for external threats breaking into buggy network services, but it doesn't help for compromised apps/images/dependencies exfiltrating your secrets.
If it's an air-gapped local network, then sure, but how useful is that? Are you disconnecting your phone/laptop from the internet when you access the air-gapped network, or do you use two network interfaces on every device?
I assumed the GP was talking about a typical home "local network", one behind a NAT - so no incoming traffic, but usually, it allows any outgoing traffic.
> Sure, that means you can no longer access your self-hosted stuff when you're out of the house, but the tradeoff is peace of mind about your data leaking or worse.
Lots of things I'd consider self-hosting are functionally useless if I can't access them from my phone while out and about.
I could put my phone on a VPN, but that's just another layer of complexity to add to the self-hosting process.
I do a split approach -- Most services are available internally only, some are reverse proxied out. It used to be caddy2, but after a recent issue and switching to TrueNAS, I just use Traefik with k8s Ingresses and only set it on the few containers I would like accessible.
It does, but I found it to drain a lot of battery on mobile (iOS). I'd say that for a simple setup you're likely to be accessing your services via their LAN IPs or their TailScale IPs, so if you have these bookmarked etc. on your phone then it means you either:
1) Have separate bookmarks for accessing services via the LAN or the VPN, and need to remember to turn on and off the VPN when you need to access things out of the house; or
2) Always access services via the TailScale IP, which means you need to be connected to the VPN on your phone even to access them from your LAN, which in turn either means toggling the VPN on and off even when at home or leaving it connected all the time and letting it drain 20% of your battery every day.
It's a great service but I didn't find it to make accessing self-hosted services from out of the home on a phone to be as nice as using them at home, or using a managed service.
I'm amused by the implications here that 1) the outsourced alternatives are better than you are at keeping up with the 'hard stuff', and 2) that in an outsourced scenario you can't "flip the wrong switch and you expose something to the world". This thinking is why I can't tell you how many incident post-mortems I've done where I have to once again hear "...but, but, but...we outsourced this to them so this couldn't happen...".
Depends on whether you're referring to a SaaS provider or something more like a MSP.
I'd like to believe the engineers running Google Photos or iCloud are spending a lot more time on keeping my photos secure and available than I would be willing to put into a server running in my basement.
In the case of a business hiring an MSP to manage something complex like firewalls, Active Directory, server patching, then sure it's reasonable to assume that if they made a mistake, the impact would be equivalent to you making the mistake yourself.
It's possible you need to tell whomever you are reporting to for these post-mortems, they should be outsourcing to reputable service providers in order to free up time and man-hours, not necessarily just to save financially. I suspect that is the real problem.
Thanks for the condescension about how my clients handle outsourcing. Not knowing anything about how they do it makes that sort of low intellect, zero content second guessing easy.
Big Hint: AWS has had something like 5 times the downtime my biggest clients on-prem datacenters have had this year. So...no, the FAANG engineers aren't doing better than my clients.
To the extent permitted by the hosted service, you should still backup your data. If you manage to accidentally delete all of your hosted photos or if your account is compromised, I wouldn't rely on most services going to their backups to restore your data. Unless it's a site-wide issue, most places will say "that's too bad" and send you directions on how to protect your account.
I agree but I think about it in the reverse way: the hosting is easy, what you get when you use another company's service is the maintenance. Just like every other option where we choose who will maintain something there are trade-offs. You can maintain your own car if you want, but it'll involve things! We all look at our lives and decide which is best for us for each thing.
Personally, I tend to self host the things whose maintenance I at least find satisfying, and hopefully enjoy. Otherwise I pay someone (through ads or my own money) to do it for me.
I used to love running my own servers with all the services etc. I’d manually write beautiful bash scripts to keep it all nice and easy to rebuild on the fly. My first job had 10 Ubuntu servers (on site) and I was the only guy who used Linux at home and had experience with sql.
I have never volunteered to maintain servers since, it was horrible and everything was always my fault (it kinda was, I was a hobbyist at best with no real production Linux experience.)
I do still end up as the dev ops/infra guy at every place I’ve worked but at this point I’m probably one of those stubborn senior guys who wouldn’t like the way the juniors went about it.
Sounds like you might've had an unusually bad experience. Might've also been the distro; I don't like Ubuntu much myself. :P
Maintaining inherited environments is also much more painful than ones you get to design from the ground up. I work with varied environments, and one with ~250 RHEL / CentOS machines has approximately the same level of maintenance burden as another with a dozen or so Ubuntus because the first environment has had configuration management from the beginning and the second is a complete mess that I've slowly tried to reverse-engineer and clean up.
When your change management works, maintaining a dozen servers isn't all that different from maintaining a thousand or more; and the need for change management and automation doesn't really go anywhere even when you don't self-host things.
I like RHEL and derivatives more for servers myself. It's probably just preference, but I find that RHEL-like distros step on my toes less often. In particular, I don't like debconf at all, and Ubuntu pushing snaps everywhere also leaves a bad taste in my mouth.
Yeah I tried self hosting everything. Getting it actually running is the easiest part. Its the maintenance, backups, and security that are 90% of the job. You can get it working pretty easily and forget about it and it will run for a while until something goes wrong or it needs to be upgraded.
Now I'd rather leave hosting to a someone dedicated to it who has internalized the latest state of things for all the relevant bits of software and is constantly keeping this knowledge in their brain. Set and forget self hosting can't work in the current environment we have where things require constant security updates and complex security hardening.
I used to backup to external drives. Now I use bare ones since finding big externals got difficult.
I use (and probably abuse) docker compose. K8s is great but compose is easier.
I use a single makefile. Kinda ugly but it's fine.
Bunch of friends and family use my "services". They usually chip in for hard drives and stuff.
I have a few central points of failure but it keeps things easy. My uptime still beats most big clouds - though I have it easier.
I accidentally took down my server for a few days from a botched hardware install. It's a bit funny because now we realize how critical the home server has become to us.. on the other hand, already got the spouses blessing to build a backup standby server.
I've recently started running unraid at home on an old desktop PC and it's really nice. I've also migrated my unifi controller, plex server and pihole to it and it's very easy. Way nicer than the previous setup where I had random dedicated devices each needing their own type of maintenance (unifi controller on my gaming pc needed me to download/install updates manually, plex server hardly received any updates running on old windows laptop and I was always worried about breaking it, and I almost never looked at the pihole running on a rpi).
Now I have a single dashboard and can upgrade each container with a single click, and everything stays on the happy path.
It has also gotten much easier. For instance running your own full blown email server with docker-mailcow. There's a great UI tool that helps to setup the required DNS records. I remember doing the lengthy postfix + dovecot + SASL + MySQL + Auth + this + that guides. No need for it anymore.
> Even on something like a QNAP (which can be compared to managed hosting) this can be hard. Flip the wrong switch and you expose something to the world. Missed a security update: your device is now vulnerable.
It doesn't even require actively flipping switches, but can be from not knowing a vulnerable feature was enabled by default. My QNAP got hit with ransomware because of a vulnerability in the cloud access software that I wasn't even using. I've since locked down all non-local traffic.
Wanted to reply saying the same thing. I didn't really muck with the settings on my QNAP NAS and then checked into my files one day and everything was encrypted with some txt files telling me to send BTC to some address. I just formatted the disks, lamented not backing some stuff up, and moved on.
I'd say the point being: I'm a software engineer who knows better about these sorts of things and still got caught with my pants down. You have to be very judicious with respect to security. You can't just plug and play and say "I'm too busy to worry about that."
Another thing I'll add is the amount of software tools they have on these NAS machines strikes me as 1) very impressive for a company their size and 2) a huge surface area rife for being hacked. When it happened I wasn't surprised at all.
I've since stopped using it because at the end of the day I'd rather pay Dropbox to have peace of mind.
You can use a popular Linux dist and turn on automatic updates, and use Snap apps that update by themselves. But you still would not have control - apps could update with breaking changes.
The only way to win is by choosing simple tools that are either considered "infrastructure", or simple to build and even patch yourself if needed.
This one has bitten me hard on servers and desktop computers. And lately on mobile too. The last area were I still had automatic updates enabled.
The problem is, that one can't reasonably wait a few days on every update and look online for breaking changes. Especially with mobile apps that have sometimes a really unreasonable update frequency.
I still have not found a satisfactory solution for me personally.
For me the issue is that I now have (let me count) 15 different devices in my household with unique configuration needs that it’s up to me manage. I could handle it when it was 1, 2, 3. Now it’s just too much.
I recognize that this embarrassment of riches is in part my own fault. But this is my answer to your “why”
As someone who used to have a server in my dorm room but switched to outsourcing it I stopped because the list of technologies I had to keep track of kept monotonically growing and I had no interest in making it my day job.
If it becomes simple again I would gladly self-host.
If you really want mainstream adoption of self hosting then you need to stop calling it self hosting and rebrand to "personal cloud". The ease of use of cloud software includes zero install, zero management and consumption based pricing. Desktop and mobile had hardware packaged with software and a simple install mechanism with ease of use as a staple for mainstream users.
Self hosting has zero standardisation around hardware, software, install mechanisms. It's a Dev led movement that has everything to do with control and ownership over ease of use. You want mainstream adoption of self hosting. Rebrand it, standardise it, make it easy for non devs.
That's because its a product by western digital. No one wants that. Let's put it like this. Cloud 1.0 was infrastructure, Cloud 2.0 was services, Cloud 3.0 is personal/private.
I respect Western Digital and think they're trying their best to do a good thing. It's that word in general though. Buzzword paradigms always make me feel unwell. As someone who's usually a ahead of the herd in terms of adopting tech, once the broader public catches on and starts making up jargon, I always get a sense that it twists the meaning I personally associated with these concepts and causes me to feel negative emotion about parts of my work life once tacitly normal.
Everything starts out as a buzzword but it's only because it's trying to distil down an entire category into a word. As much as you may dislike it, every industry is quite literally built and defined that way. Something has to be a hook, even if you can explain in detail what it is. Cloud is just this idea that everything goes to a remote place that appears as one thing, which you don't control or manage. In all these trend setting new categories you either play the game or lose out and get left behind as a relic aka like WD, IBM, Seagate and everyone else.
I come from an era of datacenters, colos and whatever else but I learned to adopt new terminology. Cloud isn't just a datacenter, it's all the services on top of it that exist remotely. All the things you don't manage. All the services you make use of. Anything you are not personally installing is in the cloud. That's how we've come to know it and that is the language the mainstream user knows. Just as we'll have difficulty accepting the rebranding of the internet to the Metaverse, it will be a thing that spans far beyond network connectivity.
Who's we? In many countries Metaverse is the Internet by your definition. Lots of PR money has been spent making that the social truth for probably billions of humans. I'm sure everyone over in those continents who likes to use the actual technology that underpins the buzzwords is being force teamed too into thinking they're a dinosaur for not accepting Facebook's dominion.
You know what would be kind of neat? Like, a web site you'd go to called makemeoneofthose.com, and you'd click some buttons, and then sometime later you'd have a hosting setup that you own with some software, web server(s) and database(s) on it, and then you can go hack on it yourself, add some features, whatever. Like they send you some AWS keys and say "It's all yours. Good luck and don't forget to pay your hosting bill."
And now you have a blog, a picture-sharing thingie, a bulletin board, a whatever.
Maybe there could even be a version where you pick a datacenter and somebody racks up a PC for you with the software on it.
cPanel isn't "cool" so it doesn't get a lot of credit here, but it is actually an amazing product that solves real problems. It makes running a server -- even hosting email -- almost effortless. Combined with a decent host, you don't need to have much technical knowledge at all. It really does make running your own server accessible to many, many people who would otherwise be unable to do it.
Additionally: Setting up PHP/MySQL applications on these servers tends to be "upload files, load page" level simple, and cPanel hosting is still generally a fraction of the cost of modern "cool" cloud products.
Sure, I have some neat modern things I'd like to do, but I also have a shared hosting that's been doing it's job for pennies since 2011.
Seems like you could do this pretty easily with a Docker image and a config file. Actually, I've done this with AWS (use a pre-existing image to get some open source wiki software up and running, which I then customized)+
We tend to underappreciate the importance of time in everything. A button click can instantiate something powerful (and useful (and easy-to-use...)), but it will degrade over time, and eventually flat-out stop working.
I had a stack that worked just fine for my own needs, but it ran on shudder Python 2.7 -- everyone knows how that worked out (I chose to rebuild my stack on a different platform).
> A button click can instantiate something powerful (and useful (and easy-to-use...)), but it will degrade over time, and eventually flat-out stop working
Software doesn't degrade over time (other than, you know things like cosmic ray bit flips, but in most realistic situations that should be fully mitigatable.)
The needs of the software user (including hardware and software they want the piece of software to interact with) may evolve, but that's different than software degrading over time.
> I had a stack that worked just fine for my own needs, but it ran on shudder Python 2.7 -- everyone knows how that worked out
While there's no further first party support for that version of Python, if it worked properly before, Python 2.7 and the software running on it probably still works properly now.
This comment was brought to you by someone who never produced/maintained software that had to withstand a 24/7 onslaught of automated exploit kits and port scanners over an extended period of time.
Sure, but my old Google cloud apps on python 2.7 will one day get rug-pulled and forced to upgrade. It can only stay working forever if the platform doesn't change underneath it.
> Sure, but my old Google cloud apps on python 2.7 will one day get rug-pulled and forced to upgrade
“Degradation over time” was being cited as a reason not to self-host. Pointing out that not self-hosting exposes you to risk of others changing the environment so it no longer supports your software is a diametrically-opposed argument.
I would absolutely use "degrade" to describe what happens to public-facing or Internet-connected software over time—eventually you'll have to upgrade it for security reasons, and you'll often find that this is way more involved than just upgrading the server-side package itself, or even its immediate dependencies. The alternative is even more work back-porting security patches. All this is assuming someone's actively working on the software you're self-hosting, at least enough to spot, advertise, and fix vulnerabilities.
Ditto the average Rails/Python/Javascript project, as anyone who's tried to resurrect one that's gone so much as six months without being touched can attest. Which might not matter except that a ton of the software people might actually want to self-host are in one or more of those high-entropy ecosystems. Extraordinary levels of care and organization on the part of the creators and maintainers can mitigate this, but that amount of taste and effort is vanishingly rare.
These are degradation due to a changing environment, sure, but I wouldn't describe it as due to evolution in the needs of the user (presumably "must not have any well-publicized remote vulnerabilities" was a need from the beginning).
If your software is not publicly accessible, it may be possible for you to continue running on 10+ year old dependencies indefinitely. For anyone else, other than a hobbyist, it is just not practical.
Otherwise, you are going to be influenced by external factors (security vulnerabilities, wanting to use a feature only available on a newer language version or OS, etc.) If you are a business, you'll also run into more practical concerns, like engineers not wanting to work on a mountain of technical debt.
I have thoughts but not a lot of time - so forgive the terseness. I love the idea of this, but I'd take it further and even have a category in upwork for getting services spun up and maintained.
But that's really the problem - maintenance. Right? Once something goes wrong _for whatever reason_ the user is then (for the immediate needs) just as stuck as with a cloud provider who disabled their access.
Thankfully there is a better course of action - e.g. find someone to fix it for you. Maybe on upwork as well?
But where are you hosting this? Is it AWS? Did _they_ suspend your account? I guess my point is that unless you host on hardware in your house (or another accessible place) you're at the risk of losing access to your data for any myriad of reasons. And even then, there have been warrants where devices were collected and went into a years-long battle as evidence.
This, but they also manage all the updates for me too.
Ideally the only difference between self-hosting and relying on a cloud service would be, I own the servers and therefore the maintainer has no legal right to bar my access.
A lot of hosting providers do offer OSS applications which can be installed with one click, like WordPress or Coppermine. The latter is, I quote:
> a multi-purpose fully-featured and integrated web picture gallery script written in PHP using GD or ImageMagick as image library with a MySQL backend.
And SSL certificates are for free and automatically generated.
But then you have to know how to maintain it all yourself. This is hard. If you already have the knowledge to maintain such a tech stack, that allegedly neat tool would only be marginally useful.
A lot of cloud providers offer this. Cloud ocean for example, you search for the application you're interested in, click lauch and you've got it deployed in a docker container on a remote machine.
I was so sad when sandstorm kind of fizzled out. I'm still hoping Kenton is on a secret mission to somehow bring it to life within Cloudflare. How cool would that be? One-click installs of docs, email hosting, photo sharing, etc apps from a server app marketplace, onto a cloud server you control. (Insofar as you "control" anything on a cloud host, but I feel like that's pretty far, still.)
We've discussed the one-click install thing at some point (not necessarily with Cloudflare), I imagine that's still of interest. There were some issues with the setup process that would need to be addressed first.
Kenton is in the loop and he still has the keys. But, he's busy with other things so he only does a few occasional but vital things.
Or a box in your house, which is where my Sandstorm server lives. :) I think there's a lot of potential for actual self-hosting, though servers like Sandstorm need to have reasonable defaults and make it easy to manage domain setup and backups and security updates, such that one can get a box, plug it in, and reasonably quickly get to "don't need to touch this ever" territory.
We used to host our own software. It was called an application and it ran on your personal computer. We just need that, but running on some appliance instead, like a NAS. Package the service up in something like docker-compose, have a way to sell it, install it, update it and support it. Synology is pretty close with their Docker support, but still pretty far.
You also need stuff like networking, TLS/certs, and DNS which aren't easily packaged, at least not in a way that doesn't require you to make sketchy changes on every client device.
Something like Cloudflare Argo tunneling would work great for this. No certs at all for the user to mess around with, it terminated on the public internet, not in your house.
I think you're misunderstanding the objective. I don't want most of my services (e.g., personal finance, photos, Plex, etc) to terminate at a public IP, that's the whole point of the private network in the first place. So for those explicitly private services, we now need DNS and TLS and in the latter case ideally something like LetsEncrypt so you don't have to manually rotate your certs (but the normal verification methods don't work because your service isn't accessible to LE in the first place--maybe you can run some bastion/proxy?).
> You also need stuff like networking, TLS/certs, and DNS which aren't easily packaged
The only thing that cannot be packaged is changing your home network settings. For this you need to click buttons on your modem/router. Fortunately, many selfhosted programs (eg. prosody for XMPP chat) and distros (eg. yunohost) have check commands or panels to figure out what's not configured well on your network and guide you through the process.
Also worth pointing out, Yunohost distro is also intended to be used over a VPN precisely so you don't have to deal with networking setup. Yunohost was bred in the non-profit ISP scene here in France and so your local ISP will provide you with an "internet cube" (SBC) and a VPN access giving you real public IPv4/IPv6 so that:
- you don't have to configure the network
- you don't have to change DNS settings when you change connection (your server works if you take it with you over 3G/4G/whateverG)
- your ISP doesn't get to filter the network (unless it filters VPN access but that's rather uncommon)
Not to advertise, but I'm building exactly that at https://pibox.io - also solving other problems people have identified in this thread like automatic valid certificates, DNS, remote access, etc :)
Wow, love it! I host a matrix server on my current NAS, but I can’t put the database there cus spinning drives are just so slow. I’ve got the DB on a random Mac right now, but this is my new upgrade path.
The problem is you’re fighting a battle against global economies of scale for what is essentially a hobby or personal project. This is not a winning battle and most companies prefer to outsource the risk to someone else they can point to shareholders and blame.
People get caught up in the technical aspects of developing for cloud but I’d bet those weren’t anywhere near as important as risk outsourcing for the executive. At that point cloud was still new and the thought was we can run our infra if we need to.
Interesting landing/marketing page. But once I clicked to test it I ran into these notifications for nearly everything I would have preferred to use:
> We have not yet started working on our XXX implementation, but you can select it and submit to let us know you're interested. Development is prioritized by demand.
Or the "We are currently working on our YYY implementation..."
So nearly nothing I would have wanted to use was available currently.
Looked interesting from the outside. Looked to me as of more time was spent polishing the marketing than the actual offering.
Can recommend https://cloudron.io for those looking to get started with self-hosting and don't have a whole lot of time figuring out how to install/update a variety of apps.
I love promoting self-hosting.. self-host, self-host, self-host!
Having said that, I'd say: Chose your battles wisely...
You can run your hardware in X number of physical locations that you have access to (personal house, family etc.). But that doesn't always suffice for backups, so go with an additional cloud provider for additional backups.
Emails: Do you want to be hit with tonnes of spam traps because you're an unknown IP (any individual doesn't send email emails to 'warm-up' your IP). Do you want to lose emails because your personal server had a power-cut or internet connection drop?
Monitoring: I'd said for small-medium personal setups, to get the level of monitoring, central logging and intrusion detection detection that someone (at least for me) would be comfortable with in the current age, a fair chunk of computing power goes to this. Maybe you'd use an external vendor for monitoring, since your home server monitoring itself won't detect if it goes out.
Instant messaging: For IOS, at least, you need to jump through a bunch of hoops to send notifications to devices - should you use an external service for this?
Honestly, I'm rambling, but.. I absolutely recommend self-hosting everything.. but I think a foreword about the amount of effort that needs to go into setting up services that you rely on a daily basis is (or should be) pretty high.
I.e. if I were wanting to setup a single service for myself that I _heavily_ relied on.. I probably wouldn't do it. If I wanted a bunch of applications.. serving 5 applications from a k8s cluster and some additional work for monitoring, log management, backups and other bits and pieces probably starts making sense.
On another note, for me, hosting things on your own, especially for data/services that you truly care about, sometimes can have a keep-you-up-at-night feeling of "you don't know what you don't know".. what if someone is in my network.. what if there's a vulnerability in the VPN, firewall and X, Y Z that hasn't been patched and someone is on my machine deleting/stealing my data. There's also people at lot more clever than you in the world and plenty of people writing scripts to automatically break into services that require a little more knowledge than you have on the subject (whatever the attack vector maybe).
624 comments
[ 2.8 ms ] story [ 381 ms ] threadIf you really want true self hosting you would run it off your own on-prem machine and use your ISP to push & pull content. Putting things on a VPS is not really 'self' hosting as you're entrusting a third party to not get their datacenter burned down, or the hard-drives corrupted, etc
That said, the only caveat to hosting in your own house is it could suffer a fire, and your data is wiped, so having /BOTH/ a VPS and an in-house on-prem solution means you're not putting all your eggs in one basket and you have a contingency plan in place, which one day may be worth it. It buys you peace of mind because of the redundancy.
[0] https://piwigo.org/get-piwigo
[1] https://ente.io/
Well, there are other reasons to prefer using external hosting. Home connections are typically port‐filtered, have dynamic IP addresses, and have a low IP reputation, and your ISP selection is very limited. Whereas if using a VPS there are so many options that it’s easy to shop around.
But you can still self‐host while getting the benefits of a VPS. Just forward ports from the VPS over a WireGuard tunnel to your real machine. Then all the actual infrastructure is on hardware you control, and the cloud provider has no access to your TLS private keys.
Do you really have any search and seizure protections on a VPS?
I'm aware of this, which is why I do full disk encryption of any VPS instance I operate. See the Third Party Doctrine[0] which applies to the US only AFAIK.
[0] https://en.wikipedia.org/wiki/Third-party_doctrine
This is such a niche attitude/market but it has been incredible to see the surge of self-hosted applications/services over the last 5 years.
It is also relatively easy these days with modern ci/cd tools to have a "portable" enough stack that in the event of an emergency you could purchase a few linode instances and be migrated to a vps environment in an afternoon.
What do I do when all the useful software is cloud based and requires me to store my data with the service provider in order to use it? Self hosting is not a solution.
https://photoprism.app/
Self-hosting are not easy for laypeople (someone who are not familiar with it) to try to get their feet wet with it. For myself, I am on the level of beginner and I do struggle to stay on self-hosting path. When I set it up, I learn there is more steps that I have to do because the documentations and guides did not bother to explain those step and expect me to research more to find the information about it.
My biggest beef with self-hosting is that they expect us to set up the SSL/TLS certificate without explaining the step to set it up. Some guides does have section about it but never provide the details about creating CA for my self-hosting needs. I turn to Google/DDG to find information about it and they are all over the place or leading into dead-end.
There are few others thing I have gripes with self-hosting. I like self-hosting and they are pleasing for me as I don't need to rely on third party solution. The gripes I have is the documentations that are over the place or sparse information about it.
If you have your own domain pointed at your server, the Let's Encrypt certbot can automatically pull in a certificate and configure your apache/nginx webserver (alternative webserver caddy has this feature built in as far as I know).
If you don't have your own domain, don't go with self-signed certificates. Get a free https://desec.io/ subdomain, and they have their own certbot plugin to generate automatic certificates.
Yeah, but don't have a mistake too many times, or Let's Encrypt will block you for a week until your rate limit times out.
I hit this. I understand why Let's Encrypt has to do this, but it's very annoying and you have no choice but to do nothing for a week.
There needs to be something in between Let's Encrypt (free) and a couple thousand a year (other CAs).
A DNS misconfiguration can cause your Let's Encrypt to do weird things on a configuration that was (and still is) perfectly correct.
That was how I hit it. I eventually figured out what people screwed up in DNS. But certificates still didn't clear. So I spent an extra couple hours staring at DNS trying to figure out what I missed when the issue was that we bumped into the rate limit at Let's Encrypt (which is REALLY low--I think 5 failures is enough to trip it) while the DNS was bad and the only thing we could do was sit around for a week with dead certificates.
Not fun.
I've hit the problem you describe, and I feel your pain. I also respect LetsEncrypt's choice to rate limit failures. I renew a couple dozen domains at a time, so one error can quickly cascade into being blocked. IIRC the block timeout starts at 24 hrs and goes up from there if you keep trying -- this is easy to do if you don't see the raw response error message!
After being bitten by this a couple times, I added a dry-run step to my autorenewal script. If the dry-run exits with success and generates a good new cert for the domain, I repeat by pointing to the LE prod server. This works every time (so far, but for years now).
I'm suggesting that any LetsEncrypt certificate automation system (or docs) targeted at relatively low-sophistication users (i.e. not you or me) should include this sort of dry-run check so that the user doesn't paint themselves into a corner with a somewhat persnickety, but essential, service.
Also of course, it should attempt to renew after 60 days, so that if things go badly wrong, there are a few block-timeout retries available before the 90 day expiration.
With all the tools out there and easy access to VPS services and even bare metal for your basement, there's never been a better time to self host. And not just web servers, but E-mail, git, photos and media, and so on, it's very accessible.
I took me ages to work out how to setup postfix properly from about 10 slightly different "guides". The Postfix book wasn't even that helpful. There are also lots of very out-of-date guides that might have been OK for 2015 but not anymore. They don't get deleted because "link juice"
It is sad but true but you get one little bit wrong and you potentially leave a door wide-open.
Lots of services are barely more than - apt install, systemctl enable --now, ufw allow 8080 (if you even firewall within your network).
I've been self-hosting a lot of load-bearing household stuff (I have stuff on the "wife-critical" path: if it goes down, "the internet goes down" and I get a text from her) for almost 10 years and I've only had 2 incidents of particular reputational-risk note:
1) a routine reboot of the main server triggered a BTRFS bug that blocked mounting it again. This took an evening and a reboot into an arch linux ISO to fix (arch had a new-enough version of the btrfs tools that had the ability to fsck/repair the fs).
2) my proxmox setup was initially installed with zfs and zfs-on-root. This exploded and the "on root" part stopped working one day. This was the most annoying thing to fix so far because I ended up dumping any interesting data to an external HDD and just re-paving the server, this time reinstalling with just ext4 and lvm (which is admittedly a setup I'm much more comfortable debugging). No issues since then.
Both these events are from over 3 years ago, so it's been smooth sailing in recent times.
I understand those documentations are not for laypeople for me. However it is annoying when people out there kept pushing the self-hosting for beginners narrative without providing the necessary tools for laypeople to keep themselves secured and reliable.
And that, in a nutshell, is the problem.
A few clicks, a configuration form and integrated tools to set up external dependencies (i.e., LetsEncrypt certs), et voila! You're running a self-hosted application.
AFAICT, this is more about developers not creating the packaging/configuration/management tools necessary for effective use by non-technical users.
Sure, I can write a sql query to modify the schema of an applications' database, but my highly educated and intelligent physician brother would just throw up his hands in disgust.
Make self hosting easy and people will use it. And Docker-compose isn't "easy" for a lay person.
So I have just read the handbook, but I have read it completely, which needs more than a day.
It is likely that there are also other operating systems and Linux distributions that have good documentation, but I can testify only about those that I have used in the beginning, the FreeBSD handbook and then the Gentoo Linux handbook.
Both handbooks were good enough to convert anyone into a system administrator.
Unfortunately, both handbooks are not as good in 2022 as they were e.g. in 2002, because they have not always been updated after every change, or the updates have not been as detailed as the original parts of the handbooks.
Even so, both handbooks remain reasonably good today.
Especially the FreeBSD handbook is good for someone who lacks experience, because FreeBSD is much more self-contained, i.e. there are a lot of choices that have already been made for you and you do not have to worry about them.
So for someone who is inexperienced, I believe that the fastest way to managing a server remains to read the complete FreeBSD handbook and install and configure a server based on that.
There are programs which are available only on Linux, but the administration of a Linux server requires much more work than for a FreeBSD server (even if much less than for a Windows server), so for a beginner I think that FreeBSD with its more complete documentation and less possible choices is easier to try.
https://yunohost.org
It's tricky to be in that hybrid-box since the conversation in this area is very dichotomous--cloud things OR my own thing--but overall I like keeping my options open and swimming with the herd ;-) in making sensible use of cloud services when it seems appropriate.
Should someone interested in self-hosting do it from a literal PC in your basement, configured as a server?
Or is self-hosting on AWS / DreamHost / whatever good enough?
I ask because I like self-hosting a lot, especially when market solutions don’t really do what I need them to.
But security, man, that worries me. I can’t tell you what a three-way handshake truly is, or what a signed certificate really means: so self-hosting my own email / web server / etc. from my basement gives me a fear that someone, somewhere will take advantage of a vulnerability in some system component that I’ve never even heard of.
You can also setup something like Synology which is good enough for layman and if you keep it in your local network it is basically easier than configuring some old PC.
It's a good place to start/test. But don't open your firewall: do all of your testing on your internal network. You really don't want to open your network to the kind of problems that can occur while you're learning.
When you're ready to really host things then you should rent a cheap shared instance, or maybe a low-priced dedicated server. You can pick up something decent for $10/mo. That's not much if you're skilled enough (eg, employable enough) to learn how to self-host.
For your internal network you can use a pi-hole to set up all of your DNS entries so you can even visit "http://example.com" and have it point to an IP on your LAN.
For those who feel unable top securely self host I'd suggest looking into smaller providers of hosted E-Mail solutions. A large number of federated services is better than everyone being on Google Workspace or MS360.
Nevertheless, if someone is willing to dedicate some time for study and experimentation in the beginning, this is not an insurmountable problem.
I have been using self-hosting on "a literal PC in my basement" for about 20 years, without any problems whatsoever, and with negligible costs (the main cost being that I have a set of public IPv4 addresses and a fixed IPv4 address on my router connected to the ISP, which implied a more expensive monthly fee for the ISP).
After the first few months, during which I have made frequent changes in the configuration, while I understood better and better how it should work, the time wasted with server management during the next years has been negligible, i.e. just a few hours per year, used mainly for software or hardware upgrades.
Configuring and managing services just for personal needs or for the needs of a small number of users, e.g. a family, is much simpler than in an enterprise setting.
For reliability, it is good to have a second spare computer and a second image of the root SSD/HDD used on your server, to be able to replace the active server in case of failure. As others have already mentioned, periodic backups should be done and they should preferably be stored in a different location.
While I believe that self-hosting is not difficult, unless someone has already done such management work as a professional, it is necessary to learn many things.
For security, the first thing needed is to understand well what a firewall does, which are the firewall rules needed by whatever services you want to host and how to configure and monitor whatever firewall program you choose.
For this, some knowledge about how the main IP protocols for networking work is necessary.
The management of keys and certificates is also important, as you have mentioned, but what you need to learn for this is much less than what you need to learn about networking protocols, in order to both make a correct server configuration in the beginning and to diagnose any problems that might appear later (usually because someone at your ISP makes some changes in their configuration, which break yours, but nobody who answers the support call has any idea that they have changed anything, so you should better be able to identify yourself what they might have done, if you want a quick solution).
Speaking for myself, I don't have the goal of 100% detaching myself from "the grid", so to speak. I still want to pay an ISP to act as a gateway to the internet, and want to pay the local electric company to power my house.
To me, "backups" are a commodity service, like internet service and electricity.
Dumb file servers are offered by any number of places for a price lower than the cost of in-housing that service, and with a negligible switching cost at for my workload.
I'm personally OK with having one relatively shitty local mirror, and a background task that rsync's to backblaze. If BB makes noises about going under, I can migrate aws s3, rsync.net, digital ocean, whatever entity wants to charge me the least for my workload.
I don't think NAS's or ZFS are strict requirements, although playing with them can be fun.
Selfishly speaking, I work at Railway and our community maintains a list of self-hosted apps (we call them starters) that people can deploy to our platform. You can checkout the list of apps here: https://railway.app/starters and we even accept submissions via our GitHub repo: https://github.com/railwayapp/starters (Just reply to me here and we can get it reviewed for ya.)
I'll have to take care of backups, security, availability, updates, etc. I prefer to use a managed solution.
If you don't want to lose data on being banned, just do your own backups, which are by themselves much less time consuming to handle than full-blown self-hosting.
I'm fine with the occasional service being axed, I'll just migrate to another one. Often, somebody writes a migration script and open sources it, making that even easier.
It is good though to promote and vote with your wallet for services that give you good and dependable support.
A hosting cooperative is often a good compromise. You get to mutualize services and maintenance with other people who have the same needs.
On a spectrum from selfhosting to cloud computing, hosting cooperatives lie in the middle when users have agency but don't have to take care of everything by themselves.
I'm going through this dilemma with books. Goodreads lost my account of nine years. I've managed to recover most of the data from a backup and set up my own blog. I'm self hosting! But my blog is very spare and is not backed by a database of books, book covers, etc. Also it has no social features, no easy way to see other people's reviews or find related books or... I could imagine building all those things but that's like building a whole product! I could also imagine some self hosted book product I could just use (analagous to Picasa in the story) but it doesn't happen to exist.
Meanwhile there's a pretty great product for books in Goodreads, other than the crippling disaster of losing a user's account. Also some good cloud competitors like The StoryGraph. So maybe I should just use their product and hope my data is safe.
PS: I was at Google when Picasa was acquired. My memory is that the plan was always to focus on the hosted version. Maintaining a desktop standalone product was very much not in the Google business model.
https://openlibrary.org/developers/dumps
> This engineering talent is supposed to be solving world’s problems but instead they are ensuring how everyone wastes their time
Agreed! If software was sold for its utility instead of its addictive properties - this might start to change. Self-hosted / open-source software does need plenty of "hosted" accoutrements though: backups, remote access, etc. Shameless self-promo: we're trying to solve this over at https://kubesail.com
[0] https://github.com/sissbruecker/linkding
The software and Nextcloud data are all on an SSD, but the Nextcloud data gets a nightly backup to a mechanical hard drive. The music doesn't have any backup, but I could always re-rip the CDs if I had to.
I really want is to learn how to rent rack space from a colocation. The documentation available does not make it easy to learn. Can I just buy an old 1U blade, throw xen on it and show up at my nearest colo? What do I need to preconfigure to ensure I have remote access without giving remote access to the colo as well? Do I get physical access to the data center?
Wish I could find some guides on this topic. 95% of blog post tutorials are just ads for the latest trendy cloud startup/language framework.
That all said (and said with the clarity of age and knowing I was a stubborn kid who did things "because I could"), the experience of spinning up a VPS today on Linode or Digital Ocean is effectively the same, infinitely cheaper, and a lot more fun than racking a server somewhere. I can script up a fleet of servers from my bed at 1am just because, and can't tell the difference between SSH'ing to them versus that one box I did 15 years ago. If you want to do it, go nuts and have fun, but you really aren't really missing much over conventional VPSes these days.
I gotta disagree with you though on cost. You can get a beefy refurbished dual Xeon blade for a couple hundred bucks. Rack space where I live is like $50/month for 1U and gets much cheaper/machine as you scale up. $50 on aws will get me maybe 1 medium ec2 instance and an s3 bucket. With a used blade I get 20x the compute for the same price.
t3a.medium is under $30/month and that price only goes down when you reserve for the year. Save even more if you can run your service on ARM/Gravitron.
A VPS service like Linode will have even better pricing than AWS.
Driving over to a data center isn't free (time and cost), either. Those used Xeon blades are cheap for good reason – the companies that originally owned them consider them EOL. There's "no such thing" as dealing with hardware failure (except for occasional stop/starts) in the cloud.
But aws is extremely expensive per unit of raw compute performance. Obviously I concede that the stability and availability is unmatched compared to DIY solutions.
For my specific workload I need lots of memory and cpu cores. (Lots meaning my single 32c/64t 32Gb Xeon tower in my home). I almost fully utilize my resources and STIL need to pay aws for storage/tunneling/DNS/etc.
Consuming equivalent compute in aws would be hundreds of dollars per month.
Hybrid is just my preference!
I'm definitely not comparing to AWS, because yeah those can get super expensive, super fast. What you're paying for with AWS is Amazon-tier stability (whatever that's worth these days), but the difference in uptime between them and a Linode is more than fine for my needs.
With your $50/mo, make sure that includes power - Xeons eat watts. Also, be sure to compare apples-to-apples on bandwidth. By comparison, Linode's dedicated CPU plan (not shared, closer to bare metal, but still not) starts at $30 for 2 CPUs and 4GB of RAM and 4TB of transfer, and they'll take care of keeping you on the latest hardware. Again, I don't want to dissuade you, because colo is fun and it's cool to think about your own box out in the world. If anything, I'm envious of how easy it is nowadays compared to when I drove 3 hours to South Bend, Indiana to colo a box of my own, or the first time I needed to engage remote hands because the box got in an irreparable state.
Many rack space rentals will not permit you to just install whatever PC you fancy because it is potentially a risk to the neighbours in terms of fire or bad hardware, most will happily quote you to buy one their approved ones!
It is pretty easy to get a rack space provider where the provider cannot access the machine but this can be good or bad. In some cases, I would rather they could shutdown the host if, say, the RAM is broken and replace it but if you would prefer to do this yourself, that is fine.
In most cases, you will be given a public IP address directly mapping to your machine via a router/nat lookup so whatever services you open on your machine are open on that public IP address so pretty easy to setup RDP/ssh/whatever.
Probably the biggest issue though is the extra work or hassle if something goes wrong. I remember at a previous company where some guy would frequently have to drive for 30 minutes each way to go to a data centre to perform certain updates that couldn't be done remotely.
YMMV
I have _never_ experienced this. The only restrictions I've seen on colo contracts I've gone after were related to UPSes and things with large batteries in them. So a big stack of laptops would be a no, but if I wanted to put Atari ST's or Dell PowerEdges or white box builds or bitcoin miners it doesn't matter. I guess I've always done things at at least a half or full cab, never single Us at a time.
* you’d have to sign up with a colo provider first. since data centers in physical buildings, this just depends on where you live
* when you sign up with them they provide you with info like ip addresses or how to connect to their network (they might have dhcp, or you might have to configure static ips). usually there is a initial setup fee, around 1 month of rent.
* if you just rent a a 1U space you usually can get physical access to it while accompanied by someone working for the data center. usually this is during business hours, but each data center will have its own rules. if you rent larger units, such as a full rack (42U) or half a rack you usually get a key card and can access it 24/7 (this usually involves a phone call for them to remotely open a lock)
You can also just rent a space to place your own server but I haven't tried that.
Is power use included as well?
You will likely be charged 95th percentile mbps based on your usage. (Again, "pipe space required" to your needs.) Basically, whenever you're busiest -- 4pm-9pm are popular times for us in the USA.
Some customers limit their bandwidth themselves (like, only allow max 12mbps file downloads, etc.) especially when they have the hardware to support huge bandwidth. Or your colocation provider can perhaps limit max connection to 100mbs or 1gbps if you want.
Power is usually leased in amps. If you go over amps the circuit will break -- at worst case scenario. But typically they get in touch with you and tell you to upgrade.
Also, they do want to know vaguely what your service is. Because you'll likely lease their IPs, they will question you if you do a lot of email (caution for spam), or run a Tor exit node (legal hassles for them in many cases).
I like the sentiment and the points made, but the author uses this amorphous concept of "your data" throughout and I feel like it simplifies things a lot and conflates many different issues.
Most people shouldn't focus on self-hosting literally all the data related to them. This is a sort of perfectionist mental compulsion many of us on HN are familiar with. You have to decide what data you actually really don't want to live without in the rare event you lose access to it, and prioritize that. For most people, this data is not very complex: family photos and videos, an album by an obscure artist, a game you like to play every few years or hope to show your children.
If you are an activist, or someone creating dissident media, or something like that, you should already be wary of the cloud -- the incentives already drive you to use tools that are secure and self-host when needed.
If you truly don't like the ways the big tech companies are doing things, you should find ways to organize with others and demand change; otherwise you are just modifying your personal habits and thinking you are sticking it to the Man with a one-person boycott.
Even on something like a QNAP (which can be compared to managed hosting) this can be hard. Flip the wrong switch and you expose something to the world. Missed a security update: your device is now vulnerable.
While I host a lot of things myself I can understand self hosting is not for everyone.
_anyway_
I haven't been hacked.. yet. /me knocks on wood
The precautions I take are basic:
I figure cloud services and SaaS get hacked anyway. I can't enumerate the breaches my data has been a part of. If my self-hosted stuff gets hacked at least I can do the forensics and actually see what happened and what was accessed. With a 3rd party all I can hope for is what their PR department lets out.There's honestly so many posts that make it look so easy, but without everything else that would normally make it a job position in a company :)
All these script kiddie tutorials are terrible at showing how to maintain a server for years.
Docker is great for getting toy projects to work somewhere as a last resort, if the dependencies are strange and you need a convoluted (read: badly thrown together) environment to set up the app.
A well made application should not need docker to run.
One strategy for dealing with accidental misconfigurations is to employ a "network slug"[1]:
"A Network Slug, or "Slug", is a transparent layer 2 firewall running on a device with only two interfaces. ... The purpose of a Slug is to reinforce a security policy or to block uninentional leaks of information."
[1] https://john.kozubik.com/pub/NetworkSlug/tip.html
It also does “waterfall” egress packet delaying.
A slug should not need to be dynamic nor should it be complicated in any way ... in fact, it is one of the simpler systems I have ever deployed ...
I have implemented firewalls similar to this in the past. They typically had three network interfaces. Two of them were configured as bridges and then I use ebtables/iptables to filter traffic flowing through. These two interfaces would have no IP address and would not be visible on a traceroute, etc.
The third interface would only be connected to a separate admin network. Or it might not even be plugged in. In the latter case, the admin needing to change anything on the device would have to be physically present and bring a "crossover" ethernet cable and plug their laptop directly into the third NIC of the firewall. From there, they would be able to ssh into the firewall and change config.
Let me give you an example - I have a "port 22 slug" and what it does is block all traffic of all kinds except for TCP22. That's it. It does nothing else and it does it transparently without having an IP address of its own. If I wanted to reconfigure it, I would connect with a serial console.
Make sense ?
Almost like an overglorified but managed hub.
If you like your MAC, you get to keep your MAC.
I would really like to make such devices for home or office use. What would be a good device to use for this? Unfortunately, RaspberryPIs do not come with 2 or 3 NICs. Any recommended alternatives?
https://openwrt.org/toh/views/toh_available_16128
https://www.seeedstudio.com/Rapberry-Pi-CM4-Dual-GbE-Carrier...
... to make firewalls and bridges with rpi cm4 ...
Drop in replacement while outside LAN are admittedly a little harder and more at risk of mistakes
Easiest solution is to just host stuff on a local network without access to the wider internet. E.g. running on an old laptop/raspberry pi/server in your basement.
Sure, that means you can no longer access your self-hosted stuff when you're out of the house, but the tradeoff is peace of mind about your data leaking or worse.
You have no idea what tailscale is really doing.
please elaborate...
I assumed the GP was talking about a typical home "local network", one behind a NAT - so no incoming traffic, but usually, it allows any outgoing traffic.
Lots of things I'd consider self-hosting are functionally useless if I can't access them from my phone while out and about.
I could put my phone on a VPN, but that's just another layer of complexity to add to the self-hosting process.
1) Have separate bookmarks for accessing services via the LAN or the VPN, and need to remember to turn on and off the VPN when you need to access things out of the house; or 2) Always access services via the TailScale IP, which means you need to be connected to the VPN on your phone even to access them from your LAN, which in turn either means toggling the VPN on and off even when at home or leaving it connected all the time and letting it drain 20% of your battery every day.
It's a great service but I didn't find it to make accessing self-hosted services from out of the home on a phone to be as nice as using them at home, or using a managed service.
I think Tailscale opens up all kinds of new opportunities for self-hosting.
If you don't trust Google drive with your passwords, why would you trust a company's server that manages access to all of your devices?
I'd like to believe the engineers running Google Photos or iCloud are spending a lot more time on keeping my photos secure and available than I would be willing to put into a server running in my basement.
In the case of a business hiring an MSP to manage something complex like firewalls, Active Directory, server patching, then sure it's reasonable to assume that if they made a mistake, the impact would be equivalent to you making the mistake yourself.
It's possible you need to tell whomever you are reporting to for these post-mortems, they should be outsourcing to reputable service providers in order to free up time and man-hours, not necessarily just to save financially. I suspect that is the real problem.
Big Hint: AWS has had something like 5 times the downtime my biggest clients on-prem datacenters have had this year. So...no, the FAANG engineers aren't doing better than my clients.
Personally, I tend to self host the things whose maintenance I at least find satisfying, and hopefully enjoy. Otherwise I pay someone (through ads or my own money) to do it for me.
I have never volunteered to maintain servers since, it was horrible and everything was always my fault (it kinda was, I was a hobbyist at best with no real production Linux experience.)
I do still end up as the dev ops/infra guy at every place I’ve worked but at this point I’m probably one of those stubborn senior guys who wouldn’t like the way the juniors went about it.
Maintaining inherited environments is also much more painful than ones you get to design from the ground up. I work with varied environments, and one with ~250 RHEL / CentOS machines has approximately the same level of maintenance burden as another with a dozen or so Ubuntus because the first environment has had configuration management from the beginning and the second is a complete mess that I've slowly tried to reverse-engineer and clean up.
When your change management works, maintaining a dozen servers isn't all that different from maintaining a thousand or more; and the need for change management and automation doesn't really go anywhere even when you don't self-host things.
Now I'd rather leave hosting to a someone dedicated to it who has internalized the latest state of things for all the relevant bits of software and is constantly keeping this knowledge in their brain. Set and forget self hosting can't work in the current environment we have where things require constant security updates and complex security hardening.
I used to backup to external drives. Now I use bare ones since finding big externals got difficult.
I use (and probably abuse) docker compose. K8s is great but compose is easier.
I use a single makefile. Kinda ugly but it's fine.
Bunch of friends and family use my "services". They usually chip in for hard drives and stuff.
I have a few central points of failure but it keeps things easy. My uptime still beats most big clouds - though I have it easier.
I accidentally took down my server for a few days from a botched hardware install. It's a bit funny because now we realize how critical the home server has become to us.. on the other hand, already got the spouses blessing to build a backup standby server.
Now I have a single dashboard and can upgrade each container with a single click, and everything stays on the happy path.
automation is not a thing? I'm pretty all cloud providers do it...
It doesn't even require actively flipping switches, but can be from not knowing a vulnerable feature was enabled by default. My QNAP got hit with ransomware because of a vulnerability in the cloud access software that I wasn't even using. I've since locked down all non-local traffic.
I'd say the point being: I'm a software engineer who knows better about these sorts of things and still got caught with my pants down. You have to be very judicious with respect to security. You can't just plug and play and say "I'm too busy to worry about that."
Another thing I'll add is the amount of software tools they have on these NAS machines strikes me as 1) very impressive for a company their size and 2) a huge surface area rife for being hacked. When it happened I wasn't surprised at all.
I've since stopped using it because at the end of the day I'd rather pay Dropbox to have peace of mind.
Maybe I'm too old (experienced) and cynical, but I always read that as "apps that are going to brick themselves". No thanks.
The problem is, that one can't reasonably wait a few days on every update and look online for breaking changes. Especially with mobile apps that have sometimes a really unreasonable update frequency.
I still have not found a satisfactory solution for me personally.
I'm pretty sure we all used to that and it was mostly fine.
I get that the mainstream computer user has been lost to techno-infantilism. But why should we?
I recognize that this embarrassment of riches is in part my own fault. But this is my answer to your “why”
If it becomes simple again I would gladly self-host.
Self hosting has zero standardisation around hardware, software, install mechanisms. It's a Dev led movement that has everything to do with control and ownership over ease of use. You want mainstream adoption of self hosting. Rebrand it, standardise it, make it easy for non devs.
And now you have a blog, a picture-sharing thingie, a bulletin board, a whatever.
Maybe there could even be a version where you pick a datacenter and somebody racks up a PC for you with the software on it.
Sure, I have some neat modern things I'd like to do, but I also have a shared hosting that's been doing it's job for pennies since 2011.
We tend to underappreciate the importance of time in everything. A button click can instantiate something powerful (and useful (and easy-to-use...)), but it will degrade over time, and eventually flat-out stop working.
I had a stack that worked just fine for my own needs, but it ran on shudder Python 2.7 -- everyone knows how that worked out (I chose to rebuild my stack on a different platform).
Software doesn't degrade over time (other than, you know things like cosmic ray bit flips, but in most realistic situations that should be fully mitigatable.)
The needs of the software user (including hardware and software they want the piece of software to interact with) may evolve, but that's different than software degrading over time.
> I had a stack that worked just fine for my own needs, but it ran on shudder Python 2.7 -- everyone knows how that worked out
While there's no further first party support for that version of Python, if it worked properly before, Python 2.7 and the software running on it probably still works properly now.
“Degradation over time” was being cited as a reason not to self-host. Pointing out that not self-hosting exposes you to risk of others changing the environment so it no longer supports your software is a diametrically-opposed argument.
Ditto the average Rails/Python/Javascript project, as anyone who's tried to resurrect one that's gone so much as six months without being touched can attest. Which might not matter except that a ton of the software people might actually want to self-host are in one or more of those high-entropy ecosystems. Extraordinary levels of care and organization on the part of the creators and maintainers can mitigate this, but that amount of taste and effort is vanishingly rare.
These are degradation due to a changing environment, sure, but I wouldn't describe it as due to evolution in the needs of the user (presumably "must not have any well-publicized remote vulnerabilities" was a need from the beginning).
Otherwise, you are going to be influenced by external factors (security vulnerabilities, wanting to use a feature only available on a newer language version or OS, etc.) If you are a business, you'll also run into more practical concerns, like engineers not wanting to work on a mountain of technical debt.
But that's really the problem - maintenance. Right? Once something goes wrong _for whatever reason_ the user is then (for the immediate needs) just as stuck as with a cloud provider who disabled their access.
Thankfully there is a better course of action - e.g. find someone to fix it for you. Maybe on upwork as well?
But where are you hosting this? Is it AWS? Did _they_ suspend your account? I guess my point is that unless you host on hardware in your house (or another accessible place) you're at the risk of losing access to your data for any myriad of reasons. And even then, there have been warrants where devices were collected and went into a years-long battle as evidence.
Ideally the only difference between self-hosting and relying on a cloud service would be, I own the servers and therefore the maintainer has no legal right to bar my access.
> a multi-purpose fully-featured and integrated web picture gallery script written in PHP using GD or ImageMagick as image library with a MySQL backend.
And SSL certificates are for free and automatically generated.
An example: https://www.netcup.eu/hosting/#webhosting-details
https://www.netcup.eu/hosting/webhosting-application-hosting...
https://opencollective.com/sandstormcommunity
We've discussed the one-click install thing at some point (not necessarily with Cloudflare), I imagine that's still of interest. There were some issues with the setup process that would need to be addressed first.
Kenton is in the loop and he still has the keys. But, he's busy with other things so he only does a few occasional but vital things.
Or a box in your house, which is where my Sandstorm server lives. :) I think there's a lot of potential for actual self-hosting, though servers like Sandstorm need to have reasonable defaults and make it easy to manage domain setup and backups and security updates, such that one can get a box, plug it in, and reasonably quickly get to "don't need to touch this ever" territory.
The only thing that cannot be packaged is changing your home network settings. For this you need to click buttons on your modem/router. Fortunately, many selfhosted programs (eg. prosody for XMPP chat) and distros (eg. yunohost) have check commands or panels to figure out what's not configured well on your network and guide you through the process.
Also worth pointing out, Yunohost distro is also intended to be used over a VPN precisely so you don't have to deal with networking setup. Yunohost was bred in the non-profit ISP scene here in France and so your local ISP will provide you with an "internet cube" (SBC) and a VPN access giving you real public IPv4/IPv6 so that:
- you don't have to configure the network
- you don't have to change DNS settings when you change connection (your server works if you take it with you over 3G/4G/whateverG)
- your ISP doesn't get to filter the network (unless it filters VPN access but that's rather uncommon)
See also https://internetcu.be/
People get caught up in the technical aspects of developing for cloud but I’d bet those weren’t anywhere near as important as risk outsourcing for the executive. At that point cloud was still new and the thought was we can run our infra if we need to.
https://www.molecule.dev/
> We have not yet started working on our XXX implementation, but you can select it and submit to let us know you're interested. Development is prioritized by demand.
Or the "We are currently working on our YYY implementation..."
So nearly nothing I would have wanted to use was available currently.
Looked interesting from the outside. Looked to me as of more time was spent polishing the marketing than the actual offering.
Docker-compose comes pretty close to this. I had no idea wtf I was doing when I got started and it resulted in a functional thing surprisingly often
Not quite the SaaS vision you describe, but point is you can stumble into something functional pretty easily these days
I can share, be social, get recommendations, not worry about backups or a lost computer, not maintain anything, access from my iPhone, etc.
I have thousands of photos and music collections lost on old laptops and hard drives that I'll never see again.
I know there's huge tradeoffs (as articulated here), but there's some really amazing things about the direction the web is going.
Having said that, I'd say: Chose your battles wisely...
You can run your hardware in X number of physical locations that you have access to (personal house, family etc.). But that doesn't always suffice for backups, so go with an additional cloud provider for additional backups.
Emails: Do you want to be hit with tonnes of spam traps because you're an unknown IP (any individual doesn't send email emails to 'warm-up' your IP). Do you want to lose emails because your personal server had a power-cut or internet connection drop?
Monitoring: I'd said for small-medium personal setups, to get the level of monitoring, central logging and intrusion detection detection that someone (at least for me) would be comfortable with in the current age, a fair chunk of computing power goes to this. Maybe you'd use an external vendor for monitoring, since your home server monitoring itself won't detect if it goes out.
Instant messaging: For IOS, at least, you need to jump through a bunch of hoops to send notifications to devices - should you use an external service for this?
Honestly, I'm rambling, but.. I absolutely recommend self-hosting everything.. but I think a foreword about the amount of effort that needs to go into setting up services that you rely on a daily basis is (or should be) pretty high.
I.e. if I were wanting to setup a single service for myself that I _heavily_ relied on.. I probably wouldn't do it. If I wanted a bunch of applications.. serving 5 applications from a k8s cluster and some additional work for monitoring, log management, backups and other bits and pieces probably starts making sense.
On another note, for me, hosting things on your own, especially for data/services that you truly care about, sometimes can have a keep-you-up-at-night feeling of "you don't know what you don't know".. what if someone is in my network.. what if there's a vulnerability in the VPN, firewall and X, Y Z that hasn't been patched and someone is on my machine deleting/stealing my data. There's also people at lot more clever than you in the world and plenty of people writing scripts to automatically break into services that require a little more knowledge than you have on the subject (whatever the attack vector maybe).