You can still have a backup of your files and push them to another provider without self-hosting. It will take up 10x-100x your time to learn and use and maintain these alternatives, versus just taking a regular backup and using a managed provider.
It seems like 95% of the adherents to self-hosting do it as a hobby but pretend it's prudence.
I always wonder why people don't trust their offsite back-ups to cloud providers. I know they're trying to get away from getting locked out of their data, but what are the odds a burglar steals their computers on the exact same day their cloud provider locks them out because they violated the 'no making fun of ridiculous cloud provider lockout policies' policy?
As long as your house burning down and your cloud getting locked don't occur on the same day, you're golden and thus no messing with blue-rays and bank security boxes.
> Every last weekend of the month, I will manually backup all the data to Blu-ray discs. Not once, but twice. One copy goes to a safe storage space at home and the other one ends up at a completely different location.
This is one paragraph after mentioning a 2TB+2TB NAS. Even assuming that's RAID1, a standard Blu-ray only stores 50 GB, so you need 40 of those. And then you need another 40 for the other location... every month?? Honestly it's probably cheaper to buy a new 4 TB hard drive every month.
If you're a cheapskate like me, backing up your encrypted (e.g. Borg) backups to a cloud provider like Google Drive isn't a bad option. My org provides me with unlimited cloud space, so I have hundreds of encrypted gigs on Google Drive. No reason to think it'll disappear overnight.
> it's probably cheaper to buy a new 4 TB hard drive every month
The reason for blurays might be that he's following rule 2 of the 3-2-1 Backup Strategy. 3 backups, 2 different types of storage media, 1 copy off-site.
If your house is hit by lightning it could wipe all your magnetic and solid state drives, but optical discs would probably be ok.
Most definitions of the 3-2-1 rule count the original data as one of the 3 copies [1][2][3] and don't go as far as to specify that you should be diversified against literal medium type. (Most recommend an external drive or NAS for your second local copy)
The "my house was hit by lightning" case is covered pretty well by your 1 offsite backup.
Use borg or other encrypted backup tool (Restic is also recommended by others in this thread). rsync.net is my offsite backup. Doesn't matter what anyone demands of them, all they ever see is an encrypted blob.
You do then need to find somewhere to store an offsite backup of your encryption keys. That said, since those change far less often than your backups, options like a safety deposit box are a more realistic place to store keys than the backups themselves.
Yes, borg also has the option of storing them in the repo itself, protected by a passphrase (think encrypted ssh key files).
Anyway, my "home burned down rescue bundle" consists of a flash drive with a keepass export of my password vault and encrypted borg repo key / rsync.net ssh keys at the office.
Slightly less accessible in this pandemic world, but no safety deposit boxes needed.
Important to reiterate this, since it's an interesting feature - borg does, indeed, allow you to store the encryption key for your repository inside the repo itself and you just need to remember/safeguard the passphrase.
I can't say whether this is a good choice for any particular use-case but I appreciate that it is an option ...
For those that are curious, yes, you can later export the key from your repo so that you have a copy elsewhere ...
Yeah I tend to trust B2 for my offsite. I have redundant storage locally with snapshots. That covers up to two disks failing or even someone trying to wipe storage over SMB.
The offsite protects against catastrophic failure or theft. I would still like to add in another backup for critical data such as databases, I may use another cloud provider that has geo redundant storage for those.
> I always wonder why people don't trust their offsite back-ups to cloud providers.
Oh I trust them not to delete my encrypted data and that's all that I'd ask. But I still don't have backups on hardware that I don't own: it's super expensive. From the cost of hosted backups, I could buy a hard drive of the same storage space every three months or something. And hard drives live longer than three months.
Putting a raspberry pi with a big hard drive at a friend's place is rather power efficient, dead simple to setup if you have any technical knowledge at all, and off-site. Use Restic or something to avoid having to trust your friend or whoever will burglarize the place. They have an Internet connection anyway and where I live, there are no bandwidth caps.
The diagram alone is more than enough of an argument to dissuade me from giving this a shot right now - it's simply too complicated and too much to manage for the amount of time I can dedicate to it.
BUT - I'm really thankful for people who keep posting and sharing these sorts of projects; they're the ones iterating the process for the rest of us who need something a bit more turn-key.
I'm excited to see this eventually result in something like the following:
- Standard / Easy to update containerized setup.
- Out of the box multi-location syncs (e.g. home, VPS, etc.)
- Takes 5 minutes to configure/add new locations
I want this to be as easy as adding a new AP to my mesh wifi system at home: plug it in, open the app, name the AP, and click "Done".
just use k3s + (restic + velero [backup]) it's soo much easier you can basically install everything with the same tooling and update everything with the same tooling. if something breaks, bam you can just restore the whole cluster with velero (including local volumes)
We always have this debate at work. Do we build the system ourselves or do we purchase a product? On prem Prometheus or push everything to DataDog? I'm always a fan of building things myself because I like building things, but my company compares engineering time vs product cost.
i've actually daydreamed about starting a computing appliance company that would make a variety of services plug and play for consumers and small businesses, from email to storage, to networking, to security, and to smart home. it's actually the direction apple is headed, but they're encumbered by the innovator's dilemma, which leaves an opportunity for an upstart. google and facebook are similarly too focused on adtech, while amazon on commerce, to lock up this market yet.
It would be fun stuff to build, but I feel like you'd struggle to make money. Google and Amazon can afford to give away the hardware, and they can smuggle their ecosystem into your house as a thermostat or a smart speaker or a phone app, or whatever.
Like, how do you persuade the audience of enthusiasts (think: Unifi buyers) to pay for a subscription to managed software they run on their own computers, raspis, whatever? I would probably spend $10/mo on something like that, but much above that and you'd be fighting against the armchair commentary of users who won't appreciate the effort that goes into stability and will basically have a "no wireless, less space than a Nomad, lame" attitude.
there are actually tons of companies in this space already making money (e.g., wyze), but it’s highly fragmented and none have a unified vision or product strategy yet. so yes, they’re vulnerable to the behemoths right now, but those dynamics aren’t locked in yet.
it’s mostly tough because of the high upfront capital costs (manufacturing, r&d, and marketing). people still talk fondly about discontinued apple routers and what nest could have been as an independent venture, for example.
Maybe I'm misunderstanding the pitch from the GP, but Wyze seems like it's pretty clearly a hardware + cloud services play, similar to most other IOT ecosystems except maybe Hue. The (optional) monthly cost paid there is for loosened restrictions on an already existing works-anywhere setup— it's an upsell for power users, not a cost of entry.
This seems a lot easier to me than on-prem cloud services, either in BYOH form ("but it's just software") or as a packaged appliance ("another hub to install, really?").
I would say that the closest thing to this right now for paid is coming from the storage side— NAS providers like Synology using hardware sales to support a limited ecosystem of "one click" deployable apps. And for free, it's ecosystems like HomeAssistant, which a lot of people just deploy as a fire-and-forget RPi image, but as expected with a free ecosystem, as soon as you get off the ultra-common use cases, you're reading source code to figure out how it works, and wading through a tangle of unmaintained "community" plugins that only do half of what you want.
the primary value-add is one layer higher than a NAS, a standalone router, or homeassistant but would likely be built on those kinds of things. it's providing a range of hardware devices that can work seemlessly together in a way that you don't have to muck around with config files or programming and yet have it all be secure and private by default. the value is in an ecosystem of safe appliances that require little technical knowledge.
home audio/theater from prior to the internet revolution might be a good analogy: a bunch of separate boxes that each provide tailored functionality but all work together seemlessly without a lot of technical knowledge. that, but for all sorts of computing devices.
> there are actually tons of companies in this space already making money (e.g., wyze), but it’s highly fragmented and none have a unified vision or product strategy yet. so yes, they’re vulnerable to the behemoths right now, but those dynamics aren’t locked in yet.
I also think it's still a little too nerd-focused for the average consumer. I'd say I know far more about security, networking and hardware than the average consumer but, compared to the HN crowd, I know next to nothing. I struggle to use a lot of the current solutions because they get bogged down in doing cool technical stuff that is so far outside the scope of the average potential user's wants/needs or the DIY solution will be "easy"... for someone with an extensive CS background and years of experience.
i think you need to go up a level of abstraction from this perspective to see the utility for the average consumer. we each have computers all around us, phones, tablets, tv's, and increasingly everything else. it's so hopeless to manage, much less understand, these mysterious machines for more and more people. what you want is a company you can trust to manage these things for you but gives you the ultimate, yet cognitively bounded, control over them.
for instance, plug in a smart device and have confidence that it's not doing surreptitious things behind your back, because it's automatically segregated into its own vlan and given only enough network access to be controlled by you without needing to know much about the underlying technologies involved.
"a company you can trust to manage these things for you"
"automatically segregated into its own vlan"
Aren't these goals fundamentally at odds? I would imagine that Joe consumer (if they care at all about any of this) would be rather more inclined to entrust the role of orchestrating/segregating their home network devices to an entity like Google than to some random startup.
the average person doesn't even know what these things are, which is partially why there is a market opportunity here. what they know is that companies like google and facebook are not entirely trustworthy but they have no alternative. it's hopelss, until an entity comes along and gives them some hope in the form of an alternative. basically all of the things we talk about around preserving privacy and security on the internet need to be built into our devices, and companies like google actively oppose such limitations of their reach into our lives.
You're getting into Ken Thompson's "Trusting Trust" territory here.
When you lose trust you end up with your crazy uncle leaving Fox News for Alex Jones and YouTube. You have people becoming QAnon followers.
I say this not to make a political point, but that the problem is fundamentally hopeless and I see no way out. You end up landing on one side of the fence or the other. You either just don't think about it and continue to use Google and Facebook and remain ignorant of the problem, or you spiral down the never-ending hole of despair.
We have seen articles recently that tell us not even Signal can be fully trusted. Whether or not it's true is beside the point. The point is, not even the HN crowd is safe from the cliff of paranoia. The seed of doubt has been planted.
Is someone going to trust a small tech startup in 2021? No, not like they would have in 1997. The market for trust has effectively been sealed off today. Because, paradoxically, the Googles and Facebooks ruined it all. They stripped us (all of us, not just HN) of our innocence and naivety. We know not to trust Google, but they are also a known known. A small tech company is a total unknown. We're familiar with how Google is going to bend us over. So if someone is going to do us dirty, it may as well be a known entity. Or... you go and build a cabin in the woods and start writing manifestos.
while i agree google and facebook have certainly peed in the pool, that strikes me as overly cynical, simply through recognizing that only a fringe few actually radicalize or become helplessly paranoid in that way in practice.
most people, whether they rationalize it or not, are cognizant that we live in the grey gradient of trust for various companies and brands. the vector field is all sorts of wacky and inscrutible, but maybe we can point a few of those vectors in the right direction and some folks will happily slide down it to better (but perhaps not perfect) safety and privacy.
There are very mundane reasons that you may want to own the software stack your data depends on that aren't 'I don't trust Google/Facebook/Microsoft/The Reptilians.'
The most prominent of which is "What happens when they drop support for my use case/lock me out of my account?"
Unfortunately, the cost of running your own one-off solution is rather high. And doubly unfortunately, while I would pay money for a box that I could plug into my computer that provided all of these services, I wouldn't pay enough money to justify someone building it, and selling it to me.
It's not "hopeless to manage", learn some networking and be forever rewarded. Same with learning to manage devices, servers, etc. I develop now but I'd be much less valiable without that background.
Hardware sales. People will pay for the convenience of a device that works out of the box with minimal setup.
On the software side, integrate tightly with your own subscription services (offsite backups, VPS, etc) to upsell to those who want that, and win over the enthusiast crowd by making it possible to host your own alternatives to those services with a little technical know-how.
Open source most components to appeal to enthusiasts, but keep the secret sauce that makes everything seamless and easy to use "source available" so you don't unintentionally turn your core business into a commodity.
Alternatively, it's what Windows Home Server might have become had MSFT kept at it. OTOH, the fact that Microsoft abandoned it might be an indicator of how well such a thing might sell.
If you want to self-host email, you need a trustworthy static IP address with reverse DNS. It's considerably more expensive to get this from an ISP. Our annual fee also includes storage for offsite backups. You don't get the same privacy assurances using Protonmail as you do with self-hosting either. For example, Protonmail is privy to the content of all outbound email messages in the clear unless you are communicating with the recipient using E2EE.
From a cost perspective, Helm V2 starts at $199 for 256GB of storage. First year costs, including subscription work out to $298. With Protonmail, their entry level plan with added storage at the same price buys you an inbox with about 28GB, a small fraction of what you would get with Helm storage-wise, not to mention we don't limit users, email addresses, domains, etc.
You clearly have the upsell part, but where is the "and win over the enthusiast crowd by making it possible to host your own alternatives to those services with a little technical know-how." part?
I and probably many others would be OK with paying for the upsell part, if it's an optional convenience, but nothing I saw on your site indicates it is, or that "own your data" is in any meaningful way true. How do I own my data if any use of it requires me running stuff on your proprietary box, subscribing to your proprietary service?
If you store data on a hard drive you purchased from Best Buy, do you own that data? It's a proprietary box also...
Data from Helm is accessible using IMAP, SMTP, CardDAV, CalDAV, WebDAV on the local network (without requiring our service). You own the device, you own the data. There is a standards-based way of accessing that data just as there is with the hard drive from Best Buy.
> If you store data on a hard drive you purchased from Best Buy, do you own that data?
I can plug the Best Buy harddrive into almost any computer/SAN I want to and utilize for the purpose I bought for without any lock in. Using the hard drive for my data requires very little trust in Best Buy's good intentions at the time of purchase and zero trust in the continued existence, technical competence or good intentions beyond that -- it is very unlikely that best buy will find a way to snoop on my data even if they wanted to. Everything will continue to work fully as intended until mechanical failure sets in. Feels like ownership to me.
In your case I rent some box from you which will lose almost all its intended function the moment your company goes bust or I stop paying you an annual fee. Furthermore it seems I am completely at the mercy of your original and continued good intentions as in addition to using your lock-in for a big price hike later you presumably can also snoop on my data. As far as I can tell there is no really substantial trust differentiator to protonmail. I have to trust them when they claim they won't read my email and are competent enough to keep things secure and I have to trust you (and continue to trust you as long as I want to use the device) that you will encrypt my data and not exfiltrate any of it (or the private key), and furthermore that you run your servers securely enough that no third party will. But what is to stop it? The box is running closed source software that you can remotely update anytime you feel like it, right? I have physical access, but since I don't control the software, what use is that?
Maybe I didn't understand something right, but so far this does not feel like ownership to me.
It sounds more like the worst of all worlds: the lock-in and lack of ownership of a proprietary cloud-based subscription service with the added hassle, inconvenience, downtime and costs of babysitting (and supplying electricity to) a cloud server for you, the provider.
> Data from Helm is accessible using IMAP, SMTP, CardDAV, CalDAV, WebDAV on the local network (without requiring our service).
In what ways is that better than running an IMAP client on my laptop and using it to send data via protonmail, using my own domain, and keeping offline copies of everything (with a periodic upload to backblaze or some other e2e encrypted backup solution)? That seems to offer about the same control over my data, but is cheaper, easier, more convenient, has higher redundancy/uptime and if anything less lock-in. It also doesn't require an additional device that has no use beyond adding an additional failure point and cost center that's my responsibility.
> There is a standards-based way of accessing that data just as there is with the hard drive from Best Buy.
Accessing the data is not enough because what you are selling me is not an overpriced and unergonomic hard drive. You are selling me the ability to send, receive and store email (and likely more).
Don't get me wrong, I kind of like the idea of buying a physical box and a subscription service to self host stuff in a way that gives me better control over my data for an acceptable amount of hassle. But that really requires some amount of openness/auditability and interoperability that currently appears to be absent.
> In your case I rent some box from you which will lose almost all its intended function the moment your company goes bust or I stop paying you an annual fee.
No - we do not rent hardware. When people buy the server from us, they own it. Full stop. There are ongoing costs to make email at home work: a static IP address with good reputation, a security gateway, traffic, etc. If people don't want to pay us for those costs, they will pay them to an ISP and/or an infrastructure provider like AWS. The ease of setup and management comes from the integration of hardware, software and service.
> I am completely at the mercy of your original and continued good intentions as in addition to using your lock-in for a big price hike later you presumably can also snoop on my data.
This is true of any paid service you use right? They can increase your costs at any time. I'm not sure why you think there's something uniquely bad about us for this reason. We have pretty clear values around wanting to know as little about our customers as possible and designing our products end to end around that. We have worked pretty hard at reducing costs, bringing the server price down 60% while doubling its specifications. Our goal is to make this as cost effective and accessible as possible for everyone. We are not interested in locking in customers - it's easy for anyone to take their data off Helm and go to a server of their own making or another service of their choosing. That's not hypothetical - like any company, we have churned customers and supported them in their migration off our product. It's easy to sling these hypotheticals you are concocting but they are not borne out of any reality.
> As far as I can tell there is no really substantial trust differentiator to protonmail.
There is actually a substantial difference. Protonmail holds your data on their servers and therefore can turn it over without a warrant. Well it's encrypted, right? So what could any entity do with that data? Well, Protonmail may be compelled to modify their service to intercept the password on login to decrypt your inbox and turn it over to a government authority (if you don't think that can happen, see what the German government did to Tutanota).
We aren't in a position to do that. Even if the US government came with a court order for your encrypted backups from us, we don't have access to the keys to decrypt them. If we were asked to make firmware changes, we would be retracing the steps of the FBI/Apple San Bernardino case and would enlist the help of the EFF, ACLU and others to fight. I personally believe the case law is pretty clear that they wouldn't win, which is partly why the FBI relented earlier.
> that you can remotely update anytime you feel like it
You make this sound like a terrible thing but really it's not. It allows us to keep our products patched and secured over time.
> In what ways is that better than running an IMAP client on my laptop and using it to send data via protonmail, using my own domain, and keeping offline copies of everything (with a periodic upload to backblaze or some other e2e encrypted backup solution)?
I didn't say people couldn't roll their own solutions. Sure they can - it's just more work, hassle and fragile. And I already covered the tradeoffs of keeping that data in the cloud. Protonmail has access to all your email in the clear (inbound and outbound). We do not and anyone running a server at home would have similar privacy. That's a clear difference.
> Accessing the data is not enough because what you are selling me is not an overpriced and unergonomic hard drive. You are selling me the ability to send, receive and store email (and likely more).
Actually it is because we were talking about data ownership. Your specific dig was about how "own your data" was in any way true ("or that "own your data" is in any meaningful way true" in your parent post).
I've wanted to make something like this too. After years of iteration, my self hosted setup is now completely automated and the automation itself is super simple and organized. It would be pretty simple to setup a simple web app that allows users to simply apply the same automation steps onto their own VPSs. Hardest part would be setting up a secure process for managing user secrets to be honest.
Business wise, I'm not sure I'd be willing to pay for just the automation... in reality you don't use it very often. Could be interesting to try (re)selling tightly knit VPSs, more advanced automation features or support.
I think this solution still captures the self hosted ideology while also providing some cool value. I see people reinventing the wheel all the time while trying to automate self hosted processes... but then again maybe that's why we do it, we like the adventure!
in the late 90's there was whistle, that partnered with isp's and delivered pretty much this - router, email, web host, storage space, calendar, firewall, and easy to configure with an isp.
Having spent the past year frustratingly trying to build these types of things in AWS and spending too much money with mistakes I'd say there is a huge opportunity here. SMB or NFS as a service for example.
https://www.rsync.net/ has been selling this solution for years. Price competitive these days. Not affiliated, just looked at it recently and thought it was extremely cool.
> The diagram alone is more than enough of an argument to dissuade me from giving this a shot right now - it's simply too complicated and too much to manage for the amount of time I can dedicate to it.
Yeah. I have a basic home server and I feel like even with fairly modest needs/desires (Jellyfin, Deluge, Zoneminder, some kind of file syncing, I gave up on photos because my whole family uses Google for that), it's hard to find a reasonable workflow/setup that covers it all. It was basically down to partitioning by VM (proxmox) or partitioned by container (docker), and I went with Docker + Portainer, but I'm not really happy with it; even basic functionality like redeploying a Compose configuration has sat as a feature-ask for three years [1].
Maybe I'm wanting it to be something that it just isn't, and I'd be happier with microk8s and managing the apps as Helm charts. But is that just inviting additional complexity where none is needed?
I used to have a portainer centric setup.. now I just use docker-compose directly. I have my compose split into different files with a makefile to keep things "make start" simple. Highly recommend.
I think the diagram gives a skewed view of how hard this actually is.
I run a very similar setup only my VPS is only a proxy for my home server and it requires very little maintenance. I run everything with docker-compose and I haven't had to work on my setup at all this year and only about 8 hours in 2020 to setup the Wireguard network to replace the ssh tunnels I was using previously for VPS -> server communications.
At the end of the day YMMV and use what you are comfortable with, but it's not as crazy undertaking as it sounds.
Somewhat OT, but never realized how expensive those cloud instances are. For comparison, I pay $4.95/month (billed annually) for a KVM VPS with 2 Ghz, 2 GB RAM, 40gb SSD, 400 GB HDD in the Netherlands. That seems a lot better for selfhosting where you probably want more raw storage than more SSD space.
If that is enough to handle everything you need, then that is definitely a better deal. The electric bill for a similar home server running 24x7 would be more than $6/mo.
You are absolutely right, if you are not familiar with docker-compose, ssh tunnels, wireguard, etc... it will take more time to setup, that being said as far as maintenance go you will probably have a similar experience.
Most of my setup was done through SSH during boring classes in college so I had plenty of time to read documentation and figure out new tools.
I went down an almost identical path/plan, but then stopped due to corruption concerns with doing the VPS / home sync the way that I wanted without a NAS in the middle managing the thing. It’s still possible, but it explodes the complexity.
One of the big things I wanted to accomplish was low cost and easy to integrate / recover from for family in case of bus-factor.
I didn’t expect to compete with the major cloud providers on cost, but the architecture I was dreaming of just wasn’t quite feasible even though it’s tantalizingly close...basically, all the benefits of a p2p internal network with all the convenience of NextCloud and all the export-ability of “just copy all these files to a new disk or cloud provider.”
It’s so close, there’s just always some bottleneck: home upload is too slow, cold cloud storage too hard to integrate with / cache, architecture requires too much maintenance, or similar.
I think NextCloud is very close for personal use, if only there was a plug and play p2p backend datastore / cache backed by plug and play immutable cold storage that could pick up new entries from the p2p layer.
There is a cryptocurrency called siacoin. It offers cloud storage and there exist a nextcloud plugin for it to integrate it as a storage backend. I have some plans on trying this setup.
What do you think?
From what I hear, it’s all pretty nicely containerized/turnkey already. There are even several “meta apps” (Eg: Homelab OS, YUNoHost, etc) which are like the base layer on which many of these services are available as “applications” which have been pre-configured and can be trivially instantiated.
After reading through it all, I think this is more a condemnation of the author's diagram (or at least their decision to put that particular one up-front), than of their process in general, nor the challenge.
Breakdown of (my) issues with the diagram:
- author's interaction with each device is explicitly included, adding unnecessary noise
- "partial" and "full" real-time sync are shown as separate processes, whereas there's no obvious need to differentiate them in such a high-level overview
- devices with "partial" and "full" sync (see above) are colour-coded differently; again differentiation unnecessary
- including onsite & off-site backups in the same diagram is cool but would probably be nicer living in a dedicated backup diagram for better focus
Ow wow, I didn't realise Asciiflow has started supporting extended ASCII - I've been using Asciiflow (via asciiflow.com) for years, but haven't used it for a few months, and missed this being introduced!
As much as I love the way HN's design goes against many trending "UX" conventions, I think the long-time refusal to put in very very basic simple fixes like this one is bizarre.
The messed up presentation on mobile is 100% a mobile bug, for which there is a very easy fix on the dev side, and no good workaround on the commenter side.
I think do a little at a time and keep at it. Over time it adds up.
At sometime you will hit something interesting: Personal Sovereignty.
I've seen other folks hit this in weird ways.
My friend started working on cars with his buddy. They finally got to an old vehicle they took all the way apart and put it together. He had gotten to the point where he could pull the engine and put it on a stand, weld things, paint, redo the wiring harness.
I remember one day I went and looked at it and he sort of casually said, "I can do anything".
Anyway, I think the diagram says something else to me. It says he understands what his setup does enough to show it/explain it to someone else.
I had this with my bicycle at some point -- learning to fix and tweak oneself without having to go to a mechanic was eye-opening. Reminds me of the core premises in Zen and the Art of Motorcycle Maintenance.
The good thing (and one of the reasons I like Linux and dotfiles) is that you can start right away and keep sophisticating your setup as you go. You don't lose that configuration which is akin to knowledge.
I bought a Qnap NAS a month ago. I thought I would get it setup right away for my Linux machines, Macbooks, and network. I was wrong. But I'm slowly learning every couple days and now I have a systemd service that loads two volumes using NFS to my Linux machine.
I want what you want, yet how can society reward the work involved in creating a turnkey version of such, other than through the standard capitalist selfinterested paradigm?
These complicated setups which we see are complex because they try to save costs by using some part of the cloud. Shared VM resources in the cloud, which is all you really need, are dirt cheap compared to the really simple alternative.
Renting space in a rack at a colo facility and putting an nginx server on it is really simple, but it's also expensive compared to the complex solution in the original post.
Depends on what you want - for file sync, there is definitely similar things there. As for generic homelab, I'm writing a similar article atm, and I can tell you, it will be maybe simpler to follow, but definitely not less complex. Everything depends on your needs. What's important here that ideally you only need to do this once, and then only do light maintenance.
Like has been pointed out, this can (and probably should) be done over time.
I have migrated from using cloud provided storage to Nextcloud (been running that for over 4 years now without issues), and have my calendar and contacts in there as well.
My ongoing task is to fully migrate all my images, videos and calibre library from Dropbox to other self hosted entities.
For personal use it seems I agree with other comments that it seems like alot of work. But in a corporate setting it could be useful, wonder if these types of applications (NextCloud) is how the cloud gets broken up eventually.
the mobile app is crucial to me and its search and performance let me down when the car broke down and the time i needed it at the most at the hospital
I'm curious as to for how long you've been using this setup specifically in regard to Nextcloud, and how many and what volume of files you store in it?
I've set up a few Nextcloud instances in the last 2 years on Digital Ocean VPSs and Raspberry Pis and I ran into so many problems and difficulties which scaled with the quantity and size of files I hosted on it. I took care in setting up everything to a relatively solid standard (memcache etc.), but I found Nextcloud to be so unreliable for syncing particularly with the official Android and Linux clients. Plus, there was the whole botched version 20 upgrade.
I find Nextcloud tries to solve too many problems turning it into a bloated mess even for a moderately experienced user.
For file storage only, I've found Syncthing on a Raspberry Pi at home syncing over Zerotier (for when I'm not at home) to be a much more robust, user-friendly and scalable solution, despite it syncing whole folders only.
When I was picking a self-hosted Dropbox alternative, I ended up going with Seafile rather than Nextcloud since it was a lot more focused (just file sync) and people said it was much faster. It's got a few rough edges but the core functionality has been rock solid. Granted, I've only had about 100 GB max stored in it.
I didn’t realise Seafile Pro was free for up to 3 users. I had previously looked but gave up as no self hosted solution was really comparable to OneDrive or Dropbox.
I've been using Seafile for several years, and highly recommend it.
For the past couple of years, I've had it running in Docker on a cheap Azure VM, backed by blob storage. I've got around 2TB in there, and it all works marvellously.
Syncthing is great! Are you using zerotier for transport security. Or does it improve speed as well? Off-site syncs are far too slow for me. Luckily, it's very rare that I have to do them.
I'm using Zerotier so that my devices can stay in sync with my raspberry pi at home when I'm outside of my local network. I found this a simpler solution to dynamic DNS.
We've been using Nextcloud in my home for the better part of a year now, almost completely problem free. I even have auto-updates via watchtower. We have 136 GB of data on it (just checked now). Not sure where that lies compared to your data. It is running on a fairly beefy box though, not a rPi. Only issues so far have been needing to set up cron, which took about 5 minutes doing it the "easy" way (host runs a docker command in it's crontab). Collabora was super annoying to set up, but that was a one-time cost.
Interesting. My volumes were similar and I even had issues with my 'beefy' enough DO VPSs. The primary issues for me were with the clients, especially if I, say, moved a folder of 2000 files from one directory to somewhere else within the Nextcloud drive using the UI. Anyway, I'm not here to troubleshoot that - I've long since decided that it's just too much for my personal simple use case of keeping two folders in sync with each other on different devices. Out of curiosity, how did you install Nextcloud? Snap/Docker/Manual?
Ah, during our migration we did try to move thousands of files from a "Dropbox" folder to a "NextCloud" folder, and indeed the Windows client was not happy. Since it was a one-time thing, the solution was to move the files "manually" over SSH and just run the NextCloud "scan" utility to pick up the changes on disk.
I'm running NextCloud via the official Docker image, reverse proxied through nginx.
My good old friend, the Nextcloud scan utility :) I lost count of the number of times I ran that and the trashbin cleanup. These are both problems I never ever want to have to deal with.
eh, I ran the command, alt-tabbed to something more interesting, and checked later in the day to see that it was done. Never had an issue running it, and only ever needed to when I was doing the initial data migration.
Raspberry Pis are toys, don't use them for anything important.
I've been running NextCloud on an Intel NUC i7 in a Linux container so I can easily snapshot and backup. Recovery is easy in the event of a hardware failure as replacement NUCs are available off the shelf. All I need to do is swap the HD and RAM and I am back in business.
Indeed, I don't really consider personal projects, and niche pdfs I want to sync between devices that important. Also, I didn't want to spend ~£400 on something like an Intel NUC
i have all of that with a pi 4 with an usb 3.0 2tb disk connected..
Raspberry pis are so cheap that you can keep a spare one around just in case the hardware fail.. although i have 3 hosting different stuff in my network and does not fell the need to have a spare one.
i can get a new one by tomorrow cheaper then any used NUC out there and all i have to do is remove the sd card from one and put in the other, plug the usb disk and i am good to go.
my home servers are all pis, with the older turning 8y already, and have not had a single problem with them all this time.. likely my disk will die much sooner then the pi.
Funny headline, because every time I try to self-host anything important like mail, I learn how deep that field is and how little I know and that I'll probably need many many hours to do everything right and in a secure way (and my mails would still have a higher probability to be classified as spam). Then I think: "Screw it, I'll just use GMail"
Interestingly, it doesn't look like the author is self-hosting email. I know mail-in-a-box exists, but even with that I find it's worth the peace of mind to pay someone else for mail hosting.
Same here. Over the past 20 years (25?) I've tried numerous times to self-host email and the issues I had with spam and blacklisting were too involved for me to resolve, and only got deeper. Definitely not something to attempt casually.
No need to self host email, just use your own domain.
That way if a provider gives you lip, you move on. It’s the email address itself that is valuable, the emails can be backed up.
I ran my own email server on my own domain for years. You're right, it's kind of its own special nightmare to integrate all the parts of a comprehensive email system (I eventually started using Zimbra's free server software because of it), but spam effectively killed being able to self-host more. Even if you setup SPF and DKIM and all the rest, you'll find yourself getting blackholed anyway because you're NOT Google or Apple or Microsoft. It's not like I even sent email in bulk either. It was just my normal, personal account. But getting OFF blackhole lists became enough work that I had to route my mail through Gmail anyway, so I just gave up self-hosting entirely. That was, like, 7 or 8 years ago, though, so maybe things are different now, but I doubt it. I expect it to only have gotten worse.
No problem if you pull your complete mailboxes using a tool like isync/mbsync. I have background jobs, in addition to regular backups, which pull to all my powered on computers every 30m. As long as the source is IMAP, it's very easy to not get screwed. I couldn't care less if my email hoster today would lock me out. I'll point my domain elsewhere and I have all the data as maildirs.
Regarding email, I spent some tens of hours setting it up, including implementing DKIM, DMARC, SPF and getting my mail delivered to Gmail and O365. That was over a year ago and things mostly just work with the occasional upgrade or configuration change. You could also save a lot of time by going with a pre-packaged solution. I understand if you don't have time for that, but at least in my experience, self-hosting email isn't the impossible task it's sometimes made out to be.
My experience has been similar. The most important thing that can't be easily resolved is if the server's IP is already on the blocklists, but other than that DKIM, DMARC, SPF, and reverse resolution together solves the ~~magical ritual~~ heuristics that make the big guys happy.
People’s experience with deliverability of messages from self-hosted mail servers seems to be very hit-or-miss but I’m another one of the lucky ones. Rather than using Mail-in-a-box or something similar, I used a cautions step-by-step approach.
About 5 years ago, I read the O’Reilly book, Postfix: The Definitive Guide that had been sitting in my book-shelf for years. I installed and configured Postfix as a sending-only mail server on a Hetzner VPS. I sent a few test emails to GMail accounts and a friend’s Office 365 and they both worked! I then gradually added extra layers of functionality (TLS, DKIM, SPF, DMARC).
Once I was happy that I could successfully send emails, the next step was to receive email: I added MX records for my domain and opened port 25 on the firewall. I was able to use Mutt over SSH to read emails sent to my account. I later installed Dovecot (excellent documentation) and Squirrelmail (lacking in features but was easy to install). I don’t really use web-mail but I’ll probably install Roundcube at some stage and I plan to learn how to use Sieve for automatic filtering.
I thought I’d have serious problems with spam and have to install anti-spam software and/or use black-lists but that hasn’t (yet) been an issue. Simply using Postfix default options along with grey-listing and not accepting messages from invalid (according to SPF records) sources blocks all spam. The only times I received spam was when I had accidentally disabled the grey-listing (the mail logs show I get hundreds of connection attempts with only a tenth of successful connections being genuine). The system actually works better than GMail in that I don’t miss messages that were wrongly flagged as Spam. Another benefit of self-hosting is that I can quickly and easily set up account-specific email addresses, e.g., <hackernews@example.com> – no need for <anthony+hackernews@example.com>
I gradually started using it instead of GMail and it’s now my primary email account for important communication. In the four years of serious use, I haven’t had any problems (touch wood).
Once you start using Sieve, you can also start doing more fine-grained filtering of mail to locations and archiving things which you care about with a couple of bash scripts and systemd timers/cron jobs.
I use gmail too but I also have an email from my cheap hosting as a backup. The big problem with gmail is not that Google is reading my email but that this giants can lock you out without a reason and without a right to appeal.
I am super salty that Sony banned my PlayStation account(used by my son) for 2 months (I have a Plus subscription paid for 1 year too) without no way for me to see the exact reason (was it a text message, or a screenshot that was shared, or just a report from an troll) and no way to contest this. I made my decision and fuck consoles my son will have to learn to use a medium spec PC for gaming.
>The big problem with gmail is not that Google is reading my email but that this giants can lock you out without a reason and without a right to appeal.
I also worry about that. What I do is sync gmail with outlook.com. Now I worry that I just doubled my risk of falling victim to a security breach :-)
You could also look for alternative providers. Posteo for example promises to run on FOSS code, and is endorsed by FSF. Definitely a middle ground between an artisanal email setup and gmail.
Email is basically pointless to host yourself from a privacy perspective. Every email has one or more people on the other end that also get a copy. Privacy and email are mutually exclusive. That said, the alternative doesn't have to be something like gmail, where they can do whatever they want with your data. I use Fastmail and that's "sufficiently private" for my needs.
The Apple example was proven to be inaccurate. The loss of access to email wasn't due to him not paying his Apple Card; it was for not paying for his iCloud services. In other words, his shutoff could have happened if he had his iCloud account set up to charge to any other credit card.
I use Fastmail + a custom domain. Because fastmail is the provider, I am not on any spam lists. My emails make it through. My service is very reliable, so I always get my emails. If fastmail decides to hate me, I can just point my domain MX records somewhere else.
I successfully extricated myself from Gmail to ProtonMail, only to be getting dragged back to Office 365 due to ProtonMail not having a working calendar and FastMail not supporting calendar sharing (to non-FastMail users) or delegation.
It's fine for simple use, but without the ability to create calendar invitations it's far from the competition. (To say nothing of sharing e.g. free/busy with my work calendar or delegation.)
yah, microsoft won office productivity 2 decades ago with exchange and outlook (and owa), not just excel and word. the integration between email, calendar, contacts, documents, and access control is still unmatched, certainly not by google's hodgepodge of web apps.
proton is working on calendaring but it still has a long ways to go.
I like the article and I agree with the sentiment.
I think that self-hosting can be quite a bit of effort, but a tool like Ansible makes it so much easier.
Whatever you choose to do, the most important thing is that you create data(base) backups and store those in an environment that you can control at all times.
There needs to be a viable exit strategy, just a backup is not enough if it takes more time to restore operations/service than is viable from a business perspective.
Perform at least a risk analysis, whatever you choose, make it a conscious, deliberate decision.
But what will you do if people aren't telling you exactly how to run your life and your setup? I certainly appreciate the effort and will be digging into this. I'm so sick of the tyranny. I've started my own 'disconnect' plan, and this is giving me a lot of ideas. I've already deleted Facebook, Amazon (that was a hard one), and well on my way to independence. Google is next, and like another commenter I'm using Proton mail now exclusively. Kudos for your efforts to help those of us that are really struggling right now - much appreciated.
Are you hosting it yourself or are you hosting it with various cloud and Paas companies that just aren't Amazon? Because I read the article and it's the definitely the later. Nice try. Bad headline.
I second this, either get a Synology/Qnap NAS or take an old PC with a couple drives and install OpenMediaVault/Freenas/Unraid. All of these platforms have out-of-the-box solutions that mirror most cloud services. I found homelab redit to be great.
If you get the off-the shelf NAS, get one with at least 2GB of ram! Synology is particularly notorious for selling NAS with 512MB(WTF?!) of ram, and then when you try to run a few applications it grinds to a halt.
NAS fails for smartphone integration. Photos should auto upload. Calendar, todos, and contacts need to show up in the usual apps. It needs to be available from remote.
Synology NASes have various apps for syncing mobile devices, such as DS Photo for uploading photos to Photo Station, Synology Drive for more of a Dropbox approach, and MailPlus for contacts, emails, etcs.
> I’m living in Germany, so the obvious choice was to spin up my instances in Vultr‘s* data center in Frankfurt, as ping is the lowest to that center for me.
The author is probably aware of this, but just in case they aren't: Hetzner is an amazing company with two or three datacenters in Germany. I don't remember if any of them are in Frankfurt, but given they offer VPSs and beefy dedicated machines, I'd be fine trading a couple milliseconds for this flexibility (and overall better pricing, even if Vultr's isn't that expensive as well).
I don't know what qualifies as "amazing company" in your eyes (they're certainly cheap for what you get), but my experience was certainly very bad:
I rented a VPS from them experimentally for a month, then left to go on vacation thinking I had cancelled it, but I had not. They left it running for another month that I hadn't paid for, and then sent the bill for the extra month to collections, so that's presumably affecting my credit score now.
Sending a bill for a recurring service to collections rather than just canceling the service is trash-tier company behavior, IMO. I strongly recommend against using Hetzner.
I can see why that's a bad experience for you, but IMO Hetzner did the right thing here.
What if I forget to pay my AWS bill for a week and, because of that, all my resources get deleted? IIRC AWS will inactivate your account after 3 months without payment, so the same thing would've happened there. 3 months is a nice trade off between "forgot to pay my bills" and "no longer use the service".
ETA: for sure it sucks they sent the bill to collection. Uncalled for, they could've attempted to settle directly with you first.
Agreed. Hetzner is very strict about not leaving a penny of theirs wasted/delayed without compensation. I know a couple friends here in Turkey who were contacted by a local collection agency for late settlement and were brought up with legal proceeding if they're not settling it soon. Aware of this, I started to never ever /forget/ about paying any of my bills on time.
I would say that hosting in a German datacenter owned by a German company is the worst way to get complete exposure to the ham-handed and completely out-of-control German intelligence apparatus. Maybe being a German citizen protects you slightly from the BND, in the same way that the NSA technically doesn't spy on Americans, but I doubt it.
I'm running a few bare-metal Hetzner servers (Falkenstein and Helsinki). Can recommend. Reliable, comparatively cheap and tickets are usually responded to within 24 hours. One time I even got through to the guy swapping defective hard drives on one of the servers in the data center by phone, since there was some other issue.
Ok, I’m SUPER into self hosting, but this article? No way.
1) Duck out isn’t a thing, just stop it.
2) Half the articles cited as examples of corporate abuse were later revealed to be mistakes by the user or easily avoidable pitfalls.
3) Self hosting still requires trust (software you’re running, DNS, domains, ISP, etc...) The line of who to trust and how far is a tough one to answer, even for the informed.
How I solved it:
1) I use well vetted cloud services for things that are difficult/impossible to self host or have a low impact if lost. (Email, domains, github, etc...)
2) I self host things that are absolutely critical with cloud backups. (Files, Photos, code, notes, etc..)
I am perpetually confused about why people think that self-hosting on a VPS solves their privacy and security problems. While I'm sure there are controls in place at reputable VPS providers, it wouldn't be too difficult for them to grab absolutely anything they want. Even disk encryption doesn't save you. You're in a VM, they can watch the memory if they need to.
Using a VPS can also make you more identifiable. Your traffic isn't as easily lost in the noise. The worst thing that I know of people doing is using a VPS for VPN tunneling. While it can have its uses, privacy certainly isn't one of them. You're the only one connecting into it and the only traffic coming out of it.
A setup that probably works is vps -> tor -> vpn or some other order of these three, but I couldn't find any sort of blog that detailed setting up something like this so I imagine very few people are doing it.
VPS doesn't solve privacy and security, it solves getting locked out of your account because some algorithm decided you were peddling child porn.
If you want privacy and security and you don't trust your provider, then you have to build your own hardware and compile everything you run on it from vetted source, including your kernel. You can do it, but most people decide that on balance its better to trust someone.
Howso? The VPS can shut you down as well? You might say the migration path is easier, but there will be a weak link somewhere. Even if you put up a datacenter in the basement you need to connect to the internet somehow which can be taken away.
VPS doesn't solve privacy and security, it solves getting locked out of your account
Does it really? It just seems like instead of trusting a big company that everyone knows, you trust a smaller company that not everyone knows that involves more work for you.
I'm pretty sure I've seen articles on HN where VPS companies (maybe DO?) have kicked people off their infrastructure with zero notice. So, not at all different from being locked out of Apple/Google/Amazon.
Perhaps, depends on your needs, risk model, pricing etc...
I use DO but couldn’t use AWS due to price for example.
Hoping platforms whenever you catch the ire of the gods is a bad CX for this problem space.
As you can see, AWS is far from the only game in town. If you can't find two or three from that list that will meet your needs then perhaps you should reassess your quality metric.
(I note in passing that my preferred provider, Linode, is not even on that list.)
DO = Digital Ocean
Again, depending on your needs, VPS are not a commodity. GCP offers a few things that Azure or AWS don’t, etc...
Often times making sweeping generalizations without deep industry knowledge is a bad idea.
If you don’t even know what DO is, why do you feel experienced enough to argue?
Because I've been self-hosting my internet services on VPS's for the last 15 years on various providers. There are literally dozens of them. They are absolutely a commodity.
"Cloud" and "VPS" are not the same thing. VPS = Virtual Private Server, which is one very specific kind of cloud service. Cloud services in general are not a commodity, but VPS is.
well DO decided to lock me out of my account that I had for years because they decided that I'm a fraud and had to deal with their terrible customer service
So I agree with your sentiment, your details are a little off.
“it wouldn't be too difficult for them to grab absolutely anything they want. Even disk encryption doesn't save you. You're in a VM, they can watch the memory if they need to.”
It would be difficult because you’d have to have host access. VM disk encryption is now tied into an HSM or TPM these days, host access wouldn’t help. As for memory, that is now usually encrypted, so no dice there either. The security of a big name public VPS is astoundingly better than what you can do yourself.
“Using a VPS can make you more identifiable”
I think you have a problem of “threat model” here. You’re mixing up hiding against hackers, governments, etc and just lumping it under “privacy and security”
Using a VPS isn’t going to make you more identifiable to google, because you’re not using google now. Using a VPN isn’t going to make you more identifiable to your ISP, because all they can see is that you have a VPN up.
Why not use a VPS for VPN? Well you’re only right it would suck if your threat model includes governments or hostile actors, me hiding from my ISP or on a public Wi-Fi? Not a problem.
You conflate a few ideas and threat models.
Security = The ability to not have your stuff accessed or changed.
Privacy = The ability to not have your stuff seen.
Anonymity = The ability to not have your stuff linked back to you.
Threat model = Who are you protecting yourself from?
E.g. The steps I take to not get hacked by the NSA are going to be different then the steps I use to make comments on 4chan or whatever are different than the steps I take to use public Wi-Fi.
Ref: I work for Amazon AWS, my opinions are my own insane ramblings.
AMD secure memory encryption and secure encrypted virtualization. Intel probably has something in the works, but today you can take a GCE instance from a signed coreboot through bootloader and kernel with logged attestation at each phase resulting in a VM using per-VM disk encryption key (you have to provide it in the RPC that starts the machine; it's supposedly otherwise ephemeral) with SME encrypted RAM (again, ephemeral per-machine key). Google calls it Confidential VM and Secure Boot for now.
> It would be difficult because you’d have to have host access.
Which AWS has, by definition.
> VM disk encryption is now tied into an HSM or TPM these days, host access wouldn’t help.
Are you passing all of the data through the TPM? If no: you still need to keep the key in memory somewhere, the TPM is just used for offline storage. If yes: the TPM, and the communication with it, is still under AWS' control.
> As for memory, that is now usually encrypted, so no dice there either.
Still need to keep the key somewhere, so same concern as for disk encryption. Except I can pretty much guarantee you're not putting the TPM on the memory's critical path, so...
> The security of a big name public VPS is astoundingly better than what you can do yourself.
Feel free to back such claims up in the future. Because right now this seems to be as false as the rest of your post.
> Using a VPS isn’t going to make you more identifiable to google, because you’re not using google now.
What? It certainly won't make you less identifiable either.
> Using a VPN isn’t going to make you more identifiable to your ISP, because all they can see is that you have a VPN up.
Your VPN provider, on the other hand, can now see all of the traffic, where before they couldn't. So the question is ultimately whether you trust your ISP or VPN provider more.
> Why not use a VPS for VPN? Well you’re only right it would suck if your threat model includes governments or hostile actors, me hiding from my ISP
Sure, if you trust the Amazon over your ISP that makes perfect sense. Then again, this is the Amazon that seems to love forcing their employees to piss in bottles, and is on a huge misinformation campaign against treating their employees properly.
That seems like an upstanding place with great leadership.
> or on a public Wi-Fi? Not a problem.
Makes some sense, but it wouldn't really give you much more than hosting the VPN at home. (Well, you'd still have to do the same calculus here for home ISP vs Amazon.)
> You conflate a few ideas and threat models.
Pot, meet kettle.
> Ref: I work for Amazon AWS, my opinions are my own insane ramblings.
Good to know that AWS employees are either clueless about their own offerings, or deliberately spreading misinformation.
Again, with the insults.
I comment for your benefit, not mine. I already know the right answers here, it is you who are mistaken.
So you can either consider an alternative experience set which undoubtedly differs from yours, or not.
I don’t care either way.
With rclone you can encrypt data locally while uploading. This allows you to host everything from home and use the cloud only for backups, basically end-to-end encrypted.
Vultr/Choopa specifically is like the number 2 source of malicious hosting company traffic we see (number 1 is OVH, cisco's written extensively about them).
They're blacklisted in every environment I touch along with LACNIC and the usual suspects like China and Russia. Their traffic isn't worth it.
I always think of it as – how many examples of "I got locked out of all my data!" would there be if billions of people start following the author's advice? Definitely more than the ~5 they list (whether that is user error or actually Apple/Google/Amazon's fault).
the 'duck it out' thing really made me cringe. we really need to get away from that idea of having a searching verb that is tied to the popular search engines of the day. i use duckduckgo but it might not be around in 10 or 20 years or there might be something better by then so its pointless to expect everyone to keep learning new verbs all the time.
The most valuable thing for me is my photo library. All of them are currently in Google Photos. Is there any easy way to backup just that? I don’t care about my personal email, tasks, calendar etc. It’s just the thought of losing my photos scares me.
https://takeout.google.com/ is exactly what you want. Deselect all, check "Google Photos", click Next, chose your archive format, confirm. It'll take some time but at the end of the process, you got a nice zip with all your Photos in original quality.
I mentioned it in a different comment, but take a look at Syncthing. It does mesh-style backup to synchronize a folder between multiple machines. That provides robustness against hard drive or PC failures, and it's easy to add an offsite node for extra confidence.
I found a solution: Use Photos (iCloud photos) along with Google Photos since I already use an iPhone. Thanks for the comments, but I believe this is the easiest.
Also, his website is very slow, probably because he's not using a CDN. A noble goal, but it has an impact on credibility. The slow website makes me feel like he doesn't care about user experience, which makes me assume that is true for his whole setup, and turns me off from even considering it.
That’s a massive jump to make when he is probably being beaten to hell by HN right now.
That’s not a reasonable chain of expectation. Reevaluate your logic.
"HN doesn't make that much traffic"
Cite your sources, that claim sounds made up. Your inference that if somebody doesn't use a CDN or have acceptable load times for you, that they don't care about user experience, and so their opinion on self hosting is not worth listening to is absurd.
They said they got 18,000 hits in one day. That's a tiny amount of traffic for any decent static website (which this one that we are talking about is). Even assuming they got all that traffic in one hour, that's only 5 requests per second.
Poking around the source code I see the time on the cache is after this comment so I feel the issue is more likely cache configuration wasn't prepared for frontpage of HN. Anecdotally, I'm not experiencing any performance issues anymore.
I'm assuming that if he doesn't care about his website being performant for others, which is its main purpose, then he doesn't care about his other apps doing their main job well either.
Depending on what you need, a NAS + Syncthing is much simpler than the linked article. Building a PC isn't hard, and keeps prices down. These days, a RPi 4+2 USB HDDs would run circles around the motherboard on my NAS.
Syncthing is a great continuous backup solution. I use ~/NOTES as a scratchpad, and it updates automatically between my various computers. It gives you pretty granular control over shares, and I back up critical stuff to a cloud provider.
That said, there's no calendar/email/notes. XigmaNAS is built on FreeBSD, and will happily run NextCloud or a photo gallery or whatever.
If you really want control, what matters more is you having control of your own domain and encrypting what doesn't need to be public, such as backups and notes. Managing a self hosted system is often more expensive and more time consuming, and often those self hosted services store unencrypted versions of your data. But now you have to maintain the security of it yourself, usually worse than professional services, and your still one subpoena or hack away from it being exposed.
In the end you are still just as vulnerable getting booted off with VPSs like you are with google, but with domain control you can still switch hosts without losing your address, and you usually have customer support.
I've had my own domain for something like 22 years now, but it's been a long time since I used it to actually host stuff. Email in particular I gave up over a decade ago and pointed at a hosting provider. I still read that email with mutt over ssh.
I had the same thought as the title of the article go through my head, but we ended up with a simpler setup as I wanted something I don't have to constantly mess with:
* Put together an overbuilt NAS box running ZFS On Linux
* Simple docker-compose file for all services
* Backups through borgmatic (via ZFS snapshots)
* Auto-updates through watchtower
* Punted on email and use FastMail, switched to our own domain from gmail
Services we run include:
* PhotoPrism for semi-Google Photos functionality
* Nextcloud and Collabora for file sync, sharing
* Kodi for home media
* Tiddlywiki
* DDNS through Gandi since we're on a dynamic IP
* PiHole for some ad/privacy protection
* Robocert for SSL
* Nginx to reverse proxy everything
It wasn't _easy_ to set up, but in a year, any given week I typically spend 0 hours dealing with it. No problem that _has_ cropped up has taken more than a few minutes to fix, mostly around docker networking and auto-restarting containers after Watchtower auto-updates them, a problem I've since fixed.
This setup seems way easier than k3s or some other recommendations, doesn't require much new knowledge, and is as portable as I need it to be. If needed I could plop the docker-compose on a new machine, change some mount points, and largely be up and running again quickly. It's let us switch to "deGoogled" phones and unplug from almost every hosted service we used to use.
I created codehawke.com architecture from scratch to avoid hosting my content on other people's platforms. I make way more money than with platforms like Udemy. I think we should all be moving away from other people's platforms and tools.
For some of us, we turn it into a hobby. Only difference is that the technical knowledge and experience gained at work, can also be applied at home. (without a lot of restrictions).
What I meant is that I need some time to _not_ be productive. Like, actively not being productive. Literally wasting time for the sake of getting some peace of mind and true relaxation.
If your personal life is filled with productivity tools and optimizations, at what time in your daily life your are _not_ worried about productivity? If this time is zero, I think it's kind of sad and maybe even unhealthy. It's just my opinion, of course :)
I agree with you on the importance of non-productive time but I've found having my own infrastructure makes my life smoother day to day in exchange for some upfront cost. It's a tricky balance, and as many other commenters have mentioned that initial cost can end up not being so initial - though I think most people who engage in this 'hobby' generally find both the process and the product rewarding.
I really love the idea of self-hosting, but man, you have to go through 9 layers of configuration hell and come back out alive. It's not necessarily fun programming - more of changing variables and running commands, which you might get wrong anyway.
I wonder if there's a viable business model for this.
Automate the setup through scripts and process automation for any provider. You pay a one time fee + a reasonable amount for maintenance and for resilience built in. I would pay for it if the price is reasonable.
"A drinking game recommendation (careful, it may and probably will lead to alcoholism): take a shot every time you find out how someone’s data has been locked and their business was jeopardized because they didn’t own, or at least back up their data."
You could play a reverse game of every time somebody lost all of their data (or more probably, photos) because they owned everything and _thought_ they had backups too. (e.g.: when the OVH datacenter burned down)
493 comments
[ 2.7 ms ] story [ 321 ms ] threadIt seems like 95% of the adherents to self-hosting do it as a hobby but pretend it's prudence.
As long as your house burning down and your cloud getting locked don't occur on the same day, you're golden and thus no messing with blue-rays and bank security boxes.
If you have a friend with good internet, you could put a NAS in his house or even give it to him for him to use, just get a quota for some storage.
> Every last weekend of the month, I will manually backup all the data to Blu-ray discs. Not once, but twice. One copy goes to a safe storage space at home and the other one ends up at a completely different location.
This is one paragraph after mentioning a 2TB+2TB NAS. Even assuming that's RAID1, a standard Blu-ray only stores 50 GB, so you need 40 of those. And then you need another 40 for the other location... every month?? Honestly it's probably cheaper to buy a new 4 TB hard drive every month.
If you're a cheapskate like me, backing up your encrypted (e.g. Borg) backups to a cloud provider like Google Drive isn't a bad option. My org provides me with unlimited cloud space, so I have hundreds of encrypted gigs on Google Drive. No reason to think it'll disappear overnight.
The reason for blurays might be that he's following rule 2 of the 3-2-1 Backup Strategy. 3 backups, 2 different types of storage media, 1 copy off-site.
If your house is hit by lightning it could wipe all your magnetic and solid state drives, but optical discs would probably be ok.
The "my house was hit by lightning" case is covered pretty well by your 1 offsite backup.
1. https://www.acronis.com/en-us/articles/backup-rule/ 2. https://www.carbonite.com/blog/article/2016/01/what-is-3-2-1... 3. https://www.backblaze.com/blog/the-3-2-1-backup-strategy/
>Backup archives are mountable as userspace filesystems for easy interactive backup examination and restores (e.g. by using a regular file manager).
looks like a really nice feature of borg. Thanks.
Anyway, my "home burned down rescue bundle" consists of a flash drive with a keepass export of my password vault and encrypted borg repo key / rsync.net ssh keys at the office.
Slightly less accessible in this pandemic world, but no safety deposit boxes needed.
I can't say whether this is a good choice for any particular use-case but I appreciate that it is an option ...
For those that are curious, yes, you can later export the key from your repo so that you have a copy elsewhere ...
The offsite protects against catastrophic failure or theft. I would still like to add in another backup for critical data such as databases, I may use another cloud provider that has geo redundant storage for those.
Oh I trust them not to delete my encrypted data and that's all that I'd ask. But I still don't have backups on hardware that I don't own: it's super expensive. From the cost of hosted backups, I could buy a hard drive of the same storage space every three months or something. And hard drives live longer than three months.
Putting a raspberry pi with a big hard drive at a friend's place is rather power efficient, dead simple to setup if you have any technical knowledge at all, and off-site. Use Restic or something to avoid having to trust your friend or whoever will burglarize the place. They have an Internet connection anyway and where I live, there are no bandwidth caps.
BUT - I'm really thankful for people who keep posting and sharing these sorts of projects; they're the ones iterating the process for the rest of us who need something a bit more turn-key.
I'm excited to see this eventually result in something like the following:
- Standard / Easy to update containerized setup.
- Out of the box multi-location syncs (e.g. home, VPS, etc.)
- Takes 5 minutes to configure/add new locations
I want this to be as easy as adding a new AP to my mesh wifi system at home: plug it in, open the app, name the AP, and click "Done".
(Edit - formatting)
Like, how do you persuade the audience of enthusiasts (think: Unifi buyers) to pay for a subscription to managed software they run on their own computers, raspis, whatever? I would probably spend $10/mo on something like that, but much above that and you'd be fighting against the armchair commentary of users who won't appreciate the effort that goes into stability and will basically have a "no wireless, less space than a Nomad, lame" attitude.
it’s mostly tough because of the high upfront capital costs (manufacturing, r&d, and marketing). people still talk fondly about discontinued apple routers and what nest could have been as an independent venture, for example.
This seems a lot easier to me than on-prem cloud services, either in BYOH form ("but it's just software") or as a packaged appliance ("another hub to install, really?").
I would say that the closest thing to this right now for paid is coming from the storage side— NAS providers like Synology using hardware sales to support a limited ecosystem of "one click" deployable apps. And for free, it's ecosystems like HomeAssistant, which a lot of people just deploy as a fire-and-forget RPi image, but as expected with a free ecosystem, as soon as you get off the ultra-common use cases, you're reading source code to figure out how it works, and wading through a tangle of unmaintained "community" plugins that only do half of what you want.
home audio/theater from prior to the internet revolution might be a good analogy: a bunch of separate boxes that each provide tailored functionality but all work together seemlessly without a lot of technical knowledge. that, but for all sorts of computing devices.
I also think it's still a little too nerd-focused for the average consumer. I'd say I know far more about security, networking and hardware than the average consumer but, compared to the HN crowd, I know next to nothing. I struggle to use a lot of the current solutions because they get bogged down in doing cool technical stuff that is so far outside the scope of the average potential user's wants/needs or the DIY solution will be "easy"... for someone with an extensive CS background and years of experience.
for instance, plug in a smart device and have confidence that it's not doing surreptitious things behind your back, because it's automatically segregated into its own vlan and given only enough network access to be controlled by you without needing to know much about the underlying technologies involved.
"automatically segregated into its own vlan"
Aren't these goals fundamentally at odds? I would imagine that Joe consumer (if they care at all about any of this) would be rather more inclined to entrust the role of orchestrating/segregating their home network devices to an entity like Google than to some random startup.
When you lose trust you end up with your crazy uncle leaving Fox News for Alex Jones and YouTube. You have people becoming QAnon followers.
I say this not to make a political point, but that the problem is fundamentally hopeless and I see no way out. You end up landing on one side of the fence or the other. You either just don't think about it and continue to use Google and Facebook and remain ignorant of the problem, or you spiral down the never-ending hole of despair.
We have seen articles recently that tell us not even Signal can be fully trusted. Whether or not it's true is beside the point. The point is, not even the HN crowd is safe from the cliff of paranoia. The seed of doubt has been planted.
Is someone going to trust a small tech startup in 2021? No, not like they would have in 1997. The market for trust has effectively been sealed off today. Because, paradoxically, the Googles and Facebooks ruined it all. They stripped us (all of us, not just HN) of our innocence and naivety. We know not to trust Google, but they are also a known known. A small tech company is a total unknown. We're familiar with how Google is going to bend us over. So if someone is going to do us dirty, it may as well be a known entity. Or... you go and build a cabin in the woods and start writing manifestos.
most people, whether they rationalize it or not, are cognizant that we live in the grey gradient of trust for various companies and brands. the vector field is all sorts of wacky and inscrutible, but maybe we can point a few of those vectors in the right direction and some folks will happily slide down it to better (but perhaps not perfect) safety and privacy.
The most prominent of which is "What happens when they drop support for my use case/lock me out of my account?"
Unfortunately, the cost of running your own one-off solution is rather high. And doubly unfortunately, while I would pay money for a box that I could plug into my computer that provided all of these services, I wouldn't pay enough money to justify someone building it, and selling it to me.
On the software side, integrate tightly with your own subscription services (offsite backups, VPS, etc) to upsell to those who want that, and win over the enthusiast crowd by making it possible to host your own alternatives to those services with a little technical know-how.
Open source most components to appeal to enthusiasts, but keep the secret sauce that makes everything seamless and easy to use "source available" so you don't unintentionally turn your core business into a commodity.
Seems viable to me.
https://en.wikipedia.org/wiki/Windows_Home_Server
At that point, I might as well just go with a paid ProtonMail or similar solution.
My expectations for self hosted isn't to have annual or monthly fees.
If you want to self-host email, you need a trustworthy static IP address with reverse DNS. It's considerably more expensive to get this from an ISP. Our annual fee also includes storage for offsite backups. You don't get the same privacy assurances using Protonmail as you do with self-hosting either. For example, Protonmail is privy to the content of all outbound email messages in the clear unless you are communicating with the recipient using E2EE.
From a cost perspective, Helm V2 starts at $199 for 256GB of storage. First year costs, including subscription work out to $298. With Protonmail, their entry level plan with added storage at the same price buys you an inbox with about 28GB, a small fraction of what you would get with Helm storage-wise, not to mention we don't limit users, email addresses, domains, etc.
I and probably many others would be OK with paying for the upsell part, if it's an optional convenience, but nothing I saw on your site indicates it is, or that "own your data" is in any meaningful way true. How do I own my data if any use of it requires me running stuff on your proprietary box, subscribing to your proprietary service?
Data from Helm is accessible using IMAP, SMTP, CardDAV, CalDAV, WebDAV on the local network (without requiring our service). You own the device, you own the data. There is a standards-based way of accessing that data just as there is with the hard drive from Best Buy.
I can plug the Best Buy harddrive into almost any computer/SAN I want to and utilize for the purpose I bought for without any lock in. Using the hard drive for my data requires very little trust in Best Buy's good intentions at the time of purchase and zero trust in the continued existence, technical competence or good intentions beyond that -- it is very unlikely that best buy will find a way to snoop on my data even if they wanted to. Everything will continue to work fully as intended until mechanical failure sets in. Feels like ownership to me.
In your case I rent some box from you which will lose almost all its intended function the moment your company goes bust or I stop paying you an annual fee. Furthermore it seems I am completely at the mercy of your original and continued good intentions as in addition to using your lock-in for a big price hike later you presumably can also snoop on my data. As far as I can tell there is no really substantial trust differentiator to protonmail. I have to trust them when they claim they won't read my email and are competent enough to keep things secure and I have to trust you (and continue to trust you as long as I want to use the device) that you will encrypt my data and not exfiltrate any of it (or the private key), and furthermore that you run your servers securely enough that no third party will. But what is to stop it? The box is running closed source software that you can remotely update anytime you feel like it, right? I have physical access, but since I don't control the software, what use is that?
Maybe I didn't understand something right, but so far this does not feel like ownership to me.
It sounds more like the worst of all worlds: the lock-in and lack of ownership of a proprietary cloud-based subscription service with the added hassle, inconvenience, downtime and costs of babysitting (and supplying electricity to) a cloud server for you, the provider.
> Data from Helm is accessible using IMAP, SMTP, CardDAV, CalDAV, WebDAV on the local network (without requiring our service).
In what ways is that better than running an IMAP client on my laptop and using it to send data via protonmail, using my own domain, and keeping offline copies of everything (with a periodic upload to backblaze or some other e2e encrypted backup solution)? That seems to offer about the same control over my data, but is cheaper, easier, more convenient, has higher redundancy/uptime and if anything less lock-in. It also doesn't require an additional device that has no use beyond adding an additional failure point and cost center that's my responsibility.
> There is a standards-based way of accessing that data just as there is with the hard drive from Best Buy.
Accessing the data is not enough because what you are selling me is not an overpriced and unergonomic hard drive. You are selling me the ability to send, receive and store email (and likely more).
Don't get me wrong, I kind of like the idea of buying a physical box and a subscription service to self host stuff in a way that gives me better control over my data for an acceptable amount of hassle. But that really requires some amount of openness/auditability and interoperability that currently appears to be absent.
No - we do not rent hardware. When people buy the server from us, they own it. Full stop. There are ongoing costs to make email at home work: a static IP address with good reputation, a security gateway, traffic, etc. If people don't want to pay us for those costs, they will pay them to an ISP and/or an infrastructure provider like AWS. The ease of setup and management comes from the integration of hardware, software and service.
> I am completely at the mercy of your original and continued good intentions as in addition to using your lock-in for a big price hike later you presumably can also snoop on my data.
This is true of any paid service you use right? They can increase your costs at any time. I'm not sure why you think there's something uniquely bad about us for this reason. We have pretty clear values around wanting to know as little about our customers as possible and designing our products end to end around that. We have worked pretty hard at reducing costs, bringing the server price down 60% while doubling its specifications. Our goal is to make this as cost effective and accessible as possible for everyone. We are not interested in locking in customers - it's easy for anyone to take their data off Helm and go to a server of their own making or another service of their choosing. That's not hypothetical - like any company, we have churned customers and supported them in their migration off our product. It's easy to sling these hypotheticals you are concocting but they are not borne out of any reality.
> As far as I can tell there is no really substantial trust differentiator to protonmail.
There is actually a substantial difference. Protonmail holds your data on their servers and therefore can turn it over without a warrant. Well it's encrypted, right? So what could any entity do with that data? Well, Protonmail may be compelled to modify their service to intercept the password on login to decrypt your inbox and turn it over to a government authority (if you don't think that can happen, see what the German government did to Tutanota).
We aren't in a position to do that. Even if the US government came with a court order for your encrypted backups from us, we don't have access to the keys to decrypt them. If we were asked to make firmware changes, we would be retracing the steps of the FBI/Apple San Bernardino case and would enlist the help of the EFF, ACLU and others to fight. I personally believe the case law is pretty clear that they wouldn't win, which is partly why the FBI relented earlier.
> that you can remotely update anytime you feel like it
You make this sound like a terrible thing but really it's not. It allows us to keep our products patched and secured over time.
> In what ways is that better than running an IMAP client on my laptop and using it to send data via protonmail, using my own domain, and keeping offline copies of everything (with a periodic upload to backblaze or some other e2e encrypted backup solution)?
I didn't say people couldn't roll their own solutions. Sure they can - it's just more work, hassle and fragile. And I already covered the tradeoffs of keeping that data in the cloud. Protonmail has access to all your email in the clear (inbound and outbound). We do not and anyone running a server at home would have similar privacy. That's a clear difference.
> Accessing the data is not enough because what you are selling me is not an overpriced and unergonomic hard drive. You are selling me the ability to send, receive and store email (and likely more).
Actually it is because we were talking about data ownership. Your specific dig was about how "own your data" was in any way true ("or that "own your data" is in any meaningful way true" in your parent post).
Business wise, I'm not sure I'd be willing to pay for just the automation... in reality you don't use it very often. Could be interesting to try (re)selling tightly knit VPSs, more advanced automation features or support.
I think this solution still captures the self hosted ideology while also providing some cool value. I see people reinventing the wheel all the time while trying to automate self hosted processes... but then again maybe that's why we do it, we like the adventure!
Yeah. I have a basic home server and I feel like even with fairly modest needs/desires (Jellyfin, Deluge, Zoneminder, some kind of file syncing, I gave up on photos because my whole family uses Google for that), it's hard to find a reasonable workflow/setup that covers it all. It was basically down to partitioning by VM (proxmox) or partitioned by container (docker), and I went with Docker + Portainer, but I'm not really happy with it; even basic functionality like redeploying a Compose configuration has sat as a feature-ask for three years [1].
Maybe I'm wanting it to be something that it just isn't, and I'd be happier with microk8s and managing the apps as Helm charts. But is that just inviting additional complexity where none is needed?
[1]: https://github.com/portainer/portainer/issues/1753
Ben running like this for 3 years. No fuss.
I run a very similar setup only my VPS is only a proxy for my home server and it requires very little maintenance. I run everything with docker-compose and I haven't had to work on my setup at all this year and only about 8 hours in 2020 to setup the Wireguard network to replace the ssh tunnels I was using previously for VPS -> server communications.
At the end of the day YMMV and use what you are comfortable with, but it's not as crazy undertaking as it sounds.
[1]: https://www.vultr.com/features/one-click-apps/
[2]: https://marketplace.digitalocean.com
You can also enable automatic backups for your servers.
But it looks like I got it on a sale, TERAKVM-400 comes down to $6.94 at normal prices.
Most of my setup was done through SSH during boring classes in college so I had plenty of time to read documentation and figure out new tools.
One of the big things I wanted to accomplish was low cost and easy to integrate / recover from for family in case of bus-factor.
I didn’t expect to compete with the major cloud providers on cost, but the architecture I was dreaming of just wasn’t quite feasible even though it’s tantalizingly close...basically, all the benefits of a p2p internal network with all the convenience of NextCloud and all the export-ability of “just copy all these files to a new disk or cloud provider.”
It’s so close, there’s just always some bottleneck: home upload is too slow, cold cloud storage too hard to integrate with / cache, architecture requires too much maintenance, or similar.
I think NextCloud is very close for personal use, if only there was a plug and play p2p backend datastore / cache backed by plug and play immutable cold storage that could pick up new entries from the p2p layer.
https://nextcloud.com/blog/introducing-cloud-storage-in-the-...
Use a cloud provider or your own automation to create a cluster, then apply a set of configs to bring your services up.
Those curious can check out /r/SelfHosted.
Breakdown of (my) issues with the diagram:
- author's interaction with each device is explicitly included, adding unnecessary noise
- "partial" and "full" real-time sync are shown as separate processes, whereas there's no obvious need to differentiate them in such a high-level overview
- devices with "partial" and "full" sync (see above) are colour-coded differently; again differentiation unnecessary
- including onsite & off-site backups in the same diagram is cool but would probably be nicer living in a dedicated backup diagram for better focus
Here's a simplified version of the same diagram:
The messed up presentation on mobile is 100% a mobile bug, for which there is a very easy fix on the dev side, and no good workaround on the commenter side.
¯\_(ツ)_/¯
At sometime you will hit something interesting: Personal Sovereignty.
I've seen other folks hit this in weird ways.
My friend started working on cars with his buddy. They finally got to an old vehicle they took all the way apart and put it together. He had gotten to the point where he could pull the engine and put it on a stand, weld things, paint, redo the wiring harness.
I remember one day I went and looked at it and he sort of casually said, "I can do anything".
Anyway, I think the diagram says something else to me. It says he understands what his setup does enough to show it/explain it to someone else.
I bought a Qnap NAS a month ago. I thought I would get it setup right away for my Linux machines, Macbooks, and network. I was wrong. But I'm slowly learning every couple days and now I have a systemd service that loads two volumes using NFS to my Linux machine.
Renting space in a rack at a colo facility and putting an nginx server on it is really simple, but it's also expensive compared to the complex solution in the original post.
I have migrated from using cloud provided storage to Nextcloud (been running that for over 4 years now without issues), and have my calendar and contacts in there as well.
My ongoing task is to fully migrate all my images, videos and calibre library from Dropbox to other self hosted entities.
It is a process made over a long time.
the mobile app is crucial to me and its search and performance let me down when the car broke down and the time i needed it at the most at the hospital
i wish it was the not this way i really do
I've set up a few Nextcloud instances in the last 2 years on Digital Ocean VPSs and Raspberry Pis and I ran into so many problems and difficulties which scaled with the quantity and size of files I hosted on it. I took care in setting up everything to a relatively solid standard (memcache etc.), but I found Nextcloud to be so unreliable for syncing particularly with the official Android and Linux clients. Plus, there was the whole botched version 20 upgrade.
I find Nextcloud tries to solve too many problems turning it into a bloated mess even for a moderately experienced user.
For file storage only, I've found Syncthing on a Raspberry Pi at home syncing over Zerotier (for when I'm not at home) to be a much more robust, user-friendly and scalable solution, despite it syncing whole folders only.
For the past couple of years, I've had it running in Docker on a cheap Azure VM, backed by blob storage. I've got around 2TB in there, and it all works marvellously.
I'm running NextCloud via the official Docker image, reverse proxied through nginx.
I've been running NextCloud on an Intel NUC i7 in a Linux container so I can easily snapshot and backup. Recovery is easy in the event of a hardware failure as replacement NUCs are available off the shelf. All I need to do is swap the HD and RAM and I am back in business.
Raspberry pis are so cheap that you can keep a spare one around just in case the hardware fail.. although i have 3 hosting different stuff in my network and does not fell the need to have a spare one.
i can get a new one by tomorrow cheaper then any used NUC out there and all i have to do is remove the sd card from one and put in the other, plug the usb disk and i am good to go.
my home servers are all pis, with the older turning 8y already, and have not had a single problem with them all this time.. likely my disk will die much sooner then the pi.
And of course SPF, DKIM, DMARC are all prerequisites.
But if you can avoid the first two problems, hosting email yourself is rock-solid reliable and requires very little attention.
About 5 years ago, I read the O’Reilly book, Postfix: The Definitive Guide that had been sitting in my book-shelf for years. I installed and configured Postfix as a sending-only mail server on a Hetzner VPS. I sent a few test emails to GMail accounts and a friend’s Office 365 and they both worked! I then gradually added extra layers of functionality (TLS, DKIM, SPF, DMARC).
Once I was happy that I could successfully send emails, the next step was to receive email: I added MX records for my domain and opened port 25 on the firewall. I was able to use Mutt over SSH to read emails sent to my account. I later installed Dovecot (excellent documentation) and Squirrelmail (lacking in features but was easy to install). I don’t really use web-mail but I’ll probably install Roundcube at some stage and I plan to learn how to use Sieve for automatic filtering.
I thought I’d have serious problems with spam and have to install anti-spam software and/or use black-lists but that hasn’t (yet) been an issue. Simply using Postfix default options along with grey-listing and not accepting messages from invalid (according to SPF records) sources blocks all spam. The only times I received spam was when I had accidentally disabled the grey-listing (the mail logs show I get hundreds of connection attempts with only a tenth of successful connections being genuine). The system actually works better than GMail in that I don’t miss messages that were wrongly flagged as Spam. Another benefit of self-hosting is that I can quickly and easily set up account-specific email addresses, e.g., <hackernews@example.com> – no need for <anthony+hackernews@example.com>
I gradually started using it instead of GMail and it’s now my primary email account for important communication. In the four years of serious use, I haven’t had any problems (touch wood).
I am super salty that Sony banned my PlayStation account(used by my son) for 2 months (I have a Plus subscription paid for 1 year too) without no way for me to see the exact reason (was it a text message, or a screenshot that was shared, or just a report from an troll) and no way to contest this. I made my decision and fuck consoles my son will have to learn to use a medium spec PC for gaming.
I also worry about that. What I do is sync gmail with outlook.com. Now I worry that I just doubled my risk of falling victim to a security breach :-)
Most hosting providers with a $3/mo CPanel option gives you an out-of-the-box, highly configurable email server.
Email is basically pointless to host yourself from a privacy perspective. Every email has one or more people on the other end that also get a copy. Privacy and email are mutually exclusive. That said, the alternative doesn't have to be something like gmail, where they can do whatever they want with your data. I use Fastmail and that's "sufficiently private" for my needs.
Or if you forgot to pay your apple card you don't loose access to the email etc.
I think diversification is valuable here.
TODO: need a non-fastmail backup of my email
It's fine for simple use, but without the ability to create calendar invitations it's far from the competition. (To say nothing of sharing e.g. free/busy with my work calendar or delegation.)
proton is working on calendaring but it still has a long ways to go.
So we're just going to pretend that iCloud doesn't exist, then? Is that it?
I think that self-hosting can be quite a bit of effort, but a tool like Ansible makes it so much easier.
Whatever you choose to do, the most important thing is that you create data(base) backups and store those in an environment that you can control at all times.
There needs to be a viable exit strategy, just a backup is not enough if it takes more time to restore operations/service than is viable from a business perspective.
Perform at least a risk analysis, whatever you choose, make it a conscious, deliberate decision.
1. Don't feed the FAANG
2. Store your SoR media, notes, documents on your own NAS
3. Automate a backup of the NAS, preferably both on and off site (I use rsync from a pi + large disk + cloud blob storage)
If you get the off-the shelf NAS, get one with at least 2GB of ram! Synology is particularly notorious for selling NAS with 512MB(WTF?!) of ram, and then when you try to run a few applications it grinds to a halt.
https://syncthing.net/
https://www.synology.com/en-nz/dsm/feature/photo_station https://www.synology.com/en-nz/dsm/feature/drive https://www.synology.com/en-nz/dsm/feature/mailplus
The author is probably aware of this, but just in case they aren't: Hetzner is an amazing company with two or three datacenters in Germany. I don't remember if any of them are in Frankfurt, but given they offer VPSs and beefy dedicated machines, I'd be fine trading a couple milliseconds for this flexibility (and overall better pricing, even if Vultr's isn't that expensive as well).
I rented a VPS from them experimentally for a month, then left to go on vacation thinking I had cancelled it, but I had not. They left it running for another month that I hadn't paid for, and then sent the bill for the extra month to collections, so that's presumably affecting my credit score now.
Sending a bill for a recurring service to collections rather than just canceling the service is trash-tier company behavior, IMO. I strongly recommend against using Hetzner.
What if I forget to pay my AWS bill for a week and, because of that, all my resources get deleted? IIRC AWS will inactivate your account after 3 months without payment, so the same thing would've happened there. 3 months is a nice trade off between "forgot to pay my bills" and "no longer use the service".
ETA: for sure it sucks they sent the bill to collection. Uncalled for, they could've attempted to settle directly with you first.
How I solved it: 1) I use well vetted cloud services for things that are difficult/impossible to self host or have a low impact if lost. (Email, domains, github, etc...) 2) I self host things that are absolutely critical with cloud backups. (Files, Photos, code, notes, etc..)
Using a VPS can also make you more identifiable. Your traffic isn't as easily lost in the noise. The worst thing that I know of people doing is using a VPS for VPN tunneling. While it can have its uses, privacy certainly isn't one of them. You're the only one connecting into it and the only traffic coming out of it.
If you want privacy and security and you don't trust your provider, then you have to build your own hardware and compile everything you run on it from vetted source, including your kernel. You can do it, but most people decide that on balance its better to trust someone.
Yes. but VPS is a standardized commodity. If one provider shuts you down you can just switch to another.
Does it really? It just seems like instead of trusting a big company that everyone knows, you trust a smaller company that not everyone knows that involves more work for you.
I'm pretty sure I've seen articles on HN where VPS companies (maybe DO?) have kicked people off their infrastructure with zero notice. So, not at all different from being locked out of Apple/Google/Amazon.
Yes. VPS is a standardized commodity. If one provider shuts you down you can just move to another.
https://techjury.net/best/vps-hosting/
As you can see, AWS is far from the only game in town. If you can't find two or three from that list that will meet your needs then perhaps you should reassess your quality metric.
(I note in passing that my preferred provider, Linode, is not even on that list.)
NOT EVERYONE HAS THE SAME NEEDS AS YOU.
DIFFERENT CLOUD PROVIDERS OFFER DIFFERENT SERVICES
YOU OBVIOUSLY HAVE NO IDEA WHAT THE MARKET OFFERS
/rant
“Using a VPS can make you more identifiable” I think you have a problem of “threat model” here. You’re mixing up hiding against hackers, governments, etc and just lumping it under “privacy and security” Using a VPS isn’t going to make you more identifiable to google, because you’re not using google now. Using a VPN isn’t going to make you more identifiable to your ISP, because all they can see is that you have a VPN up. Why not use a VPS for VPN? Well you’re only right it would suck if your threat model includes governments or hostile actors, me hiding from my ISP or on a public Wi-Fi? Not a problem.
You conflate a few ideas and threat models.
Security = The ability to not have your stuff accessed or changed. Privacy = The ability to not have your stuff seen. Anonymity = The ability to not have your stuff linked back to you. Threat model = Who are you protecting yourself from? E.g. The steps I take to not get hacked by the NSA are going to be different then the steps I use to make comments on 4chan or whatever are different than the steps I take to use public Wi-Fi.
Ref: I work for Amazon AWS, my opinions are my own insane ramblings.
Common on laptops, but I wouldn’t assume that for systems/SANs in a data center, much less their virtual disks. Would love to be corrected.
Which AWS has, by definition.
> VM disk encryption is now tied into an HSM or TPM these days, host access wouldn’t help.
Are you passing all of the data through the TPM? If no: you still need to keep the key in memory somewhere, the TPM is just used for offline storage. If yes: the TPM, and the communication with it, is still under AWS' control.
> As for memory, that is now usually encrypted, so no dice there either.
Still need to keep the key somewhere, so same concern as for disk encryption. Except I can pretty much guarantee you're not putting the TPM on the memory's critical path, so...
> The security of a big name public VPS is astoundingly better than what you can do yourself.
Feel free to back such claims up in the future. Because right now this seems to be as false as the rest of your post.
> Using a VPS isn’t going to make you more identifiable to google, because you’re not using google now.
What? It certainly won't make you less identifiable either.
> Using a VPN isn’t going to make you more identifiable to your ISP, because all they can see is that you have a VPN up.
Your VPN provider, on the other hand, can now see all of the traffic, where before they couldn't. So the question is ultimately whether you trust your ISP or VPN provider more.
> Why not use a VPS for VPN? Well you’re only right it would suck if your threat model includes governments or hostile actors, me hiding from my ISP
Sure, if you trust the Amazon over your ISP that makes perfect sense. Then again, this is the Amazon that seems to love forcing their employees to piss in bottles, and is on a huge misinformation campaign against treating their employees properly.
That seems like an upstanding place with great leadership.
> or on a public Wi-Fi? Not a problem.
Makes some sense, but it wouldn't really give you much more than hosting the VPN at home. (Well, you'd still have to do the same calculus here for home ISP vs Amazon.)
> You conflate a few ideas and threat models.
Pot, meet kettle.
> Ref: I work for Amazon AWS, my opinions are my own insane ramblings.
Good to know that AWS employees are either clueless about their own offerings, or deliberately spreading misinformation.
Seems like a place that I'd love to trust...
::shrugs:: I don't work for that part of AWS. My opinion came from other experience.
You're not only wrong, but you managed to insult me while being wrong. That's the worst kind of wrong.
If you want some further reading, there is some cool work being done in this space.
https://docs.aws.amazon.com/enclaves/latest/user/nitro-encla...
https://cloud.google.com/blog/products/identity-security/int...
https://azure.microsoft.com/en-us/solutions/confidential-com...
They're blacklisted in every environment I touch along with LACNIC and the usual suspects like China and Russia. Their traffic isn't worth it.
Well, 1) sorry but you don't get to decide this, 2) how would anything ever become a thing if people were not allowed to invent new things?
- I'm not favoring the term just opposing your commanding
There's a decent guide here: https://ubuntu.com/blog/safely-backup-google-photos.
I run this every night on a raspberry pi, syncing them to my local NAS which is in turn backed up to cloud storage.
You can use Takeout to bulk-download photos: https://support.google.com/accounts/answer/9666875?hl=en
I don't know if you can automate syncing Google Photos to a local disk. Wouldn't be surprised if there was.
(edit: Wow. Lots of people wanting to help! I wasn't expecting two sibling responses.)
Also, his website is very slow, probably because he's not using a CDN. A noble goal, but it has an impact on credibility. The slow website makes me feel like he doesn't care about user experience, which makes me assume that is true for his whole setup, and turns me off from even considering it.
They said they got 18,000 hits in one day. That's a tiny amount of traffic for any decent static website (which this one that we are talking about is). Even assuming they got all that traffic in one hour, that's only 5 requests per second.
Syncthing is a great continuous backup solution. I use ~/NOTES as a scratchpad, and it updates automatically between my various computers. It gives you pretty granular control over shares, and I back up critical stuff to a cloud provider.
That said, there's no calendar/email/notes. XigmaNAS is built on FreeBSD, and will happily run NextCloud or a photo gallery or whatever.
In the end you are still just as vulnerable getting booted off with VPSs like you are with google, but with domain control you can still switch hosts without losing your address, and you usually have customer support.
I suppose I should have another go at a blog.
* Put together an overbuilt NAS box running ZFS On Linux
* Simple docker-compose file for all services
* Backups through borgmatic (via ZFS snapshots)
* Auto-updates through watchtower
* Punted on email and use FastMail, switched to our own domain from gmail
Services we run include:
* PhotoPrism for semi-Google Photos functionality
* Nextcloud and Collabora for file sync, sharing
* Kodi for home media
* Tiddlywiki
* DDNS through Gandi since we're on a dynamic IP
* PiHole for some ad/privacy protection
* Robocert for SSL
* Nginx to reverse proxy everything
It wasn't _easy_ to set up, but in a year, any given week I typically spend 0 hours dealing with it. No problem that _has_ cropped up has taken more than a few minutes to fix, mostly around docker networking and auto-restarting containers after Watchtower auto-updates them, a problem I've since fixed.
This setup seems way easier than k3s or some other recommendations, doesn't require much new knowledge, and is as portable as I need it to be. If needed I could plop the docker-compose on a new machine, change some mount points, and largely be up and running again quickly. It's let us switch to "deGoogled" phones and unplug from almost every hosted service we used to use.
If your personal life is filled with productivity tools and optimizations, at what time in your daily life your are _not_ worried about productivity? If this time is zero, I think it's kind of sad and maybe even unhealthy. It's just my opinion, of course :)
I wonder if there's a viable business model for this. Automate the setup through scripts and process automation for any provider. You pay a one time fee + a reasonable amount for maintenance and for resilience built in. I would pay for it if the price is reasonable.
1. stand up a cluster. (two commands, install docker, install k3s) 2. apply yaml files (kubectl apply -f .)
I run 6 services/"apps" on 2 OVH servers ($6.70/mo)
That one put a smile on my face.