Rclone can also mount cloud storage to local disk, especially nice from kubernetes. Write/read speed isn't the fastest when using as a drive with lots of files in the same folder, but a quick and easy way to utilize cloud storage for projects
It can also e2e encrypt locations, so everything you put into the mounted drive is getting written encrypted to Dropbox folder /foo, for example. Nice, because Dropbox and other providers like s3 don't have native e2e support yet
All in all, Rclone is great! One of those tools that is good to have on hand, and solves so many usecases
It's also a trivial way to set up an ad-hoc ftp server which serves either local or cloud storage. e.g. my Pi4 runs an rclone ftp server which exposes my dropbox to the rest of my intranet, and a separate ftp server which my networked printer can save scans to. The same machine uses rclone to run automated backups of cloud storage for my household and a close friend. rclone is a godsend for my intranet.
Assuming the Dropbox is synchronized somewhere to your file system, the FTP server could serve that directory. Although I guess not everyone synchronizes their Dropbox locally.
> Although I guess not everyone synchronizes their Dropbox locally.
And i've yet to see a pi-native dropbox client for doing so.
PS: i actually do sync dropbox to my main workstation but selectively do so. The lion's share of my dropbox is stuff i don't need locally so don't bother to sync it. The rclone approach gives me easy access to the whole dropbox, when needed, from anywhere in my intranet, and i can use my local file manager instead of the dropbox web interface.
No configuration needed (beyond rclone's per-cloud-storage-account config (noting that serving local dirs this way does not require any cloud storage config)) and some variation of that can be added to crontab like:
Has anyone used it successfully as a replacement for two-way sync apps (like Insync for Google Drive)?. Insync sucks, and Google Drive sucks, but for something I depend on every day I feel like I really need to have a local copy of the files and immediate sync between local and server, particularly when Internet access is spotty.
You can still do this, but Dropbox can’t use the File Provider API for that yet, so the experience won’t be quite as integrated as it is with Dropbox for macOS on File Provider. See https://help.dropbox.com/installs/dropbox-for-macos-support for more.
FWIW, i've been using Insync on Linux since it went online (because Google never released a Linux-native client). Aside from one massive screw-up on their part about 8 or 10 years ago (where they automatically converted all of my 100+ gdocs-format files to MS office and deleted the originals), i've not had any issues with them. (In that one particular case the deleted gdocs were all in the trash bin, so could be recovered. Nothing was lost, it was just a huge pain in the butt.)
> Aside from one massive screw-up on their part about 8 or 10 years ago (where they automatically converted all of my 100+ gdocs-format files to MS office and deleted the originals)
Why. Just why. How does that shit ever happen in a public release?
But absolutely 100% remember to block automatic updates because even minor-minor version updates change the protocol (and even different versions of the ocaml compiler with the same version of unison source can mismatch.)
This pain has always stopped me using Unison whenever I give it another go (and it's been like this since, what, 2005? with no sign of them stabilising the protocol over major versions.)
I have bi-dir sync from/to two computers, one running Windows 10, the other Linux, using Google Drive as buffer. All data are encrypted client-side because I don't want Google to know my business. But the sync is every 30 min, not immediate. I have no fancy GUI over it, like synchthing etc. just plain CLI command triggered by cron and the task scheduled.
When you say “e2e” encryption do you mean client-side encryption? Because S3 supports both client and server side encryption. (It doesn’t really need to do anything on the service side to support client-side encryption tbf)
It is indeed great for this, but you need to make sure your network is stable.
I use it on my desktop and laptop to mount Google drives. The problem on the laptop is that the OS sees the drive as local, and Rclone doesn't timeout on network errors. So if you are not connected to wifi and an application tries to read/write to the drive, it will hang forever. This results in most of the UI locking up under XFCE for example, if you have a Thunar window open.
Thanks, but unfortunately this doesn't work - for my issues at least. I have this (and conntimeout) set to 15 seconds, but it makes no difference. I tried those based on another user reporting the same issue here:
The timeout param is listed as "If a transfer has started but then becomes idle for this long it is considered broken and disconnected". This seems to be only for file transfers in progress.
I traced it once, and Rclone gets a "temporary DNS failure" error once the network is down, but just keeps retrying.
I use rclone daily. It's replaced sshfs for some cases. Another is to push home server $Archive share to rsync.net. another is to pull photos into $Archive from my Google account, my wife's, moms, dads, and kids account. Also my 3 clound account for different $WORK into $Archive (and then to rsync.net).
This I'm not sure of. I also have a habit to use Takeout quarterly (but this blob is not merged). I think I should check these configs; I want the full fidelity.
Sadly, it’s not configurable. It’s just an inherent limitation of the API (1). Takeout is the best alternative, but for something more realtime you can use tools (2) that wrap the browser UI which also exports full quality.
Running headless chrome and scraping the photos via dev tools sounds like good way to get your account banned by some automation with no human support to help you :/
It's really stupid that the API doesn't support this. For now I'll stick to regular takeout archives. Syncthing directly from the phone might be a better option for regular backups.
So I see this type of headline happen a lot, and despite being a long time reader and short time commenter, is there a reason the author doesn't make a more specific headline? Rclone has existed forever and has been mentioned here over a thousand times (https://www.google.com/search?client=firefox-b-1-d&q=rclone+...)
Most of the time the poster links to a specific news item at least (which is also not okay without a specific headline) but sometimes the link just points to the home page.
Regardless, it has been mentioned before.
EDIT: Just to be clear, I'm not again an occasional mention of a project. Hacker news has allowed me to find some real gems, but most of those posts are new features added to the software, not just a generic link to the homepage.
Fair enough - as a user, that prompts me to ask myself, why am I using Hacker News for tech news when most of the content I see daily isn't actually news.
I guess I will just be more aggressive in hiding posts that I didn't want to waste my time reading.
What cloud storage provider officially supports this tool or the protocol used by it?
The instructions look mostly like a reverse engineering effort that can fail anytime for any reason. Of course you'll get some notification, but still seems a thing to avoid in principle.
It's a wrapper around a bunch of API's. If that makes you nervous then wait until you hear about how the rest of the modern web is strung together. It's been working reliably for me for a decade.
That's wonderful! Full endorsement of the support of rsync.net.
I sincerely doubt most tools will be supported by most providers (by usage), however, and I question the evaluation of tools by this metric. Support is generally a fairly bad metric by which to evaluate tooling—generally speaking, if you care which client is consuming an API someone has done something horrifically wrong.
What if we looked at all of the cloud providers APIs, which may or may not be similar to one another, may be in constant flux, and may have undocumented quirks, and hid all that complexity inside of one black box that provided a single clear API. The guts of that black box might be kinda ugly, and might need at lot of constant updating, but for a lot of users (including me), that is more appealing dealing with all the raw complexity of the various cloud providers "offical" APIs.
There is no rsync protocol, only whatever protocol exist already. Most providers will have their version of S3, so if a provider wants to implement a strict copy of the AWS api they will be compatible. They can also provide an SFTP API and rclone will work.
Every time I see this pop up on HN, I upvote. I've been using rclone for years for my main backup [1] (now over 18 TB) from my home NAS to Amazon S3 Glacier-backed Deep Archive, and it is by far the easiest backup setup I've ever had (and cheap—like $16/month).
The two work together well too - I prefer to let Restic back up to a separate local (magnetic) HDD for my repo, and then use rclone to sync the local repo to Backblaze B2. Compared to a direct cloud repo set up in Restic, it gives me much better restore times and lower costs for the common case where your local site is not completely destroyed (e.g. data corruption etc...), while still giving you the safety of a slower and more expensive off-site backup for when you really need it.
Egress costs are significant by the time you want to use your backup, around $1800 to download the 18TB, which is around 10 years of storage cost. If you assume 5 years of HDD data lifetime, you need to roughly triple that monthly cost, which is not too bad, but you can buy an enterprise 20TB drive per year for that cost. Personally I would only use AWS in a way that I would almost never have to pay for egress for the entire dataset except when my house gets burned, as a last-resort backup instead of a main one.
I have a primary and backup NAS at home, and a separate archive copy updated weekly offsite locally. Then Glacier is my "there was a nuclear bomb and my entire city is completely gone" solution.
And in that case, assuming I had survived, I would be willing to pay most anything to get my data back.
Nice, that's a great point and a good use case. I normally try to stay away from AWS (when I have to care about the cost) but I think you've found a case where it makes sense!
I've already got backblaze (behind rclone) set up for my backups so adding a glacier archive would be quite easy. Thanks!
Last time I priced it out, Backblaze was more expensive per GB — note that I'm using Glacier Deep Archive, which is something like an order of magnitude less expensive than plain Glacier-backed buckets.
It also incurs a delay of at least 6-12 hours before the first byte of retrieval occurs, when you need to restore. So there are tradeoffs for the price.
Oh, I see. I'm shamefully ignorant about everything concerning backups, so I just googled "Amazon S3 Glacier" and compared some numbers that pop up to some other providers.
> a delay of at least 6-12 hours before the first byte of retrieval occurs
Huh. I wonder how it actually works behind the curtains. Do hey actually use HDDs to store all of that, or maybe is there some physical work involved in these 12 hours to retrieve a tape-drive from the archive…
Pro tip is to create a Google Workspace org and pay the $18/mo per 5TB of Drive storage to also get no egress fees. With rclone, the process is fairly simple especially when using a service account to authenticate (so no oauth revocations or refreshing).
I used `rclone` in one startup to add the storage services the biz people were using, to engineering's easy&inexpensive offline "catastrophic" backups.
I used the normal ways of accessing AWS, GitLab, etc., but `rclone` made it easy to access the less-technical services.
I'm definitely a Rclone fan, it's an invaluable tool if less technical folks give you lots of data in a consumer / business cloud, and you need to somehow put that data onto a server to do further processing.
It's also great for direct cloud-to-cloud transfers if you have lots of data and a crappy home connection. Put clone on a server with good networking, run it under tmux, and your computer doesn't even have be on for the thing to run.
That way doesn't let you check on your progress, right? Screen / Tmux let you re-attach to the session at any time and see what's going on, if there were errors etc.
> That way doesn't let you check on your progress, right
It does if you redirect the output to a file other than /dev/null, but in my experience checking on the progress is irrelevant - it's done when it's done.
Just browsing the list of supported providers and such makes me a bit overwhelmed. There are so many tools that are kinda similar but slightly different, that from "wow, rclone seems like a great tool" I quickly come to "what should I even use?" Rclone, restic, borg, git-annex… Is there a case to use several of them together? Are there, like, good comprehensive real-life case studies to see how people use all that stuff and organize vast amounts of data (which are kinda prerequisite for anything like that to be used in the first place)?
Rclone is the general purpose "swiss army knife". It can be used to read/write/sync/copy/move files between any supported service and your local device, and also mount remote storage as a local drive.
Restic and Borg are for making backups only.
To help avoid getting anyone's hopes up: rclone does not do automatic two-way sync. In rclone parlance, "sync" means to either pull all files from a remote and make a local copy match that, or do the opposite: push a local dir to a remote and update the remote to match the local one. Note that "match" means "delete anything in the target which is not in the source."
New user to rclone and I am wondering if the setup I want can be done through rclone.
I want to have a local directory in my computer that bi-directionally syncs to both Google Drive and DropBox as well as to my Synology NAS. There will be two Google Drive accounts (one for me and one for my wife) and one Dropbox account. I also have two Linux workstations running Ubuntu 22.04 and two Macbooks where I want the local directory to exist.
Basically anytime we put anything in either the cloud drives or the NAS or the local directories in the respective computers, it should get synced to all the other destinations.
There could be other stuff in the cloud drives or the NAS that I would just have the tool ignore. The sync should just be for that one specific folder. I do not want to deal with separate local folders for each cloud storage solutions.
Is this feasible? I am okay to run cron tasks in any of the computers; preferably the Linux ones.
I use rclone and git-annex together (there's a git-annex special remote plugin for rclone). Rclone handles interfacing with the various cloud providers, while git-annex tracks where the files are.
I use Borg to backup my stuff to a local disk repository, which I then synchronize -- encrypted -- to several cloud accounts, and that is done by rclone.
I appreciate how its home page proudly calls out that it always verifies checksums. I've been boggled in the past how many tools skip that step. Sure, it slows things down, but when you're syncing between cloud storage options, you're probably in a situation where you really, really want to verify your checksums.
I thought the “indistinguishable from magic” testimonial at the top was a bit arrogant, but am currently trying to find out if these extraordinary claims have extraordinary evidence ;)
My Dad left us 30 TB of data when he passed. I trimmed it down to 2 TB and tried to use Google's desktop sync app to upload it to cloud. It ran on my work computer during COVID for 8 months straight before finishing.
When I tried to back up that and my other data to a hard drive, Google takeout consistently failed, downloading consistently failed. I went back and forth with support for months with no solution.
Finally I found rclone and was done in a couple days.
> Google takeout consistently failed, downloading consistently failed
I get extra angry about situations like this. I have paid Google many thousands of dollars over the years for their services. I just assume that one of the fundamentals of a company like that would be handling large files even if it’s for a fairly small percentage of customers. I get that their feature set is limited. I kind of get that they don’t provide much support.
But for the many compromises I make with a company that size, I feel like semi-competent engineering should be one of the few benefits I get besides low price and high availability.
In the past few years it has been a leech of brain power that's optimized itself into producing nothing of value except demos that get abandoned immediately. From what I read, they seem to have managers and executives with bad incentives and too much power. So it seems that it doesn't really matter how competent the engineer is, their work goes into numerous black holes in the end.
> We should also remember how a foolish and willful ignorance of the superpower of rewards caused Soviet communists to get their final result, as described by one employee: “They pretend to pay us, and we pretend to work.” Perhaps the most important rule in management is “Get the incentives right.”
-- Charlie Munger, Poor Charlie's Almanack ch. 11, The Psychology of Human Misjudgment (principle #1)
Devil's advocate: egress prices are always extremely high, so if you use their service that doesn't transfer the cost over to you it means it has been factored in that you won't be downloading that much. Making obstacles to actually doing it is, for them, the cost-effective way, and if you want to have an actual full access you're supposed to use their cloud storage.
Why would you expect that? It is not profitable to put extra work into serving a small proportion of customers. It is also not profitable to put work into helping customers stop using your services.
It is something providers do to keep regulators off their back. They are not going to put money into making it work well.
A feature that makes it easier to switch will also make it less risky to sign up in the first place.
Giving away a feature that is a competitor's cash cow could weaken that competitor's stranglehold on customers you want to connect with or drive the competitor out of business.
At a particular point in time, increasing market share could be more important for a company than making a profit.
It could be culturally uncommon to charge for some essential feature (say bank accounts), but without that feature you wouldn't be able to upsell customers on other features (credit cards, mortgages).
Ad funded companies give away features and entire services for free in order to be more attractive for ad customers.
Of course, if a feature is bad for a company in every conceivable direct and indirect way, even in combination with other features, now and in the future, under any and all circumstances, it would not introduce that feature at all.
Introducing a completely broken feature is unlikely to make much sense. Introducing lots of low quality features that make the product feel flaky but "full featured" could make some sense unfortunately.
> A feature that makes it easier to switch will also make it less risky to sign up in the first place.
Yes, if buyers think that far. Consumers do not. SOme businesses may, but its not a major consideration because the people making the decision will probably have moved on by the time a swtich is needed.
> Ad funded companies give away features and entire services for free in order to be more attractive for ad customers.
Yes, but that means those services do make a profit.
The same applies to the banking example but they can make money off free accounts as well.
> Introducing a completely broken feature is unlikely to make much sense. Introducing lots of low quality features that make the product feel flaky but "full featured" could make some sense unfortunately.
The latter is similar to what I am suggesting here. Being able to export data will probably satisfy most customers who want to do so, even if it does not actually work well for everyone. It will also mollify regulators if it works for most people. If they can say "data export works well for 95%" of our customers regulators are likely to conclude that that is sufficient not to impede competition.
They absolutely do. It's hard not to, because migrating data is the very first thing you have to think about when switching services. It's also a prominent part of FAQs and service documentation.
>Yes, but that means those services do make a profit.
Of course. What I said is that not every single feature can be profitable on its own. Obviously it has to be beneficial in some indirect way.
Inability to download from the cloud is happening to me every with every provider. Proton from the web, OneDrive Cryptomator vault through Cyberduck. I’ll have to use rclone, I guess
Rclone is a magic tool, I've used it for many different use cases.
When I last checked it doesn't use the AWS SDK (or the Go version is limited). Anyway, it isn't able to use all settings in .aws/config.
But it is kind of understandable that it doesn't support all backend features because it's a multifunctional tool.
Also the documentation is full of warnings of unmaintained features (like caching) and experimental features. Which is a fair warning but they don't specifically tell you the limitations.
I use restic for everything Linux or windows except for Mac’s. There I use kopia. It just works. There is not much to mess up really. Just keep your password safe and be sure to test your backups.
Like with all contingency systems it's necessary to do a "fire drill" exercise from time to time. Pretend that you lost your data and attempt to recover a few files from backups. Pretend that your house burned down and you can't use any item from home. This will give you confidence that your system is working.
The main disadvantage with pure Restic is that you usually have to end up writing your own shell scripts for some configuration management because Restic itself has none of that.
As for remote backups - I use ssh with Borg and it works fine. If this is a NAS you can probably enable ssh (if it is not enabled already).
BTW for my remote backups I do encrypt them but this is a choice the author of Borg left open.
There are other issues with Borg such the use of local timestamps (naive date format, no timezone) instead of a full ISO8601 string, and the lack of capacity to ask whether a backup is completed (which is a nightmare for monitoring) because the registry is locked during a backup and you cannot query it.
I have used rclone for several years now and it works so good. I use it in place of rsync often too. It’s great for file moving on local file systems too.
I have been using rclone for over two years now. Typically, I run cron to copy any new files in the directories that are most important to me.
The combination of rclone and B2 storage is quite effective, especially considering its ease of use and cost efficiency.
173 comments
[ 4.7 ms ] story [ 209 ms ] threadIt can also e2e encrypt locations, so everything you put into the mounted drive is getting written encrypted to Dropbox folder /foo, for example. Nice, because Dropbox and other providers like s3 don't have native e2e support yet
All in all, Rclone is great! One of those tools that is good to have on hand, and solves so many usecases
Thank you
And i've yet to see a pi-native dropbox client for doing so.
PS: i actually do sync dropbox to my main workstation but selectively do so. The lion's share of my dropbox is stuff i don't need locally so don't bother to sync it. The rclone approach gives me easy access to the whole dropbox, when needed, from anywhere in my intranet, and i can use my local file manager instead of the dropbox web interface.
- startup only when the network is up
- proper logging
- automatic restarts on failure
- optional protection for your ssh keys and other data if there's a breach (refer to `systemd-analyze security`)
Run:
this opens a text editor; paste these lines: and then enable and start the service:systemd is far from perfect, and Poettering is radically anti-user. But it's the best we got and it serves us well
What does that mean?
It's not immediately apparent what this means—does it use FUSE, 9p, a driver, or some other mechanism to convert FS calls into API calls?
EDIT: it's FUSE.
FWIW, i've been using Insync on Linux since it went online (because Google never released a Linux-native client). Aside from one massive screw-up on their part about 8 or 10 years ago (where they automatically converted all of my 100+ gdocs-format files to MS office and deleted the originals), i've not had any issues with them. (In that one particular case the deleted gdocs were all in the trash bin, so could be recovered. Nothing was lost, it was just a huge pain in the butt.)
Why. Just why. How does that shit ever happen in a public release?
I've been using it to great effect for over 10 years on a daily basis.
This pain has always stopped me using Unison whenever I give it another go (and it's been like this since, what, 2005? with no sign of them stabilising the protocol over major versions.)
For client side encryption they have a whole encrypted S3 client and everything. (https://docs.aws.amazon.com/amazon-s3-encryption-client/late...)
I use it on my desktop and laptop to mount Google drives. The problem on the laptop is that the OS sees the drive as local, and Rclone doesn't timeout on network errors. So if you are not connected to wifi and an application tries to read/write to the drive, it will hang forever. This results in most of the UI locking up under XFCE for example, if you have a Thunar window open.
I shorten it to prevent lockups like you are describing.
https://forum.rclone.org/t/how-to-get-rclone-mount-to-issue-...
The timeout param is listed as "If a transfer has started but then becomes idle for this long it is considered broken and disconnected". This seems to be only for file transfers in progress.
I traced it once, and Rclone gets a "temporary DNS failure" error once the network is down, but just keeps retrying.
This is top-tier tooling right here.
I assume you are pulling from Google photos? If so, then I think the only way to get original quality is to use takeout?
(1) https://github.com/gilesknap/gphotos-sync#warning-google-api...
(2) https://github.com/JakeWharton/docker-gphotos-sync
It's really stupid that the API doesn't support this. For now I'll stick to regular takeout archives. Syncthing directly from the phone might be a better option for regular backups.
I'd been using sshfs for some years until I learned that rclone can mount remotes to the file system, and I've been using that happily since then.
https://rclone.org/commands/rclone_mount/
> at present SSHFS does not have any active, regular contributors, and there are a number of known issues
https://github.com/libfuse/sshfs#development-status
Most of the time the poster links to a specific news item at least (which is also not okay without a specific headline) but sometimes the link just points to the home page.
Regardless, it has been mentioned before.
EDIT: Just to be clear, I'm not again an occasional mention of a project. Hacker news has allowed me to find some real gems, but most of those posts are new features added to the software, not just a generic link to the homepage.
https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
Like you mentioned, some real gems are found, and for me are often found in the comments. So, these reposts are helpful.
It’s like how many posts were made for “What do you use for note taking?” Some gems are are in the comments.
This is discussed in the site docs, it's fine to get news from HN but HN is not really about news.
I guess I will just be more aggressive in hiding posts that I didn't want to waste my time reading.
The instructions look mostly like a reverse engineering effort that can fail anytime for any reason. Of course you'll get some notification, but still seems a thing to avoid in principle.
I’ve been using rclone for 3 years without issues. Across many different backends.
Man I wish I lived in a universe where providers actually supported tooling. You're completely delusional tho.
I sincerely doubt most tools will be supported by most providers (by usage), however, and I question the evaluation of tools by this metric. Support is generally a fairly bad metric by which to evaluate tooling—generally speaking, if you care which client is consuming an API someone has done something horrifically wrong.
https://www.rsync.net/resources/howto/rclone.html
[1] https://github.com/geerlingguy/my-backup-plan
I have a primary and backup NAS at home, and a separate archive copy updated weekly offsite locally. Then Glacier is my "there was a nuclear bomb and my entire city is completely gone" solution.
And in that case, assuming I had survived, I would be willing to pay most anything to get my data back.
I've already got backblaze (behind rclone) set up for my backups so adding a glacier archive would be quite easy. Thanks!
It also incurs a delay of at least 6-12 hours before the first byte of retrieval occurs, when you need to restore. So there are tradeoffs for the price.
> a delay of at least 6-12 hours before the first byte of retrieval occurs
Huh. I wonder how it actually works behind the curtains. Do hey actually use HDDs to store all of that, or maybe is there some physical work involved in these 12 hours to retrieve a tape-drive from the archive…
There's a ticket covering everything you might ever want to know:
https://github.com/rclone/rclone/issues/1778
Seems like rclone would support it if Apple ever created an official way to do so.
I used the normal ways of accessing AWS, GitLab, etc., but `rclone` made it easy to access the less-technical services.
[1]: https://rclone.org/overview/
[2]: https://rclone.org/crypt/
It's also great for direct cloud-to-cloud transfers if you have lots of data and a crappy home connection. Put clone on a server with good networking, run it under tmux, and your computer doesn't even have be on for the thing to run.
tmux isn't strictly necessary:
then log out and it'll keep running.It does if you redirect the output to a file other than /dev/null, but in my experience checking on the progress is irrelevant - it's done when it's done.
To help avoid getting anyone's hopes up: rclone does not do automatic two-way sync. In rclone parlance, "sync" means to either pull all files from a remote and make a local copy match that, or do the opposite: push a local dir to a remote and update the remote to match the local one. Note that "match" means "delete anything in the target which is not in the source."
Oooh, nice. That's not _quite_ the same as real-time two-way sync, but i guess it's the next best thing.
<https://rclone.org/bisync/>
I want to have a local directory in my computer that bi-directionally syncs to both Google Drive and DropBox as well as to my Synology NAS. There will be two Google Drive accounts (one for me and one for my wife) and one Dropbox account. I also have two Linux workstations running Ubuntu 22.04 and two Macbooks where I want the local directory to exist.
Basically anytime we put anything in either the cloud drives or the NAS or the local directories in the respective computers, it should get synced to all the other destinations.
There could be other stuff in the cloud drives or the NAS that I would just have the tool ignore. The sync should just be for that one specific folder. I do not want to deal with separate local folders for each cloud storage solutions.
Is this feasible? I am okay to run cron tasks in any of the computers; preferably the Linux ones.
When I tried to back up that and my other data to a hard drive, Google takeout consistently failed, downloading consistently failed. I went back and forth with support for months with no solution.
Finally I found rclone and was done in a couple days.
I get extra angry about situations like this. I have paid Google many thousands of dollars over the years for their services. I just assume that one of the fundamentals of a company like that would be handling large files even if it’s for a fairly small percentage of customers. I get that their feature set is limited. I kind of get that they don’t provide much support.
But for the many compromises I make with a company that size, I feel like semi-competent engineering should be one of the few benefits I get besides low price and high availability.
-- Charlie Munger, Poor Charlie's Almanack ch. 11, The Psychology of Human Misjudgment (principle #1)
But that's only one possible reasoning.
It is something providers do to keep regulators off their back. They are not going to put money into making it work well.
Giving away a feature that is a competitor's cash cow could weaken that competitor's stranglehold on customers you want to connect with or drive the competitor out of business.
At a particular point in time, increasing market share could be more important for a company than making a profit.
It could be culturally uncommon to charge for some essential feature (say bank accounts), but without that feature you wouldn't be able to upsell customers on other features (credit cards, mortgages).
Ad funded companies give away features and entire services for free in order to be more attractive for ad customers.
Of course, if a feature is bad for a company in every conceivable direct and indirect way, even in combination with other features, now and in the future, under any and all circumstances, it would not introduce that feature at all.
Introducing a completely broken feature is unlikely to make much sense. Introducing lots of low quality features that make the product feel flaky but "full featured" could make some sense unfortunately.
Yes, if buyers think that far. Consumers do not. SOme businesses may, but its not a major consideration because the people making the decision will probably have moved on by the time a swtich is needed.
> Ad funded companies give away features and entire services for free in order to be more attractive for ad customers.
Yes, but that means those services do make a profit.
The same applies to the banking example but they can make money off free accounts as well.
> Introducing a completely broken feature is unlikely to make much sense. Introducing lots of low quality features that make the product feel flaky but "full featured" could make some sense unfortunately.
The latter is similar to what I am suggesting here. Being able to export data will probably satisfy most customers who want to do so, even if it does not actually work well for everyone. It will also mollify regulators if it works for most people. If they can say "data export works well for 95%" of our customers regulators are likely to conclude that that is sufficient not to impede competition.
They absolutely do. It's hard not to, because migrating data is the very first thing you have to think about when switching services. It's also a prominent part of FAQs and service documentation.
>Yes, but that means those services do make a profit.
Of course. What I said is that not every single feature can be profitable on its own. Obviously it has to be beneficial in some indirect way.
When I last checked it doesn't use the AWS SDK (or the Go version is limited). Anyway, it isn't able to use all settings in .aws/config.
But it is kind of understandable that it doesn't support all backend features because it's a multifunctional tool.
Also the documentation is full of warnings of unmaintained features (like caching) and experimental features. Which is a fair warning but they don't specifically tell you the limitations.
https://www.bobek.cz/blog/2020/restic-rclone/
The rclone backend means I can backup anywhere.
https://github.com/gilbertchen/duplicacy
Duplicacy is slower and also only free for personal use.
Fortunately there is https://github.com/creativeprojects/resticprofile to solve that problem.
I hate such an approach when someone assumes that whatever happens their opinion is the best one in the world and everyone else is wrong.
I do not want to encrypt my backups because I know what I am doing and I have very, very good reasons for that.
Restic could allow a --do-not-encrypt switch and backup by default.
The arrogance of the devs is astonishing and this si why I will not use Restic and i regret it very much because this is a very good backup solution.
Try Borg.
I'm using duplicity because none of these can sync to a rsync server that I run on my NAS.
As for remote backups - I use ssh with Borg and it works fine. If this is a NAS you can probably enable ssh (if it is not enabled already).
BTW for my remote backups I do encrypt them but this is a choice the author of Borg left open.
There are other issues with Borg such the use of local timestamps (naive date format, no timezone) instead of a full ISO8601 string, and the lack of capacity to ask whether a backup is completed (which is a nightmare for monitoring) because the registry is locked during a backup and you cannot query it.
Restic for various machines to hourly backup on a local server. Rclone to sync those backups to S3 daily.
Keeps the read/write ops lower, restores faster, and limits to a single machine with S3 access.
Most cloud space providers don't show you how much space each folder and subfolder actually occupies. Enter rclone ncdu.
https://kapitainsky.github.io/RcloneBrowser/
https://github.com/rclone/rclone-webui-react