On of my favorite program, and it keeps getting better.
You can even set an encryption key per share, and if you want to use an untrusted device in your nodes, you can simply not provide it with the encryption key and it will simply synchronize the encrypted payload across nodes.
So if you have a shady/cheapo VPS, you can use its storage without worrying about the plaintext data being stored there.
Syncthing is pretty awesome. It's versatile, I use it to sync photos from my phone to more stable storage but also on webservers to keep files in sync across a couple boxes.
TL;DR: give syncthing a try. first time setup is odd, then bam, your files magically sync anywhere with internet.
I carried around my 1GB Data Traveler flash drive with great conviction. Then, a massive 4G SanDisk drive I would 'never fill up'.
Then, when my 16GB Nexus 4 filled up, one day while I was MTPing my old pictures to my computer, someone told me about Dropbox and how it would sync my pictures for me. After living out of Dropbox for all my non-cold-storage files...I hit the free limit. But I was a student and didn't have money to pay per month.
Skydrive gave me 25 GB. And 3 successive phones ate that like candy. So start paying up Mister. ugh.
But then, Syncthing came along here on HN. I installed it onto my phone and pc for camera download. then another, and another, and a separate folder for sharing files with a colleague, then my work SAN got laggy and lost files, so syncthing to keep my work files local and synced between work devices to the rescue.
And now, I don't pay for storage. or I do, but it just happens to come with word 365, but I don't bother using it.
And the flash drives? I'm up to a 256GB Sandisk Pro of some kind, but only for 'just in case' scenarios. Syncthing handles all of my LAN files access needs.
I used to use Dropbox as well, but it started freaking out when I had folders with over 500,000 files in them. I started writing scripts to zip up folders first, moved some things off Dropbox, but it would just take up several CPU cores when going through everything. I tried Syncthing, and it had zero issues with the number of files I had. Everything just worked. I was amazed that the free solution was so much better than the commercial solution for which I was paying $10+ each month.
Over long enough timeline, free software almost always gets better, acheiving parity and eventually surpassing the proprietory service. Free software doesn't need to be reinvented and can keep on steadily fixing issues. It happened for gcc, Linux, GNU, Blender, OBS, VLC, Calibre and obviously Syncthing. As long as scope is clear and target is not moving too much, FLOSS will catch it.
Syncthing is a great tool : I've been using it for years for backups, picture synchronization between phone and computer, and as a way to share a KeePass DB.
In all those years, not a single issue arise : you connect a device, decide what to share or not, and then just forget about it
I love Syncthing. I've found some unconventional uses for it in syncing program settings. My fonts folder is now synced between my Mac, Windows and Linux computers and that works surprisingly well. My projects folder does a one-way archive (like rsync) for backup to my home NAS and an offsite VPS. I have a Streamdeck that I use between my Mac and Windows work machines, and I sync the settings on that so it operates the same way regardless of which machine I plug it into. I love it.
One of my unconventional uses is to sync my HomeAssistant configuration to my desktop, where I have it in a git repository. I can edit manually from my desktop, and/or manage from the HomeAssistant UI, and either way, commit discrete, working changes to git once I'm ready.
This. I'd like to sync my ~/development directory, including all the git directories, so I don't need to push and pull branches between machines. Is this at all feasible?
I've read of other HN'ers doing this successfully (syncing git repos between machines) but I just seem to get a lot of issues - git repos often broken temporarily, conflicts etc.
Don't know how Syncthing works, of it it would even work, but what if Syncthing would detect that it recurses into a .git folder, and then use git push to push all refs to the remote .git repo? That should give a transparent "sync" of the entire git repo and workdir for the user, but via the safety of git's own sync mechanism.
It works fine. I've been doing it for ages with no problems. Note that I added node_modules, etc. to the ignore file so that it's not always syncing thousands of tiny files, which always tend to take a long time for some reason.
Yes exactly that, plus it ignores .log, .db and core.restore_state.
I should note I specifically use this for doing source control of configuration and making it easy to manually edit -- I have an entirely separate process for backups of the server itself.
Like you, I've also gone through several iterations but this has been working for me for almost a year now.
Cool, thanks for elaborating. I may give this approach another try.
That said, I’m making far fewer YAML changes these days than I used to. I’m doing almost everything via their UI so maybe I need to consider how much I actually need source control now vs just having solid backups.
Originally I was against their move from YAML config to the more user friendly GUI approach but I have to admit it’s a lot easier to make quick on-the-fly changes to my automations these days. I’m begrudgingly coming around to the idea of not firing up VS Code every time I want to tweak something!
My NAS does btrfs snapshotting of the Synchthing folder.
So I have an up-to-date backup of my phone pictures, laptop files and can go back in time if I accidentally delete something.
Does it sync photos from the iOS camera roll? I assume not since the screenshots in App Store don’t look like it. But if it does, and keeps the albums, then it would be useful to me. Currently I transfer my photos from my phone to an external drive using the iMazing app on macOS with the phone connected.
Also worth mentioning if it makes a difference, I only need to be able to transfer photos off of the phone (along with info about which albums they belong to as mentioned), and don't need to sync in the other direction onto the phone.
In the end this might be the most simple and workable solution. Still feels kind of bad having to roundtrip all the data through the cloud when trying to get it from one local device to another.
I am paying about EUR 20 / month for my somewhat paranoid backup system that includes Apple, Dropbox, Synology and Backblaze (and excludes depreciation). Worthwhile all things considering, but if Möbius Sync could sync the iOS Photo Stream ("further down the roadmap") I'd pay EUR 100 or more for that. It would make the backups in my house a lot more manageable.
Muse Group could have taken a lesson from this with their (attempted?) addition of telemetry to Audacity. If the community feels a sense of ownership of the data that's being collected from said community, you'll probably receive less push back.
Recently, there had been an article in a prominent German IT magazin about using Synthing to create backups from any type of machine on a Raspberry Pi. So last week I tried it out and quickly run into a problem that could have been dangerous. What I tried out was to backup my Firefox and Thunderbird profils with synthing. I switched on simple file versioning to be on the save side even in case that something is accidentally deleted. The problem was that, probably due to interrupted synchronisation sessions, Synthing could not decide whether the file version on the orginal or on the backup machine is the most recent one. In this case it transfered an old file version with some additional intermediate filename extension from the backup to the original machine. So I ended up having doublicates of my email inbox and other folders in my Thunderbird profil that Thunderbird nevertheless recognized as valid inboxes, etc. The problem was easily solved by searching for the intermediate filename extension and deleting all those files manually. This experience suggests to me that it is perhaps not a good idea to use Synthing for anything else than the deliberate exchange of manually managed files in folders only and specifically dedicated to synchronisation.
In the old days the Syncthing on Android had a pretty clumsy UI; Fork fixed that. But when I evaluated this in June 2021 the main app had improved enough the fork didn't seem necessary any more.
Better handling of sync rules. For example syncthing-fork can have one folder set to sync over mobile data and others to wifi only. Stock client is all or none.
Syncthing is great, I've used it for years to sync all my stuff and it has always been solid. I even use it on my kindle fire tablet (there's a Syncthing app in F-droid).
My only annoyance with Syncthing is when I reinstall an OS on one of my machines. Let's say I'm syncing files between my main workstation, two laptops, a phone, and a cloud backup. Now I want to reinstall the OS on one of the laptops. When I install syncthing on that machine, it gets a new ID. I can make it join the sync swarm but all the other members will think it's a new machine and won't trust it so I have to go to each of the other machines and manually remove the "old" laptop and bring in the "new" one.
I keep a backup of Syncthing configs and then re-use it. Config.xml can even be manually edited to fix directory paths.
~/.config/syncthing in Linux, the pem files and config.xml. Similarly in Windows too (from C:\Users\<username>\AppData\Local\Syncthing).
Even in Android, you can export and back-up. When you are switching to new device, copy from the back-up and then import.
Does it support syncing local folders? For example a game put saves in Documents folder and I would to sync it to my Dropbox folder, is such thing possible?
Thanks for pointers. I looked how to create a junction and I found this small app: https://schinagl.priv.at/nt/hardlinkshellext/linkshellextens..., it's a GUI for creating symbolic links/junctions/etc. Which for me is great as I'm not an avid user of cli.
I've been using Syncthing for years now, between ~10 Linux servers, Linux desktops, Linux and Windows laptops and Android smartphones, over LAN, wired and wireless Internet.
I'm synchronizing less than 100 MiB of data, but it's changing all the time. I've not yet had a single issue with it. I've a few conflicts every week when the same file is changed in different locations at the same time, but Syncthing keeps all versions of a file until I resolve the conflicts; I can't see how it could handle that better.
You may have to manually step through the conflicts menu to resolve them, choosing which device's version to keep. But typically that won't happen unless both device's versions of a file change at the same time.
I have been using syncthing for at least 3 years now. Sync’ing a 7gb directory across 6 devices. I rarely get conflicts, when it happens i just click on the conflicts link via the browser and check which files are conflicting and try to resolve it. Sometimes i have to call a REST action to reset the local db:
curl -X POST -H "X-API-Key: xxx" "http://localhost:8384/rest/system/reset?folder=YYYY"
This is my last resort cause sometimes syncthing gets confused.
Also good to note my setup has 1 introducer which is my NAS (synology) that is 24/7 online which always has the latest changes. Other devices are mostly mobile or turned on at irregular times. For devices without syncthing support like iOS, i rely on exposing the directory on the NAS via webdav. This way i can still access my data on the go.
And my synology creates regular backups/snapshots of my synced directory. So that i can always recover files. Even outside of syncthing.
Syncthing is one of the "fire and forget" kind of programs. It takes some time to setup when doing for first time, but then you pretty much forget its there, and when you need it, you realise it has been chugging along doing its job.
Two things I like about it in particular, are
1. It can and favors syncing over local network. This has a huge cost savings in developing world. Even when you do have massive bandwidth, local network sync still has more throughput and lower latency. My music is shared between devices and if I add a track, it takes less than a second to appear on my other devices.
2. You can set conditions, such that delets can be ignored. Eg. I have a WhatsApp message+media backup going back almost 7 years now, ~65GiB. But I don't need all that on my phone. So I just disable syncing delets on my storage. Now my phone can get away with ~3 GiB of whatsapp data (only because I don't delete very often) and I still have complete backup in case I need it.
3. Absolute rock solid stable. At $DAYJOB we use syncthing to sync multi dozen GiBs of new data every day, total file count in 10s of thousands in about ~300 directories. And in past 5 years we have had about 3 instances where we had something that needed attention, out of which 2 were not syncthing's fault.
I've been using Syncthing for years and have absolutely zero complaints against it. Everyone with 2 or more devices should give it a try, just to see there are better options out there.
> 2. You can set conditions, such that delet[e]s can be ignored.
I was evaluating Syncthing for 1-way, append-only phone photo backup, and my Google searches warned me away from this, talking about "unsupported" and "database corruption."
I assume you have not experienced this, but I guess I should have just read the docs and trusted them, instead of forum posts.
I have been using that feature for, I don't even remember how long, but definitely since it was marked beta or something. I am not a heavyweight like some other in this thread, but my current local state is 70GB and ~30k files, global state is ~50GB (I need to prune WhatsApp on phone again) and ~10k files. I am no biggie, but its not a slouch to keep all this working in 7 devices, with overlapping directories shared between them, across 2 timezones, over local and public network. Zero complaints is exactly my experience. Syncthing Just Works™
I agree. And that's my gripe with Dropbox, Drive, OneDrive (the big 3 that I was able to try, no apple devices). Every one of them has shit the bed on me, on more than one occasion. That's after burning my bandwidth and taking away my privacy. Syncthing just wins out every way (as long as problem scope is file sync, not cloud storage or sharing).
That scared me too. I just move what I want to delete from the original sync folder over to another folder, causing the phone data to be deleted, I even wrote a script that "partitions" that old data, also syncing to some random server after being fully encrypted (that server ofc doesn't have decryption keys).
i had some of these problems with earlier versions. removing the database (after making sure everything is on sync) usually solved the issue. but that's years ago.
>I was evaluating Syncthing for 1-way, append-only phone photo backup, and my Google searches warned me away from this, talking about "unsupported" and "database corruption."
Same here. What do you use instead? I haven't been able to find a good open-source multiplatform project for backups/1-way sync.
It's not open-source, but I ended up using PhotoSync for myself and my wife to a locked-down samba share on my LAN .
They had a pretty reasonable (~$6) in-app purchase to unlock samba access, and the fact that it's a German GmbH means I'm slightly more confident they're not trying to copy all our photos to a dark data-vault.
Happy user so far, though it took some cross-checking against the samba share feel confident it was backing up reliably (I'm just particular about trust-but-verify :)
I have several folders in SyncThing set for one-way only sync (from one device or machine to another only, never syncing back). It's easy to set various nifty folder options in the SyncThing web UI and on the Android app. (Dunno about Apple devices, because I own none.)
There are a few applications that can read WhatsApp database. I used one couple years ago to run some analytics on my chat, but don't remember now. I'm fairly certain it is still around and works, though.
Tangent but: I sometimes feel like programs that just work like this don't get traction because they don't provoke as much discussion, search volume, etc. A complex mess that needs to be constantly babysat creates a cottage industry around it and gets discussed a lot, but something that works is like the quiet kid who sits in the corner and gets A's and that everyone ignores
Kubernetes is today's poster child for this. The Hashicorp stack can do just about all the same things, but it just works so there's no cottage industry of consultants and no market for as-a-service implementations. Why promote something like that?
I can come up with a couple issues related to Syncthing (whether or not it's about Syncthing per se).
1) I like to minimize the number of APT repositories I use. Debian's and Ubuntu's are well behind on Syncthing versions. Every so often (much more rarely than before), they have breaking changes. As such, my phone can no longer sync with my computers. Perhaps the answer is that Debian should just not be packaging it, because they have such a conservative policy. I'm considering downgrading my phone's version, or building the latest version from source. Worst case I may add their APT repo but I'm not eager.
2) At least with the version I've been using, opting for local-only sync isn't so straightforward. It's per-instance, not per-folder. This means that I have to have my home server running it as local-only, even for small things that I wouldn't mind getting synced over the wire. It means that if I want to have a local-only directory shared between my phone and laptop, it has to go through my home server first. Unless there's a better way I don't know about.
Otherwise Syncthing has been pretty great. I like your no-delete thing, I did not consider that. I could definitely use it for photos.
You can always setup Nix/Guix on your Debian. Granted, its still one extra repository, per se, but it will likely be the only extra repository you'll have to add.
>2. You can set conditions, such that delets can be ignored. Eg. I have a WhatsApp message+media backup going back almost 7 years now, ~65GiB. But I don't need all that on my phone. So I just disable syncing delets on my storage. Now my phone can get away with ~3 GiB of whatsapp data (only because I don't delete very often) and I still have complete backup in case I need it.
I would like to know, how are you using Syncthing to create local backups for WhatsApp database/msgstore (crypt14) and media files. You also mention 65GiB as your complete backup ─ have you tested it via a restore? If so, how?
> I would like to know, how are you using Syncthing to create local backups for WhatsApp database/msgstore (crypt14) and media files.
Not the parent poster, but I also do this; I just configured Syncthing to share the /storage/emulated/0/WhatsApp (aka /sdcard/WhatsApp) directory. WhatsApp stores its daily local backup on the Backups subdirectory of that directory, and the media files are all in the Media subdirectory of that directory.
> have you tested it via a restore? If so, how?
I have actually used it to move all the WhatsApp and Signal data from an old phone to a new phone. Just have Syncthing synchronize these directories (with the same path) on the new phone before installing WhatsApp and Signal, then install and launch WhatsApp and Signal. When first launched, if that directory already contains a backup, both WhatsApp and Signal ask if you want to restore from that backup. Signal then asks you to type a long backup encryption key you should have written down somewhere, while WhatsApp asks its servers for the backup encryption key.
But to add, the message store the where the texts are stored. I usually keep it all. WhatsApp doesn't care about the media more than you do, so you can delete anything not needed, and now you have full text message history + important media on your new phone.
I have tested it while migrating across 3 devices and half dozen phone factory restores, so I'm fairly confident it still works. There are very ugly restrictions coming in future Android version with scoped storage and whatnot and I'm worried Google is finally going to cut me out. But until then, I keep mine.
I.. don't. My days of tinkering directly with WA db are over. These days I just want an easy and reliable backup. After restoring, WhatsApp recognizes the files and takes care of the rest.
What are the “3 instances where we had something that needed attention, out of which 2 were not syncthing's fault”? I’m curious what rare exceptions there are to watch for.
Unfortunately I don't clearly remember because last one was a while ago, but it was Sync conflicts due to external factors. The one where it was Syncthing's fault was years ago and is very likely fixed now.
I use syncthing to keep keepass and my private keys in sync across my devices, it's great for that works perfect with easy setup and its cross platfrom.
I tried to use it to sync SASS and webpack between developers and it didn't work, it seems to have an issue with node_modules has too many files for it to deal with and it thinks the directory is hundreds of gb and just stalls with a million years eta.
The versioning is cool too but I found it a bit flakey.
If you need reliable versioning and syncing of lots of files use GIT, if you need easy cross platform syncing syncthing is the way to go.
My rule of thumb is that anything in gitignore should be in syncthing’s list to ignore. There is some work on both sides when doing simultaneous development, but it keeps local files local.
I syncthing all devices/systems onto staging server and just rclone it via encrypt target onto S3-compatible remotes. It is about 4TB. After the initial "long" backup, the hourly runs take minutes
SD cards work on Android 11, although I think it's because of the "scooped storage" change Google made, not because Syncthing fixed it. It works as if it was internal storage.
On Android 10 and older, Syncthing can only "read" form SD cards (read/write works if you're rooted).
So to be clear, in /old/ versions of Android before they started making apps get permissions per directory, Syncthing can write to an arbitrary SD card folder just fine.
In the newest versions of Android you give it Scoped Storage.
In the intermediate, if you want 2-way sync on your phone, your data has to live in sdcard/android/media/com.nutomic.syncthingandroid/<sync folder1 >, <sync folder 2> which it does not need root to write to.
But for just grabbing your pictures, Android 10 has the stupid limitations on external file system access that Syncthing has to abide by or root over.
In 2014 I tried to reduce my "cloud" usage and went from Dropbox → BitTorrent Sync (now Resilio Sync) → Syncthing. I have it running on all computers, tablets[0], and phones[0] I have/manage. In total, it keeps ~70K files and ~9TB in sync across devices running Android, macOS, Windows and Linux/Ubuntu/Raspberry Pi OS.
I don't remember when was the last time I had to use a flash drive or a cable to move files around.
It could be due to the time between filesystem checks, which can't be set too low to avoid excessive load. It is set by default to 10 seconds however, so there's room for improvements. The option should be called "fs watcher delay".
Anyone using Syncthing to sync SSH config/keys and (eg) bash/shell profiles between machines?
I’ve gone through various attempts at keeping things in sync across a couple of Macs but always seem to hit issues.. (the issues not being specific to Syncthing)
I don't care much for cloud providers adding unnecessary restrictions, so I manage my own sshd_config on VMs. I'd like to rent a flat, not a hotel room, thank you very much.
> Git hosting platform
It appears GitLab, one of my platforms of choice, does support this in self-hosted setups[1]. Even if they didn't, it'd be trivial to extend their sshd_config by myself anyway.
It is true that you are free to change the config on a VM manually. I included it because it's the only other service I could think of where you might set up SSH authentication.
This is interesting about GitLab, thanks for the pointer. As you mentioned it only appears to be configurable instance-wide, there is no way for a user or organization to set it up for themselves (for example for your gitlab.com organization). That makes it a bad replacement for SyncThing'd private keys as moviuro suggested.
The one thing I miss would be a somewhat more user friendly way to have file versioning along with a UI (doesn't have to be terribly user-friendly, my baseline is dropbox's interface)
I've been thinking of buying a home NAS and running Syncthing as a replacement for Dropbox/Google Drive. Does anyone have any recommendations? Synology comes up a lot, I might also just use a Raspberry Pi.
Currently running Syncthing on an RPi 4 NAS, in a docker container. Works great between RPi, laptop and desktop.
Haven't tried it while out-and-about, though that's not really my use case.
*RPi4 NAS has one external thumb drive for flash storage and two spinning rust drives, one for storage, one for backup; all connected via USB. Occasionally I have to plug-unplug the spinning rust drives if the system reboots, as the drives go to sleep.
On Android it works acceptably. On iOS (at least iPad) you have the proprietary Möbius sync port which is a bit annoying in that it requires a regular push notification to perform any syncing, but with iOS 15 focus modes you can hide it without breaking the app. Only a few months left if you don't want to run the betas.
I've been doing this with an rpi+ usb3-attached ssd for years (kept upgrading rpis). Syncing across my laptop, phone and server (server backing it up nightly). Worked well for years.
Now moved to a more powerful machine, but the concept is still the same.
This setup allowed me to conveniently stop using google drive.
Not sure about mac mini, but some old machines might draw considerably more power than rpi. Rpi has a problem with sd cards, but you can make it use them in the read-only mode, in which case rpi becomes a very reliable computing device.
Has anyone here switched between (from/to) ownCloud/NextCloud and Syncthing? I'm currently using ownCloud on a BananaPro and I'm really not happy with the performance, but also with some bugs that appear to be a result of this low performance. I have considered Syncthing before but I wonder if anyone else has made the same transition. I'm mostly syncing Windows desktops.
Syncthing is a lot narrower in scope than owncloud. It just syncs. If you're using owncloud just as a dropbox replacement, the one thing you're likely to run into is syncthing does nothing to facilitate sharing with untrusted devices, like giving a share link to your friend or something.
Thanks. I mostly use it for synchronizing my various machines and as a backup. Sharing with other people is absolutely secondary and something I can do without.
I've been using Syncthing for the past few days and it has been working flawlessly. One thing I haven't tried is syncing between Windows and UNIXes (macOS/Linux) - does anyone have experience around this? How do permissions work?
Either perfectly or not at all depending on what you want to do. You get file contents, modification time, mode bits on *nix’s, and the read only bit on Windows.
So for you-to-you it’s seamless but if you want to sync shared-user directories then you lose most info.
I have Windows, Android, and Linux devices in my shares, I've yet to encounter issues.
Someone indicated they did have issues if you do a case only rename between a case sensitive (Linux) system and a case insensitive (Windows/Mac) one. I've never done this myself, but I guess that's one to run into. Also the usual caveats with any program, like that you won't be able to sync con.py to a Windows system, regardless of which program you use.
Syncthing can be a little confusing if you are coming from something like Dropbox or WebDAV. It’s not really quite like anything else I’ve used.
And with that having been said, it’s great. It’s maybe more powerful than you’d expect. My NAS for example has a one-way synced folder from a different box for backup, while having a default shared folder with all of my PCs. It’s also great because not all machines have to always be online; it can gracefully handle deferring stuff. For me, since my NAS is connected all the time, I can use it a bit like a central service, syncing between machines that have no overlap in uptime. Simultaneously, if the NAS needs to be down, it doesn’t interrupt syncing between other nodes. And unlike centralized solutions like Dropbox, I’m not limited by my internet connection, since it’s all local... unless I leave the house. In which case Syncthing appears to continue to stay connected, which is really handy.
176 comments
[ 3.2 ms ] story [ 247 ms ] threadYou can even set an encryption key per share, and if you want to use an untrusted device in your nodes, you can simply not provide it with the encryption key and it will simply synchronize the encrypted payload across nodes.
So if you have a shady/cheapo VPS, you can use its storage without worrying about the plaintext data being stored there.
I carried around my 1GB Data Traveler flash drive with great conviction. Then, a massive 4G SanDisk drive I would 'never fill up'.
Then, when my 16GB Nexus 4 filled up, one day while I was MTPing my old pictures to my computer, someone told me about Dropbox and how it would sync my pictures for me. After living out of Dropbox for all my non-cold-storage files...I hit the free limit. But I was a student and didn't have money to pay per month.
Skydrive gave me 25 GB. And 3 successive phones ate that like candy. So start paying up Mister. ugh.
But then, Syncthing came along here on HN. I installed it onto my phone and pc for camera download. then another, and another, and a separate folder for sharing files with a colleague, then my work SAN got laggy and lost files, so syncthing to keep my work files local and synced between work devices to the rescue.
And now, I don't pay for storage. or I do, but it just happens to come with word 365, but I don't bother using it.
And the flash drives? I'm up to a 256GB Sandisk Pro of some kind, but only for 'just in case' scenarios. Syncthing handles all of my LAN files access needs.
I’ve gone through so many iterations of trying to work out how to do something like this for Home Assistant config.
Do you set it up as realtime 2-way sync? And do you just exclude the .git folder from the sync to HA?
I've read of other HN'ers doing this successfully (syncing git repos between machines) but I just seem to get a lot of issues - git repos often broken temporarily, conflicts etc.
That said, I can't recall if I tried excluding node_modules when I last gave this a go. Maybe I'll try it again.
I should note I specifically use this for doing source control of configuration and making it easy to manually edit -- I have an entirely separate process for backups of the server itself.
Like you, I've also gone through several iterations but this has been working for me for almost a year now.
That said, I’m making far fewer YAML changes these days than I used to. I’m doing almost everything via their UI so maybe I need to consider how much I actually need source control now vs just having solid backups.
Originally I was against their move from YAML config to the more user friendly GUI approach but I have to admit it’s a lot easier to make quick on-the-fly changes to my automations these days. I’m begrudgingly coming around to the idea of not firing up VS Code every time I want to tweak something!
* iphone -> icloud
* icloud -> mac
* mac -> syncthing to the backup server
Standard backup policies from that point on.
I just found out about this tool from this (recent) thread: https://news.ycombinator.com/item?id=27758045
edit: seems to support live photos and exif metadata… I have a feeling some formats or metadata might be lost though, but not entirely sure…
https://data.syncthing.net
EDIT: added link
https://play.google.com/store/apps/details?id=com.github.cat...
My only annoyance with Syncthing is when I reinstall an OS on one of my machines. Let's say I'm syncing files between my main workstation, two laptops, a phone, and a cloud backup. Now I want to reinstall the OS on one of the laptops. When I install syncthing on that machine, it gets a new ID. I can make it join the sync swarm but all the other members will think it's a new machine and won't trust it so I have to go to each of the other machines and manually remove the "old" laptop and bring in the "new" one.
I simply save the syncthing config and remove the database, so that the device stays the same, but doesn't retain any sync data
~/.config/syncthing in Linux, the pem files and config.xml. Similarly in Windows too (from C:\Users\<username>\AppData\Local\Syncthing). Even in Android, you can export and back-up. When you are switching to new device, copy from the back-up and then import.
It works, but it's an hack...
https://docs.microsoft.com/en-us/windows/win32/fileio/hard-l...
I'm synchronizing less than 100 MiB of data, but it's changing all the time. I've not yet had a single issue with it. I've a few conflicts every week when the same file is changed in different locations at the same time, but Syncthing keeps all versions of a file until I resolve the conflicts; I can't see how it could handle that better.
Long story short: I highly recommend this tool.
On Windows, SyncTrayzor is the way to go.
Also good to note my setup has 1 introducer which is my NAS (synology) that is 24/7 online which always has the latest changes. Other devices are mostly mobile or turned on at irregular times. For devices without syncthing support like iOS, i rely on exposing the directory on the NAS via webdav. This way i can still access my data on the go.
And my synology creates regular backups/snapshots of my synced directory. So that i can always recover files. Even outside of syncthing.
Two things I like about it in particular, are
1. It can and favors syncing over local network. This has a huge cost savings in developing world. Even when you do have massive bandwidth, local network sync still has more throughput and lower latency. My music is shared between devices and if I add a track, it takes less than a second to appear on my other devices.
2. You can set conditions, such that delets can be ignored. Eg. I have a WhatsApp message+media backup going back almost 7 years now, ~65GiB. But I don't need all that on my phone. So I just disable syncing delets on my storage. Now my phone can get away with ~3 GiB of whatsapp data (only because I don't delete very often) and I still have complete backup in case I need it.
3. Absolute rock solid stable. At $DAYJOB we use syncthing to sync multi dozen GiBs of new data every day, total file count in 10s of thousands in about ~300 directories. And in past 5 years we have had about 3 instances where we had something that needed attention, out of which 2 were not syncthing's fault.
I've been using Syncthing for years and have absolutely zero complaints against it. Everyone with 2 or more devices should give it a try, just to see there are better options out there.
I was evaluating Syncthing for 1-way, append-only phone photo backup, and my Google searches warned me away from this, talking about "unsupported" and "database corruption."
I assume you have not experienced this, but I guess I should have just read the docs and trusted them, instead of forum posts.
https://docs.syncthing.net/advanced/folder-ignoredelete.html
Same here. What do you use instead? I haven't been able to find a good open-source multiplatform project for backups/1-way sync.
They had a pretty reasonable (~$6) in-app purchase to unlock samba access, and the fact that it's a German GmbH means I'm slightly more confident they're not trying to copy all our photos to a dark data-vault.
Happy user so far, though it took some cross-checking against the samba share feel confident it was backing up reliably (I'm just particular about trust-but-verify :)
https://www.photosync-app.com/home.html
Kubernetes is today's poster child for this. The Hashicorp stack can do just about all the same things, but it just works so there's no cottage industry of consultants and no market for as-a-service implementations. Why promote something like that?
Agree very much. That's why I never let go of a chance to enavgelise Syncthing and its OSS ilk in other fields.
1) I like to minimize the number of APT repositories I use. Debian's and Ubuntu's are well behind on Syncthing versions. Every so often (much more rarely than before), they have breaking changes. As such, my phone can no longer sync with my computers. Perhaps the answer is that Debian should just not be packaging it, because they have such a conservative policy. I'm considering downgrading my phone's version, or building the latest version from source. Worst case I may add their APT repo but I'm not eager.
2) At least with the version I've been using, opting for local-only sync isn't so straightforward. It's per-instance, not per-folder. This means that I have to have my home server running it as local-only, even for small things that I wouldn't mind getting synced over the wire. It means that if I want to have a local-only directory shared between my phone and laptop, it has to go through my home server first. Unless there's a better way I don't know about.
Otherwise Syncthing has been pretty great. I like your no-delete thing, I did not consider that. I could definitely use it for photos.
What’s wrong with adding a first party apt repo for syncthing (or other software)?
I would like to know, how are you using Syncthing to create local backups for WhatsApp database/msgstore (crypt14) and media files. You also mention 65GiB as your complete backup ─ have you tested it via a restore? If so, how?
Not the parent poster, but I also do this; I just configured Syncthing to share the /storage/emulated/0/WhatsApp (aka /sdcard/WhatsApp) directory. WhatsApp stores its daily local backup on the Backups subdirectory of that directory, and the media files are all in the Media subdirectory of that directory.
> have you tested it via a restore? If so, how?
I have actually used it to move all the WhatsApp and Signal data from an old phone to a new phone. Just have Syncthing synchronize these directories (with the same path) on the new phone before installing WhatsApp and Signal, then install and launch WhatsApp and Signal. When first launched, if that directory already contains a backup, both WhatsApp and Signal ask if you want to restore from that backup. Signal then asks you to type a long backup encryption key you should have written down somewhere, while WhatsApp asks its servers for the backup encryption key.
But to add, the message store the where the texts are stored. I usually keep it all. WhatsApp doesn't care about the media more than you do, so you can delete anything not needed, and now you have full text message history + important media on your new phone.
I have tested it while migrating across 3 devices and half dozen phone factory restores, so I'm fairly confident it still works. There are very ugly restrictions coming in future Android version with scoped storage and whatnot and I'm worried Google is finally going to cut me out. But until then, I keep mine.
I tried to use it to sync SASS and webpack between developers and it didn't work, it seems to have an issue with node_modules has too many files for it to deal with and it thinks the directory is hundreds of gb and just stalls with a million years eta.
The versioning is cool too but I found it a bit flakey.
If you need reliable versioning and syncing of lots of files use GIT, if you need easy cross platform syncing syncthing is the way to go.
Check it out every now and then to make sure, but it just kinda works
It's not a backup solution though... you really want to use actual backup software.
On Android 10 and older, Syncthing can only "read" form SD cards (read/write works if you're rooted).
https://forum.syncthing.net/t/android-11-and-syncthing-what-...
In the newest versions of Android you give it Scoped Storage.
In the intermediate, if you want 2-way sync on your phone, your data has to live in sdcard/android/media/com.nutomic.syncthingandroid/<sync folder1 >, <sync folder 2> which it does not need root to write to.
But for just grabbing your pictures, Android 10 has the stupid limitations on external file system access that Syncthing has to abide by or root over.
I don't remember when was the last time I had to use a flash drive or a cable to move files around.
---
[0] Android. iOS is too restrictive to do this.
I’ve gone through various attempts at keeping things in sync across a couple of Macs but always seem to hit issues.. (the issues not being specific to Syncthing)
See e.g. https://news.ycombinator.com/item?id=20955465 ; https://goteleport.com/blog/ssh-certificates ; and the usual man pages
I don't care much for cloud providers adding unnecessary restrictions, so I manage my own sshd_config on VMs. I'd like to rent a flat, not a hotel room, thank you very much.
> Git hosting platform
It appears GitLab, one of my platforms of choice, does support this in self-hosted setups[1]. Even if they didn't, it'd be trivial to extend their sshd_config by myself anyway.
1: https://docs.gitlab.com/ee/administration/operations/ssh_cer...
This is interesting about GitLab, thanks for the pointer. As you mentioned it only appears to be configurable instance-wide, there is no way for a user or organization to set it up for themselves (for example for your gitlab.com organization). That makes it a bad replacement for SyncThing'd private keys as moviuro suggested.
Haven't tried it while out-and-about, though that's not really my use case.
*RPi4 NAS has one external thumb drive for flash storage and two spinning rust drives, one for storage, one for backup; all connected via USB. Occasionally I have to plug-unplug the spinning rust drives if the system reboots, as the drives go to sleep.
I finally got rid of it because of the awful mobile experience, but frankly I shouldn't have been counting on that anyway.
What problems were you experiencing on mobile?
Now moved to a more powerful machine, but the concept is still the same.
This setup allowed me to conveniently stop using google drive.
I used to run Syncthing on an old mac mini with two 2.5" HDD running FreeNAS (previous name of TrueNAS).
So for you-to-you it’s seamless but if you want to sync shared-user directories then you lose most info.
Someone indicated they did have issues if you do a case only rename between a case sensitive (Linux) system and a case insensitive (Windows/Mac) one. I've never done this myself, but I guess that's one to run into. Also the usual caveats with any program, like that you won't be able to sync con.py to a Windows system, regardless of which program you use.
And with that having been said, it’s great. It’s maybe more powerful than you’d expect. My NAS for example has a one-way synced folder from a different box for backup, while having a default shared folder with all of my PCs. It’s also great because not all machines have to always be online; it can gracefully handle deferring stuff. For me, since my NAS is connected all the time, I can use it a bit like a central service, syncing between machines that have no overlap in uptime. Simultaneously, if the NAS needs to be down, it doesn’t interrupt syncing between other nodes. And unlike centralized solutions like Dropbox, I’m not limited by my internet connection, since it’s all local... unless I leave the house. In which case Syncthing appears to continue to stay connected, which is really handy.