That can be ruled out. Bit rot does not appear suddenly in a widespread manner and does not cause very similar visual effects in a variety of images. Also the original images are still available, just new modified versions were shown. To me it looks like a new compression algo gone wrong.
A bad bit wouldn’t even be noticeable in an image. Changing one bit would make literally no difference. Artifacts look like something the compression algorithm has messed up. Also since the images are actually intact according to Google this is likely just the compression algorithm used to browse the images.
Update: my limited knowledge on this is proof I’m wrong.
A bad bit might have no visible effect at all (e.g. Affecting part of EXIF metadata, or a totally black pixel turning again very close to total black) or can corrupt the whole image (flipping a bit in the file header or cosine quantization table).
As a total layman when it comes to storage: Don't they use forward correcting codes to make sure a single bad bit can reliably be detected and flipped without affecting the data at all? Or are these used only for data transmission?
I've encountered older digital photos where a small corruption affects all of the remaining parts of the image as it is presumably recorded in some linear fashion as shown in this Wikipedia entry's "Visual example."
I had noticed one of the images in my Google Photos account seemingly not showing up. It was very noticeable because it was one of the "People" images in my Photos account. It's a bit concerning that a photo storage service seemingly can't maintain the integrity of their data. Makes me wonder about what other data issues there are, lurking in the depths of Google Drive...
I personally found a corrupted file stored in online storage only, though it wasn't with Google, but with Microsoft's OneDrive.
I tried to open a Word document that I hadn't viewed for several years, but got an error that the file was damaged. It looks like this happens sometimes, after searching for the error online. Luckily I was able to recover an earlier version of the file (via version history), but it was alarming that it happened in the first place (also I'm not sure if important edits were made in the latest corrupted version).
It's unfortunate how many cloud services default to storing on the cloud only instead of also keeping a local copy, and don't even provide the option to opt out. Even if you choose the "keep offline" option for several services, for unknown reasons, this doesn't seem reliable (Maybe I'm 'using it wrong'? But in practice, I've found myself having to download files that I was sure I set to keep offline.)
Google is pretty proud of their record of not losing user data. As far as I know, they have never permanently lost even one byte of consumer data.
The same isn't true for GCP - there they lost some writes to customers persistent disks - but only ~50 megabytes worldwide, which is still rather good when you consider they store millions of terabytes.
It really depends what you consider lost. Thousands of users (at least) have been permanently and eternally cut off from their Google data after some AI system decided to axe someone's access to their own account. The loss is significantly over a byte. Or 50 MB.
> As far as I know, they have never permanently lost even one byte of consumer data.
It seems that every other day you learn about people being permanently locked out of of their Google account. Every byte of data stored in there is permanently lost to the person losing their account.
What if the policy is to lock accounts when data integrity errors are detected...
"We have always had better storage durability than Europa."
Note, I am not arguing that this is necessarily what happens. The thread full of contrary views on cloud storage just tickled this cynical take loose from wherever it was lodged.
They definitely had permanent data loss in App Engine around a decade ago, along with leaving any app active during the outage with copies of every database table to manually merge
> Google is pretty proud of their record of not losing user data.
As far as I know, they just haven't admitted to losing consumer data. Until they define that and put even a little bit of effort into checking whether it might hold true, they don't really have anything to be proud of.
There have been multiple mass-deletion events and Google has admitted to at least one of them - the infamous "only 0.02% of our users" incident, for example - but only because so many people were affected, the press picked up on it and google's PR firms couldn't keep it quiet.
For context, I know this is a display issue and not a storage issue and you can get your originals out via Takeout.
That being said my photos are the only digital files I really care about not losing. My photos and videos from my phone get backed up to iCloud, Google Photos, OneDrive and Amazon’s photo storage that comes with Prime.
My videos get backed up to all of the above except Amazon’s storage.
You should back them up locally as well even though that has its own failure points. I consider anything other than a dedicated local and/or cloud file backup system a nice supplementary place to store photos but not primary.
At one point I had them synced to my Windows computer via iTunes and that was backed up to Backblaze.
But, I don’t have a personal computer anymore and soon won’t have a personal residence. My wife and I are going to be digital nomads traveling across the country for a few years.
The local ZFS array is the “source of truth”. ZFS has built-in end-to-end checksumming. I also have a monthly scrub that should correct any untouched corrupt blocks.
Thanks, that is one way to do it. But if you don’t know about par2 (sorry, I can’t tell from what you’ve said) it’s worth chasing down. It is way, way beyond checksums in what it can do.
It’s an old project and information on the internet is getting thin but the math is sound, the tool works, and there is a wikipedia page:
Once a year I select about 100 of my favourite pictures from my phone and have them printed and bound to a book. I suspect this is likely to be the only way my descendants will be able to see them.
That's why I, and every other enthusiast photographer I know, keep my archive locally, meaning on my laptops hardrive and an external one as backup. You cannot trust any cloud provider to properly maintain your data, and external storage is becoming cheaper every year.
I love having a NAS that replaces Time Machine, Dropbox, iCloud, Netflix, Spotify, Disney plus, Hulu, runs home automation, has my security system on it, etc etc.
People use so many cloud services and having it all private and self-hosted is absolutely amazing. It’s really not that hard to set up either, and there’s little to maintenance outside replacing a failed drive.
I vastly prefer content be cloud based. Ever since Spotify came out, I've never downloaded music illegally again -- it's just so much better and more convenient. Can't imagine going back to what.cd days and having to download and maintain a music collection manually.
People here likely know how to do all that safely, but lots of less technically inclined people end up with their data encrypted and have to choose between starting over or paying a ransom. I have a QNAP device so I follow their subreddit and there's a steady stream of people who have lost everything to ransomware because their NAS was visible from the internet. I follow the standard advice from that subreddit - keep it behind a firewall with UPnP turned off and no ports forwarded to the NAS and never enable MyQNAPCloud (or whatever it's called).
Yup. QNAP software quality (and hardware) is complete garbage. I’ve off and on tried to incorporate them into my “system” and they always let me down. You’re putting everything at risk using one of their boxes.
Law of propability: I had zero dataloses due to fire, some data loss due to loosing a thumb drive backup and some data losses due to cloud storage shennenigans.
You are right, a cloud storage would make things a tad saver. Personally, I made a point in not putting anything in the cloud, especially none of the big providers, since I value my privacy and just don't trust any o those companies.
For the really important digital stuff (almost 30 years of family digital photography and video and some paperwork) I have a rotating backup schedule. Two of the drives in the rotation stay at my mother's place. Nothing fancy and it's mostly when I can bother to remember. About once a year or so. Still, some peace of mind.
> You cannot trust any cloud provider to properly maintain your data
I trust any cloud provider a lot more than my lazy self.
> and external storage is becoming cheaper every year
Sure, but electricity isn't. At the current price in the UK it seems like running a small NAS would be 10 GBP per month, more than most online storage solutions.
That is without counting the initial investment of buying the NAS, the cost of maintaining it (if I have to spend an hour a month on maintenance, that's $1000+ a year of time wasted), the footprint of the NAS, the noise, the heat, etc.
Yeah. I have a NAS for other purposes but redundant external USB drives is the easiest solution. You can even periodically make a clone and store it somewhere else.
It sounds like you know... very little on the topic. My NAS requires no maintenance, it makes no sound while running, and costs maybe pennies to run.
Meanwhile, there hasn't been a cloud provider without massive reliability or data loss issues. My NAS has better uptime than Amazon, Microsoft, or Google.
Their estimate of £10/month seems reasonable to me – that's basically 1 or 2 kWh of electricity per day (can't find current UK electricity rates). Is your NAS under 40W power draw?
My RPi 4b runs on a 18W power supply, which would put it at 432Wh per day if running at max load (It doesn't, it idles at <3W most of the time). Any arm NAS is going to have a similar power draw, and you really should pay attention to the power draw when choosing a NAS.
What about the storage devices (the S in NAS)? My external HDD (just for reference) has an AC adaptor that outputs 12V at 1.5A, so that's another 18W. And most people who care enough about storage to have a NAS have multiple disks...
I'm not saying it isn't under 40W - as you note, if it's idling properly, it's pretty easy to have a NAS including disks under 10W idle. But the electricity cost is not negligible when you are running something 24/7. Even 10W24hr30day is 7.2kWh per month, and most places cant count on 8c/kWh electricity anymore.
> My RPi 4b runs on a 18W power supply, which would put it at 432Wh per day
Which would be 13 kWh per month. If we're comparing to monthly fees for cloud storage, it's important to look at the same time frame.
I have two SSDs on it that are powered by the RPi itself, so the whole package is < 18W. Also, SSDs use way less power than HDDs.
> Which would be 13 kWh per month. If we're comparing to monthly fees for cloud storage, it's important to look at the same time frame.
Yes, running at max load. A more realistic load will be 3W/hr -> 72W/day -> 2kWh/month. That would put it at ~0.3€/month at a realistic load and ~1€/month at max load (0.15€/kWh where I live).
You also have to take into account there's other stuff running on such "NASes", like the PiHole, a git server, a plex server, which are going to cost you separately if you need them. On the other hand, my setup cost me ~300€ (RPi, charger, drives, USB-SATA adapters), which buys a whole lot of VPS and a whole lot of online storage.
Yes, SSDs would certainly help cut your power load down significantly! My system has 3 HDDs and it's a fairly small NAS compared to many - SSDs are prohibitively expensive once your storage needs get above a couple TBs.
So you took it out of the shipping box, plugged it to power, and pressed the power button, and everything was up and running?
And since then you haven't had to update any software on it, reboot it, unplug and replug an ethernet cable, shut it down and opened it to add storage, etc.?
> it makes no sound while running
Looking at a basic NAS on Synology shows that it's around 20 dB, which is admittedly very little, but is not 0.
> and costs maybe pennies to run
Again, looking at the same NAS and using my actual price per kWH, it's around $10 a month.
> there hasn't been a cloud provider without massive reliability or data loss issues
My searches haven't turned up any article about a data loss from any of the 3 big cloud providers. Even the story here, while scary, is just about compressed files, not originals.
---
I'm not denying that a NAS has advantages (the fact that it's a lot more than just a data backup), but in the context of backing up files, it's not at all convenient compared to using online solutions.
Can you share some specifics on the setup? Trying to do something similar. Haven’t tried it yet, but MacOS can create encrypted “sparse images”, which can be mounted and contain files. I’d like find to a better way, especially something that could stand the test of some time (10 years seems reasonable).
That’s a pretty bold statement. The null hypothesis is that there’s nothing in the body that cannot in principle be emulated in silico to whatever precision necessary.
Why is that the null hypothesis? For one thing, the only experience we have is existing as bodies in a real world, so that should be the default with any other claims being supported by evidence. For another thing, simulating a bridge falling down is not a bridge falling down, and it's not proven that simulated thinking is thinking (or even, possible).
Sad news for people hoping for a life after death, though! (ie nearly everyone)
I think it’ll be possible to create inorganic ‘brains’ that have human-like subjective experience, because to think otherwise reduces to some form of vitalism, but I doubt we’ll be able to move specific individual consciousness across technologies.
Well, I’m assuming the majority of people currently alive (and even more of those who have ever lived) are religious, and the majority of religions followed hold with the Heaven & Hell / Cycle of Reincarnation until Enlightenment model - and that experience in the afterlife was still recognisably ’human’.
Though, saying that, OP’s point about needing a human body to have human experiences may be correct if souls get a new human body in the afterlife - I’d just assumed liberated souls, like ghosts, had no physical form.
What makes you say that? You could literally be a brain in a jar or be hooked up to god knows what "deep diving" / "full VR" experience humanity will come up with.
> The human experience is inseparable from the human body.
I'd say that the human body is just one very sophisticated I/O device.
The fact that we can't replicate it now doesn't mean that it wouldn't be possible after a millenia of engineering, even though this replacement device might as well be a partially organic one in the end.
That said, there are no guarantees that we don't fall into a sort of "dark age" of no more growth/research into any new directions, or don't wipe ourselves out altogether.
My family has/had a handful of generations worth of records. The problem is finding out who last had what. Some of it has been backed up digitally, but finding a thumb drive is a lot harder than spotting a man sized pile of boxes.
But at least people will notice the photo book and at least have one look through. And it might even be a great moment when family members together can look at it and discuss memories. And quite likely it will be kept by a descendant. And it can be found one day by their descendants.
The digital archive or cloud storage will likely in many cases not even be discovered. And if it is discovered, it will probably be such a load of images, good and bad, that no-one will have the time to really look it through and appreciate it.
But for sure, both options combined is probably best.
My gmail account has a deadman’s switch to grant full access to my next of kin.
If I had a gigabytes of photos of my grandparents or great grandparents I would spend hours pouring over them looking at what their life was like before I knew them. That would be such a neat keepsake.
I imagine someone may one day do the same for me, but sadly most of my photos are clogged with mundane screenshots and repeint pictures and other junk.
I hate to tell you this but many people have been locked out of their Google Accounts despite knowing the correct username/password combination. Their security policies and ML algo locks accounts and requires 2FA to be used from time to time. How confident are you that in your passing, your next of kin will be able to get into that account if 2FA or backup codes are inaccessible?
Given their is no real customer service team that handles their free tier accounts, I am personally reluctant to assume my Gdrive data will be salvageable.
According to the Google Inactive Account support page, they aren't actually assuming the account, they're just getting access to the data. So, presumably, they'd just download the data and that would be it.
My experience is that when someone dies, all their paperwork gets left in a drawer for years until the next house move and then it's all thrown out.
My experience has been the opposite, with relatives eagerly snatching up the old photographs.
My boss is young and one of those people for whom digital is the only way. Then one day she got a box of old photographs and was absolutely amazed that she could turn them over and read where they were taken, on what occasion, and who was in the photos.
She now prints out all of her photos and writes notes on the back of them and keeps them in a box, just like people have done for the last century.
I was on the subway in Toronto in 2017, two college age girls had an inch thick stack of 4x6's that they were going through. It was a strange throwback to the pre-digital era.
Nope. It will be passed to the children. But likely not the grandchildren. Even if they do, the knowledge behind the pictures will be lost and grandchildren wont know who's who in most of the pictures. (grand uncles aunts etc). And grand-grandchildren may not even know grand-grandfathers anymore...
My wife does a photo book every year, combining the best of both our photos. It’s a bit of a chore, but I’m so glad she does it, they are so much nicer to browse through. The kids love looking at them all from time to time.
My first thought was that Google as recompressed everyone's images to save space and corrupted them in the the process. Hopefully everyone gets their images restored.
I was thinking something similar: it has a distinct posterization look that I tend to get when I mess around with images and reduce the bit-width of my input a little too much and my integers overflow. But those tend to result in "zebra stripes" like patterns, so you're probably right that the issue is more one of not using saturating integers while amplifying a value above the bit-width of its container.
Do you have any suggestions for a secure/private option to store photos? I'm not sure what would be best. (Dang, I really want to move away from Goo now)
Your own devices, with a proper backup solution (which you should already have with or without your photos in it). Nobody can mess with your data then.
Personally, I use Syncthing to sync between devices, including a NAS as a target, which regularly backs up to Backblaze.
This sounds a bit like what I do. I r-sync two different libraries to NAS then upload them to Backblaze B2. Post r-sync success result to a muted Slack channel and failure to an unmuted alert channel.
As a starter, look for a "dumb" service that doesn't treat photos differently from any other type of binary blob. No thumbnails. No grouping by date or location. No tagging other people. The storage service should only concern itself with safekeeping the binaries you upload, not analyzing and processing them. Use your own machine to do the latter.
I stood up a pair of QNAP NAS for both my parents and I, and encrypt-then-backup their photos/documents to my NAS, and vice-versa. RAID 5 or 6 keeps hardware failure at bay, and keeping them tucked away in our respective homes makes fire or flood irrelevent (for this).
QNAP or Asustor has standalone apps my parents can use as well. Without getting your parents involved, you could either pay for remote hosting for your off-site, or trade RAID space with a friend.
If you subscribe to the cypherpunk mantra that ”code is law“, then yes. Otherwise, if you stick to the law that lawyers and judges apply, then mostly no. NFT ownership does not come with any rights to the underlying asset, unless specified in an accompanying contract.
That's a fairly serious accusation to level. Google's bad behavior here is not that their compression algorithm ruined users photos. Google's bad behavior is running a compression algorithm at all.
That's not the point. I doubt that the engineers who worked on the compression algorithms would store their own precious photos using the same techniques.
Also, when was the last time you heard a banker say that everyone makes mistakes all the time.
Hi, I work on the processing algorithms at Google like the one here (not sure about this particular issue) and I use Google photos for all my old photos. People at Google read Hacker News and fix things, they just don't comment on specifics for things like this.
> This is an indication that Google underestimates the value of data stored by users.
That becomes apparent if you ever had any issue with a google product. There's no way to resolve issues outside of canned answers from "AI" systems and public forums.
>Google underestimates the value of data stored by users.
Of course they do. How can they possibly put any value on endless PB of holiday snaps? They only care that you are now reliant on them to store your memories.
Yikes, this is kind of scary. Google Photos is the only place to find some of my older photos. What’s the easiest way to download my entire library for backup on a physical drive?
Google Photos used to have the option to sync all photos into Google Drive but that feature was removed. I’m guessing the raw data is still available via Google Takeout
I have been having real problems trying to do Google takeout with my Google Photos. I keep getting an error emailed to me saying "However, we were unable to create a copy of all your files. You can still download your files until September 25, 2022, but please be aware that there will be data missing."
Apparently I am not the only one[1][2]. Other people have a mix of errors[3]. It is not the reliable data extraction I was expecting for something as valuable as my family photos. Google is losing a lot of trust from me based on this.
It works extremely well, and you can re-run it any time to sync new files locally too.
I used it like that for a few months before I finally installed syncthing on my phone and stopped using Google Photos altogether.
Now what I do is take photos on my phone, have them sync to a NAS. And on the NAS I used a modified version of this https://forum.syncthing.net/t/android-photo-sync-with-exifto... to build up a YYYY/MM folder organisation and move files older than 30 days from the syncthing folder into my archive. My archive is then in my Plex so it's still accessible to me.
In essence: 1) Take photo (implicit sync to NAS), 2) wait 30 days, 3) archive photo into long term directory naming convention, whilst making available to Plex and deleting the version from my phone (by deleting the syncthing version it will delete the one on the phone after 30 days too).
I've been putting off a Google photos back up for years, and this was my excuse - waiting for this issue to be resolved. But I'm being a fool, 60% compression is better than losing everything.
I have just compared originals I had which were uploaded to Google Photos vs the ones downloaded.
Visually there is a barely perceptible difference in a side-by-side comparison on a high quality and calibrated monitor. I wouldn't have noticed if it weren't side-by-side and zoomed in.
On the file size, it's gone from a 6MB jpeg down to about a 1.5MB jpeg.
On the smaller files that difference is far less, but jpegs above 3MB seem to see the biggest reduction in file size.
Thankfully I have all of my originals that weren't taken on the phone camera so I don't mind this so much as running my exiftool script over the original archive produces the same naming convention and I can merge those folders and still have the originals I care about.
Sheesh though... such a thing for Google to screw up on.
Nothing a little Stable Diffusion can't fix. Soon, digital memories will be just like real ones — slightly different every time we look back, but with no way of knowing what's changed
The promise of the cloud was that you don't need local storage, because Someone Else will remember things for you, and you just ask them what you remember. This is also the problem of the cloud.
"The promise of the cloud was that you don't need local storage, because Someone Else will remember things for you ..."
Agreed. And that promise has been fulfilled - although it costs more than nothing and you need to do your homework.
You know these providers are proven bad actors. You know their business model(s) are antagonistic to your goals. You also know there are stable, time-tested alternatives.
> slightly different every time we look back, but with no way of knowing what's changed
This is already the case with different image compression algorithms improving on one another. Users couldn't and shouldn't tell the difference between JPG, or slightly smaller / resized JPG, or WebP, or whatever. I personally don't think image hosts should have to alert users when making undetectable changes to their images like this.
I'm sorry, but JPEG compression is extremely noticable especially with non-photographic material such as web graphics. The compression is lossy by nature, and predictably introduces noticeable artifacts. WebP on the other hand has support for lossless encoding.
It's simply not the same thing. If I opened my file browser one day and saw that all of my FLACs had been converted to MP3, I would go on a murder spree.
I think you misinterpreted the comment you're responding to; they meant that "sadly, you [the user] never thought to store a high-quality hash (e.g. SHA256) for integrity comparison with your stored data; and the hash you did decide to archive for integrity-comparison was a low-quality one (e.g. MD5) that can be trivially preimage-attacked such that a cloud provider could silently replace your data with a different one — with the same low-quality hash — without your knowledge."
Nothing about rsync.net per se, other than the general idea that any data you put on "somebody else's server" can't be trusted to stay the same if you don't have a high-quality integrity-comparison content-hash of that data kept somewhere.
To be fair, it may yet turn out that MD5 is (and has always been, we just didn't know it yet) a low-quality hash that can be trivially preimage-attacked. I think that's unlikely[0], but I can easily imagine someone reasonable being a bit more pessimistic. I certainly wouldn't willingly rely on MD5 preimage resistance for anything important.
0: I'd be willing to bet money at 1:1 odds that MD5 preimages still cost at least 2^96 bit operations (out of a nominal 2^127 hash invocations) 5, 20, or 100 years from now.
Extraordinarily difficult, judging by GP's cynicism. Personally it is appalling just how little people are motivated to be nomically tech-literate in this extremely tech-driven society.
Tech is the only industry that asks this of it’s users, it’s just laziness on the developers part. People needing to know about hashing in order to save photos would be like people needing to know what refrigerant their A/C uses in order to keep cool.
Can you? Sure. Are their weird edge cases where it’s mildly helpful to know as opposed to having to ask someone? Debatably. Should you need to in order to not have things break for no reason? Absolutely not.
I kind of agree with you both. We do ask a lot of our users. But also, there is no A/C system out there nearly as complex as software. It is hard. For everyone. We can do better, but what does that even look like? Consumer level technology consumption is not in a great spot, but it also isn’t horrible. Power users who can get more out of computing will be a thing for a long time to come.
Lots of things are complex and confront people to deal with it directly, we just also have legions of people that we can pay to step in for us and a culture where laziness and convenience are core principles to our everyday lives. Not to mention a political establishment that touts the "virtues" of a simple, ignorant life.
Maintaining a house or yard, for example, or taxes. Driving also has a pretty high learning curve -- high enough to mandate driving lessons -- but once we learn it we're set for life. (Tech is the same way here....)
why do you assume GP is talking about general users? as you point out, this is HN: it’s reasonable to think that most comments here are written for that mostly-tech-literate HN audience, unless they state otherwise, no?
Yeah. And somebody may put up an Open Source repo for doing that 'easier'. But of course, nothing that any average user on the net could even know to exist, leave aside actually use...
As someone who works in generative modeling, that's probably what is creating these issues in the first place. Kinda like how you can see certain GAN type of patterns on Netflix when there is upscaling or if you don't turn off the fancy features on your TVs. Of course, finding how these images are corrupted leads to better models so they will be solved. This isn't the first time Google has edited photos and made them worse and it won't be the last. But the trend is that the photos get better.
I've been long considering switching my Google Photos library fully over to ente [0], of course alongside my existing monthly backups to local HDDs and an annual backup to a cloud server via borgbackup.
With every passing week, every Google fuck up like this and with ente's great pace of development, I draw myself nearer and nearer to the precipice of full switchover. I will soon be having my last Google Takeout.
I remember trying ente, but the mobile app had a notoriously low amount of features and low FPS (felt like 30fps on my modern, 120Hz phone - stuck out like a sore thumb). Has there been any progress on that front?
Hey! One of the founders of ente here. We have shipped a bunch of performance improvements and features[1] over the last few months. But we are far from where we want to be.
If you happen to revisit and find something lacking, please write to vishnu[at]ente.io, we'd love to make ente work better for you.
"duck" means "mine" in Malayalam (my native language), and I have a weird obsession with rubber ducks, so it just fit, and perhaps most importantly, the domain was available :)
This great Hetzner page[0] explains it really well - its pretty much exactly what I'm doing. My Google Photos are extracted from the Google takeout.zip and those folders are backed up into a borg archive.
Seems unlikely Dropbox would be immune from some sort of issue like this. Probably best to have your photos in storage somewhere else (ideally both locally and off-site).
You know how the mantra in the crypto community is "not your wallet, not your coins?" This applies to this situation. Once you trust Google with your bits, don't be surprised when they modify them without telling you.
The susceptibility of the SD card to data corruption is a big showstopper. The only digital files I've had corrupted were stored on my phone's external SD cards (after 3 to 7 years). When I looked into it, it seemed to be an issue related to lack of radiation hardening. So if you do make this service, please include some error detection/error correction metadata.
Only tangentially related but I just checked the Google Photos iOS app to see if any of mine are corrupted, and it won't even allow me to view my cloud photos without giving it full permission to all current photos on my device?! Isn't this against the app store policy? You can't deny me access to one feature just because I refuse to grant permissions to a different feature.
But because it's Google I'm sure they can break whatever rules they want as long as it doesn't involve trying to avoid Apple's 30% fees.
Institutions of that nature are incapable of following through on nearly any promise made to the public at large; every promise is a lie, whether they break it or not, because they can't be said to have the intention to uphold it.
I see no reason not to believe that Page and Brin sincerely desired Google not to be evil; but it's the institution decides to do evil, not any individual, so they did not have (or could not maintain, as the organization scaled) the power to guarantee this. No one does.
Maybe they store physical photos and the albums got soaked in water. Sadly it happened to all our 90s photo albums during a flood. Some of the photos were stained really bad.
It immediately presented me with a screen with the choice to "save originals to cloud or save a compressed version to the cloud" (I don't recall the exact words) with no option to not upload to the cloud, and then immediately started uploading my library.
I scrolled back and checked some pre-2015 photos, they look fine, and then deleted the app.
A typical feature of these cloud drives is automatic uploading of pictures from the camera roll, that’s the legitimate use case for demanding access to all pictures. Who knows when Google whoops! accidentally makes that the default. You know, it’s optional, you can just turn it off!
I uploaded a photo to google maps. A week later it found some random picture of a local park and offered up a notification suggesting that I upload this photo to the destination it had located in the maps app.
How on earth are they doing this without "scanning" the photo? Seriously, please tell me.
This is an engagement hack. Get the user to give max permissions at the start so that any future call to action will have less friction and/or not require a modal.
It’s less about Google wanting to scan your data today and more about Google’s stingy strategy for photos plus Google’s lazy engineering (too many other interesting games to solve). Google Photos was born out of Google Plus, and is under heavy revenue efficiency pressure.
Each passing month I am more and more convinced that I should urgently get a) my mails out of my personal (non paying) Gmail account and b) get my associated photos out of Google Photos, mainly because of baby pictures which I cannot fathom to lose.
But then I think about the convenience of my 2TB Google Drive and I can't do it.
And I think, too, of the Polaroid pictures my parents took and gave me in a box once I got married and are now fainted. Mayhaps algorithms deciding that it's time to let go of your own data is just the modern form of yellowing.
> I should urgently get a) my mails out of my personal (non paying) Gmail account and b) get my associated photos out of Google Photos, mainly because of baby pictures which I cannot fathom to lose.
I highly recommend it. It feels like pure freedom, putting a price tag on my personal data. The migration is something done in two steps: set up your alternative service, but also keep the old one (gmail/gdrive/calendar/...) running. Then start to use the new email service, while keeping the old one.
There are enough affordable options and you do not have to migrate everything at once. I started with my emails - gmail is still running and forwards all incoming emails, so I wont miss anything. I already updated all my accounts/logins to the new email address.
The next step was copying all my files from Dropbox, GDrive, and so on to {service I am using}.
The data in the other clouds are still there, but I am going to delete it all soon.
There are things that are not perfect - like accepting appointments, which I still want to show up in my Google calendar. To those mails I still have to respond in google mail. But I am getting there, one step after another. The win so far is much bigger than the loss/pain.
I take periodic backups of my Google data using Takeout, primarily for the GMail backup. When I took one a few months ago, I noticed that the Drive part of the backup had some mangled filenames that the backup from a year ago didn't have. I don't remember exactly what it was but it was something like long-filename.pdf got truncated to long-file~01.pdf, like back in the 8.3 DOS days. Luckily I already have a local copy of the Drive contents so I didn't need to rely on it, but anyone else should be worried.
a) was very freeing. I didn't realize there were services so much better than Gmail. Even ones I pay a small amount for and get as many custom domains as I want.
b) did this a long time ago. Glad I did. I can sync any file I want too, not just photos. Really been liking Seafile + AutoSync (android) for this.
I'm going through the process of switching off of Gmail and onto a Fastmail account under my own domain. I'm loving the freedom of having my own domain that I can switch to another email provider whenever I want. Also, Fastmail with a custom domain has some nice features that were unavailable with a Gmail account (aliases, masked emails, automatically snoozing certain emails so I get a bunch all at once, etc.)
As for pictures, I do use Google Photos, but it's not my primary photo video store. It's more for the app that lets me lookup pictures.
For storing original versions of pictures/videos, I sync from my phone to a home computer using Syncthing, and store backups of my files on AWS S3 using Restic for encrypted backups.
If I lose Google Photos, no big deal. I still have local and cloud copies of my data.
This is definitely not a plug, but I'm actually working on an alternative to email, cloud storage, authentication/authorization/sso/etc. solutions from Google, Microsoft, etc. and one of my specific focuses is to allow free personal use and free smaller company use. It is refreshing to see I'm not building this in a vacuum. I started the project for no other reason except Google yanking out their free accounts. I absolutely do plan to commercialize it, however, I would still be happy with the result even if it failed, since under Google's pricing, I'd pay more than $1,000 a year vs. less than $10/year with my own solution.
Second this. Several years of use with nary an issue, plus Linux support.
Speaking of which, as an aside there's no excuse for big tech to not support the Linux desktop (I also use macOS and Windows 11 so not a die-hard fanboy). It doesn't matter if the userbase is tiny. Linux enables massive profits in the cloud so any cloud provider that supports multiple platforms but not Linux, whilst also generating huge profits from it, is doing a very bad thing. And that includes MS, Apple, and Google - full support regardless of the (comparatively small) cost as thanks for the profits they generate from it.
Most writable optical media use organic dyes that aren’t stable for more than a couple years—if you can still read the contents 10 years later, consider yourself lucky.
M-DISC use a more stable media, and should last much longer, but are (much) more expensive and require more recent burners. There are dual-layer M-DISC with higher capacities, too.
As mancerayder says below, why can't you back them up to an external USB drive?
I rsync my 'archive' partition (photos, videos, music, programs, etc) daily to several external 4TB USB drives. Those are dirt-cheap these days. The process takes maybe 5 or 10 minutes depending on whether or not there have been any new files to transfer in the last 24 hours.
I refuse to relinquish my files to some insouciant third-party. And what recourse would you have if they lost them?? Very probably them saying "Sorry. Not our problem."
Yes, I ran into this issue after I switched to iOS a couple years ago. Google demanded full permissions to even open the app. I promptly downloaded my image library on the web, and also, related to OP's post, some of my images had artifacts. Some were even corrupt. I moved everything over to onedrive, and restored the corrupted ones that I had local copies of (gradually, it took a lot of work) with local copies I have. I now mirror in "the cloud" with both iCloud and OneDrive. I also have a local copy and a couple of USB sticks with a backup handy on my keychain (solely for if a fire breaks out and I remember to grab my keys).
292 comments
[ 3642 ms ] story [ 3096 ms ] threadUpdate: my limited knowledge on this is proof I’m wrong.
It totally depends.
Even most data transmission components don't use forward correcting codes. They typically just have error detection.
https://en.wikipedia.org/wiki/Data_degradation
I tried to open a Word document that I hadn't viewed for several years, but got an error that the file was damaged. It looks like this happens sometimes, after searching for the error online. Luckily I was able to recover an earlier version of the file (via version history), but it was alarming that it happened in the first place (also I'm not sure if important edits were made in the latest corrupted version).
It's unfortunate how many cloud services default to storing on the cloud only instead of also keeping a local copy, and don't even provide the option to opt out. Even if you choose the "keep offline" option for several services, for unknown reasons, this doesn't seem reliable (Maybe I'm 'using it wrong'? But in practice, I've found myself having to download files that I was sure I set to keep offline.)
The same isn't true for GCP - there they lost some writes to customers persistent disks - but only ~50 megabytes worldwide, which is still rather good when you consider they store millions of terabytes.
https://duckduckgo.com/?q=google+drive+data+corruption
It seems that every other day you learn about people being permanently locked out of of their Google account. Every byte of data stored in there is permanently lost to the person losing their account.
"We have always had better storage durability than Europa."
Note, I am not arguing that this is necessarily what happens. The thread full of contrary views on cloud storage just tickled this cynical take loose from wherever it was lodged.
As far as I know, they just haven't admitted to losing consumer data. Until they define that and put even a little bit of effort into checking whether it might hold true, they don't really have anything to be proud of.
Back in the aughts Google was infamous for disappearing mail, if not entire accounts just disappearing into thin air.
https://techcrunch.com/2006/12/28/gmail-disaster-reports-of-...
https://www.theinternetpatrol.com/has-your-gmail-email-disap...
Years later:
http://www.cnn.com/2011/TECH/web/03/01/gmail.lost.found/inde...
That being said my photos are the only digital files I really care about not losing. My photos and videos from my phone get backed up to iCloud, Google Photos, OneDrive and Amazon’s photo storage that comes with Prime.
My videos get backed up to all of the above except Amazon’s storage.
But, I don’t have a personal computer anymore and soon won’t have a personal residence. My wife and I are going to be digital nomads traveling across the country for a few years.
My backup flow is: phone, iCloud, local ZFS array served by PhotoPrism (synced using PhotoSync), Backblaze.
Details on ZFS checksums: https://openzfs.github.io/openzfs-docs/Basic%20Concepts/Chec...
It’s an old project and information on the internet is getting thin but the math is sound, the tool works, and there is a wikipedia page:
https://en.m.wikipedia.org/wiki/Parchive
Edit: btw use of par2 files can be added on as an extra step with minimal storage overhead, without replacing anything you have now.
My home backup is going to be it's own problem after my death.
Exactly, and setting up a NAS has never been easier.
People use so many cloud services and having it all private and self-hosted is absolutely amazing. It’s really not that hard to set up either, and there’s little to maintenance outside replacing a failed drive.
That started me down the path to owning all my own media.
It also didn’t help that iCloud Photos is a privacy nightmare.
[0]: https://www.backblaze.com/blog/the-3-2-1-backup-strategy/
I trust any cloud provider a lot more than my lazy self.
> and external storage is becoming cheaper every year
Sure, but electricity isn't. At the current price in the UK it seems like running a small NAS would be 10 GBP per month, more than most online storage solutions.
That is without counting the initial investment of buying the NAS, the cost of maintaining it (if I have to spend an hour a month on maintenance, that's $1000+ a year of time wasted), the footprint of the NAS, the noise, the heat, etc.
A backup in one place isn't a backup. The 3, 2, 1 rule and all.
Meanwhile, there hasn't been a cloud provider without massive reliability or data loss issues. My NAS has better uptime than Amazon, Microsoft, or Google.
> costs maybe pennies to run.
Pennies per hour, per day, or per month?
I'm not saying it isn't under 40W - as you note, if it's idling properly, it's pretty easy to have a NAS including disks under 10W idle. But the electricity cost is not negligible when you are running something 24/7. Even 10W24hr30day is 7.2kWh per month, and most places cant count on 8c/kWh electricity anymore.
> My RPi 4b runs on a 18W power supply, which would put it at 432Wh per day
Which would be 13 kWh per month. If we're comparing to monthly fees for cloud storage, it's important to look at the same time frame.
I have two SSDs on it that are powered by the RPi itself, so the whole package is < 18W. Also, SSDs use way less power than HDDs.
> Which would be 13 kWh per month. If we're comparing to monthly fees for cloud storage, it's important to look at the same time frame.
Yes, running at max load. A more realistic load will be 3W/hr -> 72W/day -> 2kWh/month. That would put it at ~0.3€/month at a realistic load and ~1€/month at max load (0.15€/kWh where I live).
You also have to take into account there's other stuff running on such "NASes", like the PiHole, a git server, a plex server, which are going to cost you separately if you need them. On the other hand, my setup cost me ~300€ (RPi, charger, drives, USB-SATA adapters), which buys a whole lot of VPS and a whole lot of online storage.
So you took it out of the shipping box, plugged it to power, and pressed the power button, and everything was up and running?
And since then you haven't had to update any software on it, reboot it, unplug and replug an ethernet cable, shut it down and opened it to add storage, etc.?
> it makes no sound while running
Looking at a basic NAS on Synology shows that it's around 20 dB, which is admittedly very little, but is not 0.
> and costs maybe pennies to run
Again, looking at the same NAS and using my actual price per kWH, it's around $10 a month.
> there hasn't been a cloud provider without massive reliability or data loss issues
My searches haven't turned up any article about a data loss from any of the 3 big cloud providers. Even the story here, while scary, is just about compressed files, not originals.
---
I'm not denying that a NAS has advantages (the fact that it's a lot more than just a data backup), but in the context of backing up files, it's not at all convenient compared to using online solutions.
I think it’ll be possible to create inorganic ‘brains’ that have human-like subjective experience, because to think otherwise reduces to some form of vitalism, but I doubt we’ll be able to move specific individual consciousness across technologies.
Are you sure about that? That does not sound right to me.
Though, saying that, OP’s point about needing a human body to have human experiences may be correct if souls get a new human body in the afterlife - I’d just assumed liberated souls, like ghosts, had no physical form.
I'd say that the human body is just one very sophisticated I/O device.
The fact that we can't replicate it now doesn't mean that it wouldn't be possible after a millenia of engineering, even though this replacement device might as well be a partially organic one in the end.
That said, there are no guarantees that we don't fall into a sort of "dark age" of no more growth/research into any new directions, or don't wipe ourselves out altogether.
Nowadays it's only available inside the world of Black Mirror and similar fiction.
My experience is that when someone dies, all their paperwork gets left in a drawer for years until the next house move and then it's all thrown out.
The digital archive or cloud storage will likely in many cases not even be discovered. And if it is discovered, it will probably be such a load of images, good and bad, that no-one will have the time to really look it through and appreciate it.
But for sure, both options combined is probably best.
If I had a gigabytes of photos of my grandparents or great grandparents I would spend hours pouring over them looking at what their life was like before I knew them. That would be such a neat keepsake.
Given their is no real customer service team that handles their free tier accounts, I am personally reluctant to assume my Gdrive data will be salvageable.
https://support.google.com/accounts/answer/3036546?hl=en
(That's not to say there aren't other concerning reasons why that data may not be accessible.)
How do you do this?
My experience has been the opposite, with relatives eagerly snatching up the old photographs.
My boss is young and one of those people for whom digital is the only way. Then one day she got a box of old photographs and was absolutely amazed that she could turn them over and read where they were taken, on what occasion, and who was in the photos.
She now prints out all of her photos and writes notes on the back of them and keeps them in a box, just like people have done for the last century.
E.g. if this was about storing monetary data, this would never have happened.
Personally, I use Syncthing to sync between devices, including a NAS as a target, which regularly backs up to Backblaze.
rsync -av /Source /Destination rsync -h
QNAP or Asustor has standalone apps my parents can use as well. Without getting your parents involved, you could either pay for remote hosting for your off-site, or trade RAID space with a friend.
Another good test: upload your photo collection, then download it and binary diff the two sets of files.
Let’s hope none of the images were NFT “assets” then haha
Also, when was the last time you heard a banker say that everyone makes mistakes all the time.
That becomes apparent if you ever had any issue with a google product. There's no way to resolve issues outside of canned answers from "AI" systems and public forums.
https://twitter.com/samwcyo/status/1569897392560050178
Discussed in https://news.ycombinator.com/item?id=32835190
Of course they do. How can they possibly put any value on endless PB of holiday snaps? They only care that you are now reliant on them to store your memories.
https://takeout.google.com/
Apparently I am not the only one[1][2]. Other people have a mix of errors[3]. It is not the reliable data extraction I was expecting for something as valuable as my family photos. Google is losing a lot of trust from me based on this.
[1]: https://news.ycombinator.com/item?id=25591630 [2]: https://sysc.org/what-is-google-takeout/ [3]: https://www.reddit.com/r/googlephotos/comments/khited/google...
It works extremely well, and you can re-run it any time to sync new files locally too.
I used it like that for a few months before I finally installed syncthing on my phone and stopped using Google Photos altogether.
Now what I do is take photos on my phone, have them sync to a NAS. And on the NAS I used a modified version of this https://forum.syncthing.net/t/android-photo-sync-with-exifto... to build up a YYYY/MM folder organisation and move files older than 30 days from the syncthing folder into my archive. My archive is then in my Plex so it's still accessible to me.
In essence: 1) Take photo (implicit sync to NAS), 2) wait 30 days, 3) archive photo into long term directory naming convention, whilst making available to Plex and deleting the version from my phone (by deleting the syncthing version it will delete the one on the phone after 30 days too).
The only thing that it gets wrong is that it cannot restore the file created date to video. But in the grand scheme of things, that's a nit.
I've been putting off a Google photos back up for years, and this was my excuse - waiting for this issue to be resolved. But I'm being a fool, 60% compression is better than losing everything.
I have just compared originals I had which were uploaded to Google Photos vs the ones downloaded.
Visually there is a barely perceptible difference in a side-by-side comparison on a high quality and calibrated monitor. I wouldn't have noticed if it weren't side-by-side and zoomed in.
On the file size, it's gone from a 6MB jpeg down to about a 1.5MB jpeg.
On the smaller files that difference is far less, but jpegs above 3MB seem to see the biggest reduction in file size.
Thankfully I have all of my originals that weren't taken on the phone camera so I don't mind this so much as running my exiftool script over the original archive produces the same naming convention and I can merge those folders and still have the originals I care about.
Sheesh though... such a thing for Google to screw up on.
You can just hash them locally or use AI to identify modifications.
Agreed. And that promise has been fulfilled - although it costs more than nothing and you need to do your homework.
You know these providers are proven bad actors. You know their business model(s) are antagonistic to your goals. You also know there are stable, time-tested alternatives.
What did you think was going to happen here ?
Edit: to pay a minimum of $12/month??
This is already the case with different image compression algorithms improving on one another. Users couldn't and shouldn't tell the difference between JPG, or slightly smaller / resized JPG, or WebP, or whatever. I personally don't think image hosts should have to alert users when making undetectable changes to their images like this.
It's simply not the same thing. If I opened my file browser one day and saw that all of my FLACs had been converted to MP3, I would go on a murder spree.
Well, let's see:
ssh user@rsync.net sha256 photos/IMG_1234.HEIC
Yep, checks out.
The checksum commands:
(md5, sha1, sha224, sha256, sha384, sha512, sha512t256, rmd160, skein256, skein512, skein1024, cksum)
... are available to run as you see fit.
Nothing about rsync.net per se, other than the general idea that any data you put on "somebody else's server" can't be trusted to stay the same if you don't have a high-quality integrity-comparison content-hash of that data kept somewhere.
0: I'd be willing to bet money at 1:1 odds that MD5 preimages still cost at least 2^96 bit operations (out of a nominal 2^127 hash invocations) 5, 20, or 100 years from now.
Can you? Sure. Are their weird edge cases where it’s mildly helpful to know as opposed to having to ask someone? Debatably. Should you need to in order to not have things break for no reason? Absolutely not.
Maintaining a house or yard, for example, or taxes. Driving also has a pretty high learning curve -- high enough to mandate driving lessons -- but once we learn it we're set for life. (Tech is the same way here....)
By the principle of explosion[0, eg].
0: https://en.wikipedia.org/wiki/Principle_of_explosion
As someone who works in generative modeling, that's probably what is creating these issues in the first place. Kinda like how you can see certain GAN type of patterns on Netflix when there is upscaling or if you don't turn off the fancy features on your TVs. Of course, finding how these images are corrupted leads to better models so they will be solved. This isn't the first time Google has edited photos and made them worse and it won't be the last. But the trend is that the photos get better.
With every passing week, every Google fuck up like this and with ente's great pace of development, I draw myself nearer and nearer to the precipice of full switchover. I will soon be having my last Google Takeout.
[0]: https://ente.io/
I switched to the service completely.
edit: RTFA, guess ente does - https://github.com/ente-io
If you happen to revisit and find something lacking, please write to vishnu[at]ente.io, we'd love to make ente work better for you.
[1]: https://roadmap.ente.io/top-released/
[0]: https://community.hetzner.com/tutorials/install-and-configur...
It looks like Google has unpublished our app, will figure out a fix.
[1]: https://ente.io/download/apk
[2]: https://f-droid.org/packages/io.ente.photos.fdroid/
I hope to fix up my prototype and launch it next year.
https://www.clonecamel.com/
If anyone knows of a quality source of bulk SD cards or flash drives in high capacity please let me know.
The idea would be to subscribe to have this data shipped once or more per year so hopefully some of the data would survive.
Any advice on error correction metadata and tools?
But because it's Google I'm sure they can break whatever rules they want as long as it doesn't involve trying to avoid Apple's 30% fees.
Trusting a dubious central authority up until you get burned is a bad heuristic.
Just location tracking. The lies. Over, and over, and over, and over again the lies.
I see no reason not to believe that Page and Brin sincerely desired Google not to be evil; but it's the institution decides to do evil, not any individual, so they did not have (or could not maintain, as the organization scaled) the power to guarantee this. No one does.
Maybe they store physical photos and the albums got soaked in water. Sadly it happened to all our 90s photo albums during a flood. Some of the photos were stained really bad.
I scrolled back and checked some pre-2015 photos, they look fine, and then deleted the app.
I uploaded a photo to google maps. A week later it found some random picture of a local park and offered up a notification suggesting that I upload this photo to the destination it had located in the maps app.
How on earth are they doing this without "scanning" the photo? Seriously, please tell me.
It’s less about Google wanting to scan your data today and more about Google’s stingy strategy for photos plus Google’s lazy engineering (too many other interesting games to solve). Google Photos was born out of Google Plus, and is under heavy revenue efficiency pressure.
But then I think about the convenience of my 2TB Google Drive and I can't do it.
And I think, too, of the Polaroid pictures my parents took and gave me in a box once I got married and are now fainted. Mayhaps algorithms deciding that it's time to let go of your own data is just the modern form of yellowing.
I highly recommend it. It feels like pure freedom, putting a price tag on my personal data. The migration is something done in two steps: set up your alternative service, but also keep the old one (gmail/gdrive/calendar/...) running. Then start to use the new email service, while keeping the old one.
There are enough affordable options and you do not have to migrate everything at once. I started with my emails - gmail is still running and forwards all incoming emails, so I wont miss anything. I already updated all my accounts/logins to the new email address.
The next step was copying all my files from Dropbox, GDrive, and so on to {service I am using}. The data in the other clouds are still there, but I am going to delete it all soon.
There are things that are not perfect - like accepting appointments, which I still want to show up in my Google calendar. To those mails I still have to respond in google mail. But I am getting there, one step after another. The win so far is much bigger than the loss/pain.
As for pictures, I do use Google Photos, but it's not my primary photo video store. It's more for the app that lets me lookup pictures.
For storing original versions of pictures/videos, I sync from my phone to a home computer using Syncthing, and store backups of my files on AWS S3 using Restic for encrypted backups.
If I lose Google Photos, no big deal. I still have local and cloud copies of my data.
Why not sync them to a cheap external drive on a regular basis?
Speaking of which, as an aside there's no excuse for big tech to not support the Linux desktop (I also use macOS and Windows 11 so not a die-hard fanboy). It doesn't matter if the userbase is tiny. Linux enables massive profits in the cloud so any cloud provider that supports multiple platforms but not Linux, whilst also generating huge profits from it, is doing a very bad thing. And that includes MS, Apple, and Google - full support regardless of the (comparatively small) cost as thanks for the profits they generate from it.
M-DISC use a more stable media, and should last much longer, but are (much) more expensive and require more recent burners. There are dual-layer M-DISC with higher capacities, too.
I did a bunch of research on long-term data archival and wrote it up here: https://photostructure.com/faq/how-do-i-safely-store-files/
I refuse to relinquish my files to some insouciant third-party. And what recourse would you have if they lost them?? Very probably them saying "Sorry. Not our problem."