Strange thing is that anti-Apple posts on HN get lots of discussion about how Apple is bad. Then a few days later a new MacBook is released, and there is a post on HN where everybody is raving about Apple. Then the next day, everybody is hating Apple again. It's confusing.
People who care about e.g. right to repair or other issues Apple's behaviour is poor for will be more likely to dislike Apple and comment on those posts, while people who are like Apple are more likely to comment on posts about the new MacBook or iPhone or other device
The repair folks and apple is such a weird relationship. My iphone lasts so much longer than android, the updates keep on coming much much longer (my parents android phone NEVER got an update and shipped 1 version behind at time they bought it). It's pretty dang durable, waterproof, and I've never had a problem getting it fixed (I use AppleCare+).
Seriously, it seems from where I'm sitting if you want a reliable phone, one that holds its value, can be re-used and resold etc, get an iphone. I used to travel internationally and old iphones just had an INCREDIBLY long value retention period.
The better and longer lasting the basic design is, the more frustrating it is when some fiddly little part makes it inoperable and you can't replace it for no good reason.
The actual story states that something unique in the customer's storage was causing an error on Apple's side, yielding downtime.
I'm not remotely blaming the customer, but what a nothing-burger this submission is: Apple didn't "shut down" his iCloud account, they met an edge in their implementation that they had to remediate. Obviously customer support should be better and convey more information, but I picture a bunch of engineers frantically trying to deal with something they didn't plan for (cue replies claiming that the perfect programmers of HN would never have such an edge, how is it possible, etc) and things getting lost in translation.
I do find the story interesting in that he immediately filed a complaint with the BBB. That seems...risky. It's unfortunate that big corps can do this, but I wouldn't want to be perceived as a "problem" customer doing things like that when big Corps control your online existence.
I think the point is that it doesn't sound like Apple did anything to this person. This person apparently ran into a bug and it took Apple's engineers a few days to get a fix out. I do agree, though, that I wouldn't suggest hosting critical files on a service with no SLA like iCloud.
1 users had serious issue but it was handled like some low priority, it took a long time to fix too so either was a serious issue or it really was a low priority thing
2 rubbing the shitty TOS in the face of an upset customer is a terrible idea, I stopped giving money to companies that id that to me.
3 Apple did not apologize or offer to compensate for the time the user paid for a service that Apple did not delivered, this is typical Apple where only a judge can make them admit blame and do the right/legal thing.
Restating what someone said in a way which is deliberately far more outrageous and then arguing with the restatement is dishonest.
First off, the article headline:
> Apple shut down my iCloud account for five days
You:
> You would not accept to have a non-functional service for a couple of months from anyone else
Is five days acceptable? Probably not, no. Is five days remotely the same as "a couple of months"? No, and you know it, so knock it off.
And what people are pointing out is that even though "shut down my account" and "had a maintenance issue that made my account unavailable" obviously have the same effect -- you can't get to your account, and that's bad -- the two phrases carry very different semantic meaning. If I say "my Twitter account isn't working", what you take away from it is going to be very different than if I say "Twitter locked me out of my account". In this article's case, the headline says one thing and the article says another.
Nobody is saying it's just fine if you can't get to your iCloud account for five days if Apple screws something up. Nobody is defending Apple's bad communication here. Historically, not even the most enthusiastic Mac fans have ever once said "You know what Apple is great at? Cloud services." What we're saying is criticize them for the right thing.
> > I'm not remotely blaming the customer, but what a nothing-burger this submission is: Apple didn't "shut down" his iCloud account, they met an edge in their implementation that they had to remediate.
Apple acknowledged they were not providing this person with the product the person was paying for, yet (presumably), no refund was offered.
> I do find the story interesting in that he immediately filed a complaint with the BBB. That seems...risky.
Why do people think BBB matters? I’ve had customers at my businesses threaten to go to the BBB and it affects business not one bit.
I guess. Though I've never seen HN so invested in someone getting a $1.65 refund. Maybe we can setup a GoFundMe or something.
As to the BBB, personally I find it to be a worse-than-useless corrupt joke. But running to them, forcing Apple to respond, is the kind of thing that just doesn't have a lot of pull against large companies.
It's a generational thing - BBB tends to be respected by people who reached maturity before the internet became big. There was no easy way to complain or vent about business back then, so the BBB actually had clout... With businesses that actually paid to be a member... So, still not that useful, even then.
The BBB never mattered, even if my fellow olde pharts thought it mattered. Note the part where you state that "businesses...paid". What's that phrase folks on HN throw around way too much about who pays, and who the real customer is? BBB was the /dev/null for customer complaints that the businesses paid to cause customers to think they had effective complaint channel.
Source: oldster from back when folks naively thought a complaint to the BBB made any difference, and a former owner of a business that was a member of the BBB.
I think it is far from a "nothing-burger" but a strong warning not to depend on Apple for business critical cloud sevices because they don't care strongly about or take any responsibility for customer downtime.
Issues happen, but it is the response that companies give you that tells you how much they value your business amd how much you should trust them with critical infrastructure.
What makes iCloud a consumer service but iPhones not a consumer product in the same way?
Unless they are, in which case we should get out the megaphones and warn all businesses that those expensive phones they're buying can't be relied upon.
That you can use any one of Box, Google Drive or OneDrive, all of which have enterprise tiers and features for your files answers that question. In term of mail or messaging, Google Workspace and/or Microsoft 365, you knowBoth of which are backed by SLAs. Look at the consumer version, both of which offer paid consumer versions and compare the SLAs...
> I wouldn't want to be perceived as a "problem" customer doing things like that when big Corps control your online existence.
Being a PITA on social media tends to be the only way to reach someone authorized and competent to actually resolve issues.
Sure, try to resolve it through normal support channels first, but if you're given the run-around, you want to be a "problem customer". The squeaky wheel gets the grease.
We had bugs at work, but I assure you we do not take days to fix critical issues like a customer can't login and use the stuff he is paying us for. We would never show you a TOS and say to you "sucker you clicked accept, please wait until someone has the time to fix this minor issue for us but important for you).
In cases where an issue inconvenienced a customer I know we were compensating them with stuff(like upgrades or discounts).
I am just a dev but support has direct access to me, when a customer has a problem that support can't handle the issue is forwarded to me and most of the time I fix the issue.
A small company really needs to try to keep each customer. In the past I worked on even smaller company I was doing support and dev , this forced me to automate support related tasks like gathering reports with detailed information, then as dev I could get user reports and fixed. This would not have worked if the application would have been fulled of bugs but it was done in a managed language so there were few bugs and some of them were not the app but the platform.
I only use my Apple ID for getting apps from the app store. I have disabled all the 'cloud storage' features of iCloud for privacy reasons. iCloud isn't even encrypted[0]. It's a privacy nightmare.
> However, a former Apple employee said it was possible the encryption project was dropped for other reasons, such as concern that more customers would find themselves locked out of their data more often.
When I was reading all of this that was the impression I got, before it was mentioned.
How would device upgrades even work if Apple didn't have the decryption key? Not saying it is impossible, but I'm not aware of how it would work. I guess the user could use their password as entropy to decrypt? That's kind of scary.
They could store the decryption key separately (in a different network/vault) with audit trail. They could still be able to decrypt the docs, but it would be harder to break in. I believe google and amazon do that already.
That actually is what Apple (and pretty much all cloud storage) does. iCloud is built largely on Google’s hardware, hence the encryption. Of course, this doesn’t mean it is end-to-end encrypted as most people infer “encrypted” to mean, otherwise we could call Dropbox and Google Drive encrypted services.
> The subtext is that Apple's iCloud is lacking in some very basic principles of operations discipline that pertain to customer experience.
The subtext is if you are of little to no value to the opposing party, you always have the risk of getting screwed. You will experience this a lot if you try to deal with governments.
iCloud is a thorn in my side. When it comes to downloading my photos on demand, I've found it to be infuriatingly slow. (The high quality versions of photos are stored on iCloud, and only if I want to view, edit, or send one are they then downloaded).
Meanwhile I can stream a 4k YouTube video just fine.
Either sort it out, or give me more local storage space for a price that isn't exorbitant!
I've had a horrific time with Dropbox and photos. Some photos not backing up, losing permissions to my own folders, etc. I've since just spun up a server and use icloudpd to sync the originals nightly, backed up to B2.
Obviously not a scalable solution for many, but I finally feel good about having access to my photos.
> ... I finally feel good about having access to my photos.
I only realised how much joy I get from being able to access my photos when I wasn't able to anymore. Back in ~2012, I lost several years worth when the hard-disk on my Acer laptop failed and I didn't keep any backups. Devastating. Backing up is something I learned the hard way.
Not excusing apple's poor drive performance, but just to point out some of the differences.
Youtube has extensive global caching infra, both on the edge of Google's network and offnet within ISPs networks, to make sure videos are served from a location close to the user.
Also most of the videos people are gonna watch will be popular ones. I.e. they may already be filled in the cache. Compared to your icloud files which may only be accessed by you.
This is exactly why I have syncthing setup to mirror my iCloud Drive folders to my NAS. In the best of times, iCloud is used to seamlessly sync between iDevices. If it fails, I still have a local backup, accessible from iDevices, albeit less convenient.
I've gotten into data hoarding for the same reason. Google photos, youtube, icloud; storing your content on your own hardware is always best for critical features like predictability and access (I don't even want to imagine managing multiple TB through my consumer internet connection). Cloud is great for convenience and redundancy but I consider those secondary features to data preservation.
Follow-up: any recommendations for a beginner NAS? I'd love something cheap and simple; a 2-4 drive system with plug and play functionality would be awesome. I can live without streaming 4k videos or top-of-the-line features like that, though streaming 1080p would be nice.
Not the guy you asked, but for my stuff I’ve just been using icloud-pd and rclone.
I’m really only worried about my photos. Basically all of the photos and videos of my kid only exist on my phone and on iCloud, and even if I trusted Apple to not shut down my account, I’m one account compromise or dummy mistake away from a “delete” being synced and removing that content from all my devices forever.
I just have a script that runs nightly that runs icloud-pd to download all new photos and videos at full quality to the server in my basement. So that gets me a local copy totally disconnected from iCloud. From there, I have rclone encrypt and push up changes to Backblaze B2. Ends up costing me like $1.50/mo.
If you do this, make sure you put some monitoring in place. The 2FA on my Apple account tends to expire every few months and I need to go reauthenticate it.
1. I wouldn't say the problem is well known, but locked out of iCloud is a fairly frequent complain from customers. And of course he didn't find anything on the Apple Support Forum. Apple delete those threads.
2. There were security issues in some cases. But you can tell because Customer Rep would not be showing much empathy, they will tell you they have nothing / update.
3. Engineering on iCloud has been a bag of hurt since Day 1, that is nearly 10 years ago. ( Remember when Steve said why trust them when they brought us MobileMe? ) iCloud Backup corruption is still a thing even though it is extremely rare. I still dont understand how the backup could be corrupted. On the assumption they have multiple copies stored on different hyperscaler. ( I think it is Azure and AWS, not sure if they are on GCP now )
4. TimeCapsule for iOS. For Pete Sake this needs to be a thing.
5. Things like Project McQueen [1] or iCloud team infighting [2]. You could probably smell a thing or two if you were keeping an eye on turnover and hiring with Cloud infrastructure. Although that is often not a reliable indicator.
6. And in 2020, it seems Apple were finally doing something. [3] At least on paper.
I could be wrong, but its something along the lines of...
You have 500gb of files at t0, that gets backed up as Backup1.backup, then you download a 15kb text file and kick off another backup. Instead of completely regenerating a backup of all the data, it will instead recreate a new Backup snapshot, but it will still use that original Backup1.backup file and then have some data structure in place that allows you to just incrementally backup the new data while still preserving the integrity of the old files and the integrity of the previous snapshot.
If your Mac backs up with Time Machine then they literally are. But restoring a phone with the backup takes a long time since there aren't partial restores.
My rule of thumb that I try to get my family and friends to follow: Always control your own data on your own hardware and treat cloud storage as a convenience, not a necessity. And keep backups.
"I'll add that Apple operations engineering appears to function in a way that is more or less divorced from Apple support staff. Support staff appear to have to go begging for a coherent answer from systems engineers, who appear to treat such requests as unimportant. "
This scenario has been my experience for pretty much every hosted solution.
I know Elasticsearch really well. Was working at a company that used the Elastic Cloud (not AWS) version since the demands were not high. There was something very wrong at the host level, which was evident by the cgroups metrics. The support engineers basically used the same endpoints that a customer also has access to to debug problems. No insights at the host at all. Could not ssh to a box. It took a long time to get escalated beyond the support drones. Elastic Cloud was not worth it.
This should be the experience everywhere. Imagine how hard it would be to actually get any work done as a system engineer if you're constantly responding to requests from the front-line support. The reason front-line support exists is to prevent that from happening.
Admittedly things should escalated and fixed but giving answers to end-users is not a priority.
Way back when I was first graduating from college, I interviewed at IBM for a "support engineer" role. It was pitched as a software engineering job, but focused on understanding customers' problems at a pretty deep level, and being the human interface between front-line support and the dev teams.
I always thought it sounded like a good role for a company to have. Unfortunately such people seem to be in short supply.
You build a new data backend for a service, you test it a million ways, and finally you migrate all the user's over.
Except it goes wrong. 999,999,999 user accounts migrated. 1 account failed.
Leaving the old infrastructure up for that one user is costing millions of dollars a day, hurting performance reliability and failover and holding up hundreds of other engineers projects.
Turns out the failure is because the user managed to create a filename with 67000 byte order marks in and nothing else, and it trips up some parser in the new system. They probably did it using a custom client or some kind of exploit checking tool. It's probably some security vulnerability researchers test account, but we can't be sure because we can't look into the account for privacy reasons.
Yet fixing the bug for that 1 user account is going to take a while - a quick bodge solution breaks other user accounts, and the proper fix is part of Jane's project due next quarter.
The decision is made. We're going to treat the needs of the many over the few. We're going to disable this one user account and move on. That user will have to re-upload their data, and this time they won't manage to upload a file with 67000 byte order marks in the filename.
In that case, if your system leaves you with no way to contact that user and say "Hey, there's an issue with your account that is causing a problem that we need your help to solve, get back to us within 3 days or else we're going to need to reset your account", you've failed at customer service.
From the perspective of the customer to Apple - this is the opposite of customer service. It's not my problem if you built some new system that doesn't work with my data; you let me do whatever I wanted and didn't warn me that this wouldn't fly at some far-off point in the distant future. I'm not paying for the privilege to do QA for you. If you're having problems migrating my data you should let me continue using my account until you've fixed it on your end - I shouldn't have to know or be interrupted by an internal business process.
That communication attempt would cause an even bigger shit storm... From a Publicity PoV, just disabling without communication is the best course of action.
> you let me do whatever I wanted and didn't warn me that this wouldn't fly at some far-off point in the distant future
It's not possible to anticipate every need, nor is it practical. You do the best you can to build something useful with the resources you have, ship it, and know that something will go wrong at some point. When something breaks, and it will, you fix it as best you can. If suspending your account for 5 days is the best you can do, it's the best you can do.
Never thought to test a massive string of byte order marks... Extensive testing always misses something.
Testing on real user data could have been done but has privacy issues (now we can't inspect logs to check stuff is working), but even so, the user could have put the offending data in after the testing was done.
All pretty reasonable; The only thing wrong with your suggestion is it's missing the part where you contact that user and let them know what you are doing and why...
VERY FEW (two?) companies in the world have operating expenses of 1B+/year on data infrastructure. I find your story quite hard to believe after reading that.
If your normal infra is 80,000 servers which you own, then running costs are quite low. But during the migration, you are renting in 80,000 servers from AWS too, so the two systems can run in parallel, which have a far higher running cost. As soon as the migration is over, workloads can return to cheap owned hardware.
> What kind of mega-corporation renders your service unavailable and has nothing to say about it?
The kind we have. This sort of thing happens all the time. Large companies -- especially large companies with monopolies -- screwing (at least some of) their customers is very much the rule, not the exception. In fact, I'd be surprised if there were even a single example of a large company that did not have customer service horror stories like this one.
Apple is particularly bad about this because they almost never acknowledge problems of any kind. They just issue regular updates that, if you're lucky, fix more problems than they introduce. Most of the time most of the features work for most of the people. But if you're among the unlucky minority for which something doesn't work then you are well and truly screwed. This is the reason I never use iCloud.
[UPDATE]
This is the nub of the problem:
> USE OF THE APPLE SOFTWARE AND ANY SERVICES PERFORMED BY OR ACCESSED THROUGH THE APPLE SOFTWARE IS AT YOUR SOLE RISK AND THAT THE ENTIRE RISK AS TO SATISFACTORY QUALITY, PERFORMANCE, ACCURACY AND EFFORT IS WITH YOU.
This is in the EULA. Everyone who uses iCloud nominally agrees to this. But of course no one really takes it seriously. Everyone thinks, "Ha ha, Apple is including this meaningless legalese in order to cover their asses, but they're not really going to be so unreliable that they're actually going to have to fall back on this to protect them. They couldn't possibly stay in business if they screw their customers, so they won't.
Except that they can, and, manifestly, they do. The reason they can is that there are no viable alternatives, with Apple and Google (and Microsoft before that) having driven them all from the marketplace. As long as they keep the disgruntled customer rate low enough that it doesn't make the evening news, what happens after that really doesn't matter to them at all. It's the opposite actually: the cost of fixing the marginal problems is more than the marginal revenue it would produce, so taking care of things like this is actually bad business.
None of this will change until people push back en masse and refuse to buy products with terms like this in the EULA.
Good that this gets publicity but even more than anything else it shows how we should move away from centralized services and big corporations for essential services
I had a similar issue with my iCloud account. I wasn’t locked out- but I had a “phantom” iOS device backup file that was taking up several GB worth of iCloud Drive space, but could not be accessed or deleted. This was about five years ago now.
If there’s one thing I hate about Apple, it’s dealing with their support. So I call in, and convince three levels of tech support that, no, I’m not a total moron and I’ve tried deleting the errant file through a variety of different methods on my laptop, iPad, iPhone, etc.
Finally they agree and I get stuck into a queue for an engineer to look at the issue. At this point it actually gets good- honest explanations (without details obviously) about what happened and what they need to do to fix it. Regardless, it took over two months from start to finish for the issue to be fixed.
I did push them for some sort of compensation since I had to upgrade my iCloud storage plan as a result of this file taking up a significant amount of my storage quota. They gave me $50 to spend in the Apple store on accessories. I felt that was fair.
93 comments
[ 4.7 ms ] story [ 180 ms ] threadSeriously, it seems from where I'm sitting if you want a reliable phone, one that holds its value, can be re-used and resold etc, get an iphone. I used to travel internationally and old iphones just had an INCREDIBLY long value retention period.
And AppleCare+ is not cheap.
And I dont see those on the front page of HN either .
best news reader ever
I'm not remotely blaming the customer, but what a nothing-burger this submission is: Apple didn't "shut down" his iCloud account, they met an edge in their implementation that they had to remediate. Obviously customer support should be better and convey more information, but I picture a bunch of engineers frantically trying to deal with something they didn't plan for (cue replies claiming that the perfect programmers of HN would never have such an edge, how is it possible, etc) and things getting lost in translation.
I do find the story interesting in that he immediately filed a complaint with the BBB. That seems...risky. It's unfortunate that big corps can do this, but I wouldn't want to be perceived as a "problem" customer doing things like that when big Corps control your online existence.
You would not accept to have a non-functional service for a couple of months from anyone else either if you'll be honest with it.
1 users had serious issue but it was handled like some low priority, it took a long time to fix too so either was a serious issue or it really was a low priority thing
2 rubbing the shitty TOS in the face of an upset customer is a terrible idea, I stopped giving money to companies that id that to me.
3 Apple did not apologize or offer to compensate for the time the user paid for a service that Apple did not delivered, this is typical Apple where only a judge can make them admit blame and do the right/legal thing.
First off, the article headline:
> Apple shut down my iCloud account for five days
You:
> You would not accept to have a non-functional service for a couple of months from anyone else
Is five days acceptable? Probably not, no. Is five days remotely the same as "a couple of months"? No, and you know it, so knock it off.
And what people are pointing out is that even though "shut down my account" and "had a maintenance issue that made my account unavailable" obviously have the same effect -- you can't get to your account, and that's bad -- the two phrases carry very different semantic meaning. If I say "my Twitter account isn't working", what you take away from it is going to be very different than if I say "Twitter locked me out of my account". In this article's case, the headline says one thing and the article says another.
Nobody is saying it's just fine if you can't get to your iCloud account for five days if Apple screws something up. Nobody is defending Apple's bad communication here. Historically, not even the most enthusiastic Mac fans have ever once said "You know what Apple is great at? Cloud services." What we're saying is criticize them for the right thing.
Apple acknowledged they were not providing this person with the product the person was paying for, yet (presumably), no refund was offered.
> I do find the story interesting in that he immediately filed a complaint with the BBB. That seems...risky.
Why do people think BBB matters? I’ve had customers at my businesses threaten to go to the BBB and it affects business not one bit.
As to the BBB, personally I find it to be a worse-than-useless corrupt joke. But running to them, forcing Apple to respond, is the kind of thing that just doesn't have a lot of pull against large companies.
It's a generational thing - BBB tends to be respected by people who reached maturity before the internet became big. There was no easy way to complain or vent about business back then, so the BBB actually had clout... With businesses that actually paid to be a member... So, still not that useful, even then.
Source: oldster from back when folks naively thought a complaint to the BBB made any difference, and a former owner of a business that was a member of the BBB.
Issues happen, but it is the response that companies give you that tells you how much they value your business amd how much you should trust them with critical infrastructure.
Unless they are, in which case we should get out the megaphones and warn all businesses that those expensive phones they're buying can't be relied upon.
Being a PITA on social media tends to be the only way to reach someone authorized and competent to actually resolve issues.
Sure, try to resolve it through normal support channels first, but if you're given the run-around, you want to be a "problem customer". The squeaky wheel gets the grease.
In cases where an issue inconvenienced a customer I know we were compensating them with stuff(like upgrades or discounts).
A small company really needs to try to keep each customer. In the past I worked on even smaller company I was doing support and dev , this forced me to automate support related tasks like gathering reports with detailed information, then as dev I could get user reports and fixed. This would not have worked if the application would have been fulled of bugs but it was done in a managed language so there were few bugs and some of them were not the app but the platform.
[0] https://www.idownloadblog.com/2020/01/21/reuter-fbi-apple-ic...
> However, a former Apple employee said it was possible the encryption project was dropped for other reasons, such as concern that more customers would find themselves locked out of their data more often.
When I was reading all of this that was the impression I got, before it was mentioned.
How would device upgrades even work if Apple didn't have the decryption key? Not saying it is impossible, but I'm not aware of how it would work. I guess the user could use their password as entropy to decrypt? That's kind of scary.
https://support.apple.com/en-us/HT202303
https://www.naut.ca/blog/tag/shs/
The subtext is if you are of little to no value to the opposing party, you always have the risk of getting screwed. You will experience this a lot if you try to deal with governments.
Google, PayPal, Facebook, do it routinely...
Meanwhile I can stream a 4k YouTube video just fine.
Either sort it out, or give me more local storage space for a price that isn't exorbitant!
Obviously not a scalable solution for many, but I finally feel good about having access to my photos.
I only realised how much joy I get from being able to access my photos when I wasn't able to anymore. Back in ~2012, I lost several years worth when the hard-disk on my Acer laptop failed and I didn't keep any backups. Devastating. Backing up is something I learned the hard way.
Youtube has extensive global caching infra, both on the edge of Google's network and offnet within ISPs networks, to make sure videos are served from a location close to the user.
Also most of the videos people are gonna watch will be popular ones. I.e. they may already be filled in the cache. Compared to your icloud files which may only be accessed by you.
/node_modules strikes again :)
Follow-up: any recommendations for a beginner NAS? I'd love something cheap and simple; a 2-4 drive system with plug and play functionality would be awesome. I can live without streaming 4k videos or top-of-the-line features like that, though streaming 1080p would be nice.
I did a bit of research and wrote it up here:
https://photostructure.com/faq/how-do-i-safely-store-files/#...
I’m really only worried about my photos. Basically all of the photos and videos of my kid only exist on my phone and on iCloud, and even if I trusted Apple to not shut down my account, I’m one account compromise or dummy mistake away from a “delete” being synced and removing that content from all my devices forever.
I just have a script that runs nightly that runs icloud-pd to download all new photos and videos at full quality to the server in my basement. So that gets me a local copy totally disconnected from iCloud. From there, I have rclone encrypt and push up changes to Backblaze B2. Ends up costing me like $1.50/mo.
If you do this, make sure you put some monitoring in place. The 2FA on my Apple account tends to expire every few months and I need to go reauthenticate it.
2. There were security issues in some cases. But you can tell because Customer Rep would not be showing much empathy, they will tell you they have nothing / update.
3. Engineering on iCloud has been a bag of hurt since Day 1, that is nearly 10 years ago. ( Remember when Steve said why trust them when they brought us MobileMe? ) iCloud Backup corruption is still a thing even though it is extremely rare. I still dont understand how the backup could be corrupted. On the assumption they have multiple copies stored on different hyperscaler. ( I think it is Azure and AWS, not sure if they are on GCP now )
4. TimeCapsule for iOS. For Pete Sake this needs to be a thing.
5. Things like Project McQueen [1] or iCloud team infighting [2]. You could probably smell a thing or two if you were keeping an eye on turnover and hiring with Cloud infrastructure. Although that is often not a reliable indicator.
6. And in 2020, it seems Apple were finally doing something. [3] At least on paper.
7. Time to Watch WWDC.
[1] https://venturebeat.com/2016/03/17/apple-cloud-project-mcque...
[2] https://appleinsider.com/articles/16/04/21/icloud-team-repor...
[3] https://www.protocol.com/apple-hires-cloud-open-source-engin...
It will never be, because iCloud is a cash cow.
Unless something changed, they still are.
You have 500gb of files at t0, that gets backed up as Backup1.backup, then you download a 15kb text file and kick off another backup. Instead of completely regenerating a backup of all the data, it will instead recreate a new Backup snapshot, but it will still use that original Backup1.backup file and then have some data structure in place that allows you to just incrementally backup the new data while still preserving the integrity of the old files and the integrity of the previous snapshot.
Does anyone have a TM backup that has not corrupted in over a year of use?
This scenario has been my experience for pretty much every hosted solution.
I know Elasticsearch really well. Was working at a company that used the Elastic Cloud (not AWS) version since the demands were not high. There was something very wrong at the host level, which was evident by the cgroups metrics. The support engineers basically used the same endpoints that a customer also has access to to debug problems. No insights at the host at all. Could not ssh to a box. It took a long time to get escalated beyond the support drones. Elastic Cloud was not worth it.
Admittedly things should escalated and fixed but giving answers to end-users is not a priority.
I always thought it sounded like a good role for a company to have. Unfortunately such people seem to be in short supply.
You build a new data backend for a service, you test it a million ways, and finally you migrate all the user's over.
Except it goes wrong. 999,999,999 user accounts migrated. 1 account failed.
Leaving the old infrastructure up for that one user is costing millions of dollars a day, hurting performance reliability and failover and holding up hundreds of other engineers projects.
Turns out the failure is because the user managed to create a filename with 67000 byte order marks in and nothing else, and it trips up some parser in the new system. They probably did it using a custom client or some kind of exploit checking tool. It's probably some security vulnerability researchers test account, but we can't be sure because we can't look into the account for privacy reasons.
Yet fixing the bug for that 1 user account is going to take a while - a quick bodge solution breaks other user accounts, and the proper fix is part of Jane's project due next quarter.
The decision is made. We're going to treat the needs of the many over the few. We're going to disable this one user account and move on. That user will have to re-upload their data, and this time they won't manage to upload a file with 67000 byte order marks in the filename.
It's not possible to anticipate every need, nor is it practical. You do the best you can to build something useful with the resources you have, ship it, and know that something will go wrong at some point. When something breaks, and it will, you fix it as best you can. If suspending your account for 5 days is the best you can do, it's the best you can do.
He has a right to complain about his horrible experience and the way Apple customer support treated him.
Testing on real user data could have been done but has privacy issues (now we can't inspect logs to check stuff is working), but even so, the user could have put the offending data in after the testing was done.
All pretty reasonable; The only thing wrong with your suggestion is it's missing the part where you contact that user and let them know what you are doing and why...
VERY FEW (two?) companies in the world have operating expenses of 1B+/year on data infrastructure. I find your story quite hard to believe after reading that.
Also, for reference: https://www.ibtimes.com/apple-amazons-biggest-customer-spend...*
The kind we have. This sort of thing happens all the time. Large companies -- especially large companies with monopolies -- screwing (at least some of) their customers is very much the rule, not the exception. In fact, I'd be surprised if there were even a single example of a large company that did not have customer service horror stories like this one.
Apple is particularly bad about this because they almost never acknowledge problems of any kind. They just issue regular updates that, if you're lucky, fix more problems than they introduce. Most of the time most of the features work for most of the people. But if you're among the unlucky minority for which something doesn't work then you are well and truly screwed. This is the reason I never use iCloud.
[UPDATE]
This is the nub of the problem:
> USE OF THE APPLE SOFTWARE AND ANY SERVICES PERFORMED BY OR ACCESSED THROUGH THE APPLE SOFTWARE IS AT YOUR SOLE RISK AND THAT THE ENTIRE RISK AS TO SATISFACTORY QUALITY, PERFORMANCE, ACCURACY AND EFFORT IS WITH YOU.
This is in the EULA. Everyone who uses iCloud nominally agrees to this. But of course no one really takes it seriously. Everyone thinks, "Ha ha, Apple is including this meaningless legalese in order to cover their asses, but they're not really going to be so unreliable that they're actually going to have to fall back on this to protect them. They couldn't possibly stay in business if they screw their customers, so they won't.
Except that they can, and, manifestly, they do. The reason they can is that there are no viable alternatives, with Apple and Google (and Microsoft before that) having driven them all from the marketplace. As long as they keep the disgruntled customer rate low enough that it doesn't make the evening news, what happens after that really doesn't matter to them at all. It's the opposite actually: the cost of fixing the marginal problems is more than the marginal revenue it would produce, so taking care of things like this is actually bad business.
None of this will change until people push back en masse and refuse to buy products with terms like this in the EULA.
If there’s one thing I hate about Apple, it’s dealing with their support. So I call in, and convince three levels of tech support that, no, I’m not a total moron and I’ve tried deleting the errant file through a variety of different methods on my laptop, iPad, iPhone, etc.
Finally they agree and I get stuck into a queue for an engineer to look at the issue. At this point it actually gets good- honest explanations (without details obviously) about what happened and what they need to do to fix it. Regardless, it took over two months from start to finish for the issue to be fixed.
I did push them for some sort of compensation since I had to upgrade my iCloud storage plan as a result of this file taking up a significant amount of my storage quota. They gave me $50 to spend in the Apple store on accessories. I felt that was fair.