This is what I think is the bigger news and deserves far bigger pressure than the downtime. Server downtime can always occur, but then you need to inform your paying customers (such as developers whose apps are counting on your services!) what the issue is. If you are paranoid and don't want to reveal information (for example if it is due to a hack of some kind), you AT LEAST should say something along the lines of "we know there are issues, we are on it."...
But it is absolutely unacceptable and inexcusable to have hours of downtime and have BOTH your end-user and developer status page still be completely green across the board.
Tried to contact Apple via: https://www.apple.com/support/contact/ (As mentioned on the bottom of the page: "If you are experiencing an issue not listed here, contact support"), but that does not seem to work.
I'm not sure if it's as simple as that, certainly OSX has been poorly neglected with 10.10 but generally their cloud services are rock solid (at least in Australia) where Microsofts alternatives suffer from weekly unexplained outages and poor performance on a global level. The number of Amazon instances that crash and fail to ever come back or new instances that start broken is also often dismissed it seems.
Rubbish. I just sold the kids' MacBooks and iPads because iCloud and Pages/Numbers is a piece of shit from a reliability perspective. Add to that the WiFi stopped working reliably with 10.10.
iMessage still suffers from massive consistency failures; due to the incoming messages not being chronologically ordered even small network glitches mean things get confusing fast. iCloud at least in part seems to run on Microsoft Azure, amusingly enough.
I'm not questioning your anecdata, but (to offer some of my own) I have never experienced this and none of the people I work with or socialise with have ever brought it up. I only hear people on the internet talking about it as if it's true of the service as a whole. It definitely isn't.
Perhaps coincidentally I received an unknown charge from iTunes at 1:13am. Since I do not remember making this purchase I would really like to see what was purchased. Unfortunately the store is down.
Actually it's not showing outages. They're showing them as "service issues" (yellow), not "outages" (red). Same as Amazon when their EBS or whatever is on fire they show it as a green check with a small blue "(i)" instead of a proper red "it's down".
Why is so hard for service providers to admit that things are broken? Is it an attempt to weasel out of SLA uptime promises?
edit: Apple just updated the page to show nice big red outages. Kudos! Hopefully they have learned from whatever kept the status page from updating and it keeps working going forward.
It’s weird seeing specific services slowly get the red flag on the status page that have seemingly all gone down at the same time.
I can’t listen to any of my music from iTunes Match, which still shows Green on the status page. But I get an error about the iTunes Store not being able to process purchases (???) when I try to listen to my music, where iTunes Store does show a red flag. So the status page is telling a half-truth. What a mess.
It seems that the "Detailed Timeline" at the bottom shows an outage for "iCloud Account & Sign In" earlier today. That is probably the issue for the original post.
I can't commit code at CloudFlare because we use two-factor auth for the VPN (and everything else) and non-Apple apps on my iPhone are asking for my iTunes password. Tried airplane mode and apps simply don't load at all!
Total app-ocalypse.
Why do apps need me to be authenticated against iTunes to work at all?
It's a "nightmare scenario" downside of the iOS software model. When the device incorrectly decides that you are not authorized to run some binary, there's no way around that.
Curious, which apps seem to be affected? Anecdotally, I hadn't seen any issues on my phone this morning with anything asking for a password, but on my laptop I did see a login prompt when i woke it from sleep.
While this comes too late to help you right now, I recommend looking at running a 2FA app on your laptop or desktop like this - https://github.com/gbraad/gauth so that you are not in this situation again.
Either that or grab a cheap Android handset and use it as a backup. The standard 2FA app on Android needs nothing more than occasional network connectivity to keep the clock in sync. You don't even need a Google account, the app is on FDroid.
Cloudflare is huge and many of us rely on it, so I hope you can easily avoid this predicament in the future - good luck!
>I recommend looking at running a 2FA app on your laptop or desktop
I very strongly recommend against doing this: If you do that, you are giving up a lot of security provided by that second factor as the malware you are using 2FA to protect against now also has access to the keys used to create the 2FA token.
This is a fair point of course, but running it on a second laptop is probably more secure than running it as a mobile app. You wouldn't run it on the same machine you are pushing production code out from, it could be a personal laptop with no access to company systems. I didn't make this point clear in my original comment though.
If you're really desperate: When I last checked, Google Authenticator's keychain entries were not marked "this device only", so they can be extracted from an encrypted backup using something like "iphone-dataprotection" tools.
"DISCLAIMER: This open source project allows you to download the code that powered version 2.21 of the application. Subsequent versions contain Google-specific workflows that are not part of the project."[0] The Play Store version is 2.49[1], but I also don't know what "Google-specific workflows" really entails.
A lesson, perhaps, in why it's a bad idea to lock yourself (and your customers paying you hundreds to thousands of dollars every month) into third-party services (say, Authy) in order to get basic security features for which widely-adopted, standard alternatives exist.
It's about being tied to a third-party service. Currently, the one causing you a problem is Apple's. It could just as well be Authy's.
With standard TOTP, you could pull out a paper backup and use it to make any TOTP app on any platform work. But with Authy, you're screwed if they're down.
And it sounds like you're screwed right now because Apple is down. Again, this wouldn't be an issue if you were using standard TOTP.
Meanwhile, if I'd given in and setup 2FA on my company's CloudFlare account, I wouldn't be able to access it right now.
So yes, I have an axe to grind -- with CloudFlare. We pay you $200/month, and the only reason we can still access our account is because I've refused to setup 2FA on it with Authy.
Meanwhile, if I'd given in and setup 2FA on my company's CloudFlare account, I wouldn't be able to access it right now.
That's not the case. I'm in that position because of the fact that I got my phone into a bad state and can't reauth to iTunes. Others in the office have no problem at all and by talking to the technical support guys I know that I'm in an unusual position otherwise our customers would be very upset.
So yes, I have an axe to grind -- with CloudFlare. We pay you $200/month, and the only reason we can still access our account is because I've refused to setup 2FA on it with Authy.
What exactly is wrong with Authy that makes it completely unacceptable for you to use?
I recently complained about the lack of a Windows Phone client on their Facebook page, and their response was "The service you've requested is currently not on our roadmap".
I had asked the founder about a Windows Phone client a couple of weeks ago and at the time he had said a WP client was 'very likely'. Not sure which is more recent, but I would also like to see an Authy client for Windows Phone.
What exactly is wrong with Authy that makes it completely unacceptable for you to use?
Let's turn this device into a secure token
Enter your Authy cellphone
__+code__ __Authy cellphone number__
They're my private tokens. I don't want to set up an account with Authy Inc and I certainly don't want my tokens in your cloud. Sure maybe it'd be nice to sync across my devices, but not if it looks like it means doing so via somebody else's servers!
You realise that you encrypt the tokens you send and you can't restore them without that password, right? And that backups are opt-in only and you can leave that option disabled?
Do you not trust them to actually encrypt the data before backup? Or is there another issue?
Authy necessarily has the keys on its servers in cleartext. When you integrate them into your application, you send the code the user inputs to Authy's servers for verification.
Authy is a third-party authentication provider, it is not simply a synchronization service.
No, I don't realise any of that, because Let's turn this device into a secure token is the sum total of the information you get when you fire up the Authy app. No links to any explanations of why they want you to have an account, and I didn't care enough to go looking further myself.
But the real issue for me is: given a choice between (1) having my private tokens physically only on my own devices; (2) having an account and apparently some form of my tokens at a third party in another country susceptible to bulk espionage and subpoenas... why on earth would I choose (2) in today's climate?
Or in summary: I guess I don't trust them to actually encrypt anything in the face of legal threats.
> What exactly is wrong with Authy that makes it completely unacceptable for you to use?
This exact scenario is sufficient on its own. Also individually sufficient are the unnecessary revelation of personal information to Authy, and an aversion to perpetuating an authentication method that unsophisticated users can easily confuse with more secure methods that don't rely on third parties.
you do realise that Authy uses standard TOTP? And Google Auth? This is like complaining about Firefox being a "third party app" and if you were just a normal person you would parse the HTML in your brain directly from the output of your ethernet cable or something.
None of these tools have any form of lock-in, but Apple imposes its own layer, in this case.
Authy uses a modified form of TOTP. CloudFlare requiresAuthy, not a standard TOTP implementation. They make this very clear when you try to activate 2FA.
Authy works as a third-party authentication provider. Their servers are in the loop on every login. They aren't just a TOTP app + synchronization, they actually do the code validation themselves.
The correct analogy would be if every time you went to a website in Firefox, it asked Mozilla's servers if it was OK to go there. Also if, when installing Firefox, it demanded you create an account and give your cell phone number to Mozilla. (Or better yet, Google.)
Tbh, to setup sync or install extensions in Chrome, you need a Google Account, for which you need to give them your cellphone number. And it uses a blacklist hosted by Google to check if a page should show a malware warning, or not even display at all.
I just installed AdBlock in Chrome, via the Chrome web store, on a Windows 7 box, that has never had a Google account logged in on it. So at the moment, I see no evidence that you need one to install extensions. Even if you did need one to install from the Chrome Web Store, you can also side-load.
With regard to the others, there are crucial distinctions:
* Chrome synchronization is entirely optional. You can use Chrome without ever logging into Google's servers. That's not the case with Authy.
* The malware blocklist feature uses data stored on your local machine that is frequently updated from Google's servers. It does not send the URLs to Google. Even aside from privacy implications, that would be annoyingly slow. And the block can be bypassed (last time I saw it, anyway) with a single click.
It's because the device has tried to get authorization for running the app (a unique, per UDID signature) and has made the assumption that for whatever reason your access has been revoked. That will normally never happen, it's just an odd side effect of whatever this downtime is. People not seeing this just happen to have hit a local cache of some sort and will eventually get it the longer the downtime goes.
Why do apps need me to be authenticated against iTunes to work at all?
I guarantee that someone would be screaming at Apple for not authenticating against iTunes under circumstance X, Y or Z.
I'm not sure whether it's better to be inconvenienced for a day, or for Apple to attempt to cover your back if something looks amiss in the iPhone's systems.
(which, from the sounds of your upgrade and hard crash, this may well be - or "simply" some form of empty cache)
It feels a bit like six of one, and half a dozen of the other, the painful bit being that ideology doesn't help you get your work done today!
What if you were an app developer who had a paid app on the store? Would you want Apple to sit back while people pirated your app? Would you want to have to implement copy protection yourself? Do you think every app should do that?
iTunes authentication is just a DRM thing, making sure you've paid for apps before you can run them. Security is handled by code signing, and more importantly by sandboxing. The only people who would scream at Apple for not authenticating against iTunes are those who don't understand this, or those who think Apple needs to enforce strong DRM for third-party apps.
You should investigate DuoSecurity, perhaps tied with some Yubikeys - they scale TSV to enterprise scale and have systems for handling scenarios like lost phones.
May be a coincidence, but Duo on my iPhone is having issues. The request to authorize shows up, but then I get 'Unknown Error - The request timed out.'
Using the code does work, so I don't think it's Duo.
I see. So if it tries to run and gets revoked, you're screwed until Apple comes back up. But in theory, if you had been in airplane mode the whole time this incident has been going on, you'd be fine?
same case happened to me once when I'm on a 13hr flight in Jan. Simply all non-stock apps doesn't work anymore, and back to normal when I landed and connected to Internet.
I suppose this is a good example of where Azure MFA can be useful. Rather than having an app generate a code, it can be configured to call your phone instead. I hadn't really cared for this option, but as long as your phone can receive calls, you can still authenticate, so it protects against this sort of situation.
I'm always puzzled whenever I have to enable a Cloudflare script on a site that clearly doesn't have enough volume to need it and would have been more robust and simpler with a traditional data center. It also annoys me to no end that I have to permit JavaScript from random hosts to support a cloud server that should be completely transparent to me.
Everything seems fine here in Melbourne, Australia.
I am disappointed that their status page doesn't reflect the issues that it seems many are facing, I struggle with this a lot with Microsofts cloud offerings (o365 etc...) which go down or are partially unavailable every week - Yes you heard that right - weekly.
The model of trust with vendors, large and small alike has its downsides and the widespread impact of centrally authenticated systems is one of them. Is there an answer? I don't know but it certainly comes down to convenience vs trust in many cases.
This is just another example of Apple's severe structural deficit on their cloud engineering teams. From a user perspective, Apple's backend engineering is in desperate need of a management shake-up.
I don't know about that — I've never personally encountered an issue with Apple's cloud services, despite using them for years. Hell, I'm not even experiencing the problem that apparently hundreds of other people are.
Are there other examples of Apple's cloud failures?
I'm having problems every week with Ulysses' iCloud sync: syncs that won't happen, lost data... I back up my data by hand to text files just to be able to recover. It's insane how much it fails and how little visibility over what's going on it offers.
It's interesting, because I only use iCloud sync heavily with two apps, but I never have issues with with either of them, which leads me to suspect it's partly a development issue. I hesitate to blame the app developer per se though, because it's also Apple's responsibility to ensure that iCloud integration is easy to set up and difficult to get wrong.
iCloud sync (the pre-iCloud-drive incarnation) has been fraught with issues. A lot of folks in the iOS dev community have been griping (and writing really great blog posts) about this for some time now. There are times when iCloud just goes off the rails. For quite a while I'd been one of the "lucky" ones, but since then I've had to jump through workaround hoops to obliterate all of an app's iCloud data to fix syncing errors, seen things like two Macs on the same Apple ID that couldn't see each other's files and so on.
Imagine the furor if you could log into the same account from Dropbox in two different places and not see all of the same files.
AFAICT, the iCloud Drive revisions were essentially an acknowledgement that the first iCloud backend API and implementation was just a great big mess. Seems to be a huge case of overpromise and underdeliver, which sadly seems to be the banner motto of Apple's cloud offerings.
The entire business model of IAAS depends on the idea that the infrastructure is 24/7 accessible. The loss of sales during this time is nothing compared to the loss of confidence in the company will have. Measuring the damage will be an interesting discussion.
My Roku box wouldn't connect to Netflix this morning either. I wonder if there is something else going on here. :/ Maybe there is a problem with AWS or the internets in general.
Glad I saw this—having problem restoring an iPad and I could not for the life of me figure out what was going on. Now, this iPad has to leave the country in about 30 minutes...
Considerably less than ($revenue_per_minute * $downtime) would have you believe because few people are going to attempt to buy an album, have their purchase fail, then never attempt to repurchase it. True, some number will use another online seller, but I suspect they're in a tiny minority.
Kind of odd that they inserted a past outage into the history that never showed up in the status indications at the top of the screen. Makes it clear this is a manually updated dashboard, not based on monitoring.
IIRC that dashboard has been completely wrong several times before on outages - I'm very confident its a manually controlled dashboard that Apple only use to officially acknowledge major outages rather than reflect actual outages. As far as I'm concerned that dashboard has no integrity - like a smoke alarm that goes off an hour too late.
146 comments
[ 2.7 ms ] story [ 166 ms ] thread[Update 15:02 11-03-2015, GMT+1]
* iTunes Store - All users are affected
* iCloud Account & Sign In - All users were affected
* iCloud Mail - All users were affected
But it is absolutely unacceptable and inexcusable to have hours of downtime and have BOTH your end-user and developer status page still be completely green across the board.
Infuriating!
This concur with the #watchFiasco of apple, explicitly saying "you're pigeons, and we deliver"
Rubbish. I just sold the kids' MacBooks and iPads because iCloud and Pages/Numbers is a piece of shit from a reliability perspective. Add to that the WiFi stopped working reliably with 10.10.
Why is so hard for service providers to admit that things are broken? Is it an attempt to weasel out of SLA uptime promises?
edit: Apple just updated the page to show nice big red outages. Kudos! Hopefully they have learned from whatever kept the status page from updating and it keeps working going forward.
I can’t listen to any of my music from iTunes Match, which still shows Green on the status page. But I get an error about the iTunes Store not being able to process purchases (???) when I try to listen to my music, where iTunes Store does show a red flag. So the status page is telling a half-truth. What a mess.
Total app-ocalypse.
Why do apps need me to be authenticated against iTunes to work at all?
Either that or grab a cheap Android handset and use it as a backup. The standard 2FA app on Android needs nothing more than occasional network connectivity to keep the clock in sync. You don't even need a Google account, the app is on FDroid.
Cloudflare is huge and many of us rely on it, so I hope you can easily avoid this predicament in the future - good luck!
I very strongly recommend against doing this: If you do that, you are giving up a lot of security provided by that second factor as the malware you are using 2FA to protect against now also has access to the keys used to create the 2FA token.
If your machine is compromised, it's over.
It can just forward code, relay cookies, etc. 2FA protects against someone peeking at your keyboard, or reused passwords, not malware.
I recently learned that there is an authenticator in f-droid, but not the authenticator, if one reads the notes at the top of the f-droid listing: https://f-droid.org/repository/browse/?fdid=com.google.andro...
I don't even know what they would want to stick in the Play store's authenticator above the already open sourced functionality.
[0] https://github.com/google/google-authenticator-android/wiki [1] https://play.google.com/store/apps/details?id=com.google.and...
Of these I have only tested "Mac App Store" and "Apple Support" and they don't seem to work.
I could log into the Apple Store though.
With standard TOTP, you could pull out a paper backup and use it to make any TOTP app on any platform work. But with Authy, you're screwed if they're down.
And it sounds like you're screwed right now because Apple is down. Again, this wouldn't be an issue if you were using standard TOTP.
Meanwhile, if I'd given in and setup 2FA on my company's CloudFlare account, I wouldn't be able to access it right now.
So yes, I have an axe to grind -- with CloudFlare. We pay you $200/month, and the only reason we can still access our account is because I've refused to setup 2FA on it with Authy.
That's not the case. I'm in that position because of the fact that I got my phone into a bad state and can't reauth to iTunes. Others in the office have no problem at all and by talking to the technical support guys I know that I'm in an unusual position otherwise our customers would be very upset.
So yes, I have an axe to grind -- with CloudFlare. We pay you $200/month, and the only reason we can still access our account is because I've refused to setup 2FA on it with Authy.
What exactly is wrong with Authy that makes it completely unacceptable for you to use?
https://news.ycombinator.com/item?id=9100644
Do you not trust them to actually encrypt the data before backup? Or is there another issue?
Authy is a third-party authentication provider, it is not simply a synchronization service.
But the real issue for me is: given a choice between (1) having my private tokens physically only on my own devices; (2) having an account and apparently some form of my tokens at a third party in another country susceptible to bulk espionage and subpoenas... why on earth would I choose (2) in today's climate?
Or in summary: I guess I don't trust them to actually encrypt anything in the face of legal threats.
This exact scenario is sufficient on its own. Also individually sufficient are the unnecessary revelation of personal information to Authy, and an aversion to perpetuating an authentication method that unsophisticated users can easily confuse with more secure methods that don't rely on third parties.
None of these tools have any form of lock-in, but Apple imposes its own layer, in this case.
Authy works as a third-party authentication provider. Their servers are in the loop on every login. They aren't just a TOTP app + synchronization, they actually do the code validation themselves.
The correct analogy would be if every time you went to a website in Firefox, it asked Mozilla's servers if it was OK to go there. Also if, when installing Firefox, it demanded you create an account and give your cell phone number to Mozilla. (Or better yet, Google.)
So your scenario isn’t even that far off.
With regard to the others, there are crucial distinctions:
* Chrome synchronization is entirely optional. You can use Chrome without ever logging into Google's servers. That's not the case with Authy.
* The malware blocklist feature uses data stored on your local machine that is frequently updated from Google's servers. It does not send the URLs to Google. Even aside from privacy implications, that would be annoyingly slow. And the block can be bypassed (last time I saw it, anyway) with a single click.
I guarantee that someone would be screaming at Apple for not authenticating against iTunes under circumstance X, Y or Z.
I'm not sure whether it's better to be inconvenienced for a day, or for Apple to attempt to cover your back if something looks amiss in the iPhone's systems.
(which, from the sounds of your upgrade and hard crash, this may well be - or "simply" some form of empty cache)
It feels a bit like six of one, and half a dozen of the other, the painful bit being that ideology doesn't help you get your work done today!
Using the code does work, so I don't think it's Duo.
Does this mean you normally can't ever use 3rd-party apps in airplane mode? Because that is definitely not the case for me.
Hopefully these lessons teach us that single points of failure are bad.
It is an interesting variation of the Airplane situation, but no connection is no connection.
The model of trust with vendors, large and small alike has its downsides and the widespread impact of centrally authenticated systems is one of them. Is there an answer? I don't know but it certainly comes down to convenience vs trust in many cases.
Are there other examples of Apple's cloud failures?
Imagine the furor if you could log into the same account from Dropbox in two different places and not see all of the same files.
AFAICT, the iCloud Drive revisions were essentially an acknowledgement that the first iCloud backend API and implementation was just a great big mess. Seems to be a huge case of overpromise and underdeliver, which sadly seems to be the banner motto of Apple's cloud offerings.
Apple rarely has infrastructure issues this large. Mind detailing your "other examples" for evidence?
Thanks apple, for dashing the hopes and dreams of this twenty-something, now bawling their eyes out and screaming for Frozen.
Kind of odd that they inserted a past outage into the history that never showed up in the status indications at the top of the screen. Makes it clear this is a manually updated dashboard, not based on monitoring.