yeh... but, we sorta live and die by the evidence. We can't just blindly assume OP is correct. Thankfully, OP knew that and saved us the hassle or typing it ourselves.
Wasn't there that time AWS had some outage, but the red-circle picture (for failure) was itself hosted on the unavailable AWS service, so their status page didn't show it as being unavailable? Or am I mis-remembering something/repeating an urban legend?
Also, accurately reporting about an arbitrary source of downtime means you're smart enough to avoid the same sources of downtime.
Not that this can't have been an obvious reason (deleting all the servers in a datacenter or similarly trivial but severe) but it's likely impossible to ensure status page accuracy.
Most outages aren't so obvious as this one, and any ping will fail intermittently (often because the ping agent has a failure.) Google definitely has loads of pings hitting Google Calendar in various ways. Exposing this monitoring to the public is not practical or really useful. (And would aid would-be attackers.)
That only indicate the frontend of the service is up and potentially running. Being about to respond to ping and being able to serve HTTP request are two different things, and being able to serve HTTP request vs a fully functional website are two different things. Think about wrong SSL certification, wrong domain mapping between frontend/backend, broken JS/CSS etc.
> We're investigating reports of an issue with Google Calendar. We will provide more information shortly. The affected users are unable to access Google Calendar.
"We're investigating reports of an issue with Google Calendar. We will provide more information shortly. The affected users are unable to access Google Calendar."
Nothing but anecdote, but a lot of my friends at Facebook and Google are eyeing the exits.
They’ve made good money and can afford to go somewhere better aligned with their values. They’re also each remarkably talented.
These outages may be a reflection of that exodus. (Counterfactual: we started our careers at the same time and are nearing a natural switching point simultaneously.)
How do you think it relates to the failures exactly? Employee quality is down? The servers had someone manually monitoring them and they quit so now the servers are quietly on fire? What's the theory?
Hint: companies at Google's scale do not have a single point of failure where employees slowly trickling in or out can impact their infrastructure in this way. You would see many more failures in this case. The tenure for the average employee is ~2 years.
I've worked at big companies and small ones. Every place has a small pool of extraordinary technical talent (the 10x or 100x engineers or whatever). It isn't just that these people are geniuses (although some of them def are), its how much context they have around the systems that are critical to the functioning of the Company. They have that context + dedication to have learned about different failure scenarios. They probably have built the automation systems that deploy the services.
When such people leave, its not the end of the company, someone else (either a person or a group) usually are interested and step up to knowledge transfer before the person leaves and then learn the system.
However, if a critical mass of people leave at or around the same time, crucial knowledge that is necessary for the systems to operate correctly is lost. This may not surface immediately, but when something goes wrong, you will notice it.
I'm not saying this is what happened to Google, IDK. But its very much a possiblity, even at the largest companies. Especially the ones that have somewhat centralized systems, so outages tend to affect a whole bunch of seemingly unrelated services.
What's going on with google lately? First the google cloud outage earlier this month and now this? I wonder if there's some sort of systemic problem they're dealing with
Now I'm curious when the last time Google search went down? they must lose a lot of money everytime that happens. or is that one of those services where they spend ridiculous amounts of money and talent on making sure it never goes down?
I'm guessing that Google search is easier to distribute over a larger set of machines, since the data it uses is static to a large degree and less bound to specific users.
Search also has the possibility of degrading nicely, unlike a calendar. You could store the top million sites in some backup service and drop down to string matching if you had to. Searching for “how to train a hunting dog”? Here’s the Wikipedia page for Dog. Best of luck.
It's that second thing. They spend a lot and have a lot of talent behind keeping search up. Also it's an easier problem because it isn't transactional. They can fall back to generic search results if they can't load your custom data, and you'd probably never notice.
Google calendar sort of requires your personal data to function (just like gmail).
Also I would like to point out an interesting side effect that the #1 result in google for "Google calendar 404" is down with a "Too many requests" error :D
I use it with Thunderbird, and DavDroid (now DavX) to handle all my events and contacts (SMS contacts even show up correctly in Hangouts on Android phones)
I realize GCal is rarely ever down, but they do harvest all your data to sell you shit.
Here's a basic Dockerfile for Radicale if you want to try it out:
Companies pay €€€ to use it for business. Free version was good introduction.
One reason for using Gmail (which is generating ad money) is the integration with calendar. Getting travel bookings automatically in calendar is useful.
It's part of the google eco system. It's integrated with android, google assistant, gmail, ... Chromebooks need web-only replacements for Outlook and Office.
It's also a freemium product, because you can pay for gSuite.
Get people to stay logged in to Google. Then when you do your regular google.com searches, Google knows exactly who you are rather than trying to figure it out for an "anonymous" searcher. I'd guess that alone is worth the cost of operating Gmail for free for everyone.
Has Google made any public statements about whether or not calendar data is used for ad targeting? Gmail was used for ad targeting until mid-2017[1] but I haven't seen anything about calendar.
Afaik, it was never used for ad targeting so there was no announcement to make. Just think about it for minute.
What percentage of people actively use Google Calendar? Out of those what percentage put useful information in there? How much commercial value does this information have?
I generally expect Google to be using all possible data sources they have access to for ad targeting without explicit statements otherwise like Gmail has received.
You work for Google and are apparently "pretty sure" that Calendar isn't used for ad targeting but don't actually know, how is a consumer supposed to know this stuff?
But... why wouldn't they? Google already is heavily invested in doing this kind of parsing because a lot of their calendar entries are integrated with AI that figures out when you need to leave your house, how to add entries dictated over voice, how to auto-add new appointments based on emails without duplicating entries, and so on.
So Google already needs to know how to parse your calendar data in intelligent ways. And the way that data would be integrated into ads is not going to be all that different from the way Gmail data was processed.
Who cares if it's a small fraction of users? It's basically free data, being consumed by technologies that Google already has to build anyway -- and for the few users that do heavily use calendar, you're getting a ton of data on their everyday schedule.
Why build a set of tools that can do all of this parsing and then say, "nah, we're not going to deploy it everywhere."?
> What percentage of people actively use Google Calendar?
I don't know. My hunch is that it's not a fringe product though.
It's been a while since I checked, but I remember either Calendar or Keep being the default calendar app on Android. What percentage of Android owners use the built-in calendar on their phone to set reminders?
You have access to their appointment graph from all the calendar invitations sent, especially with attendees list. You have access to meetings I am invited to even though I don't have a google account since my email address appears in invitations.
About 2yrs ago I booked a flight to my relatives for a Christmas trip and pasted the info into my Google calendar. Within hours, I got google calendar notices about not having confirmed my hotel reservation. It was VERY annoying as it was borderline deceptive, presented as if I'd already gone through the whole reservation process but failed to confirm.
It was definitely a Google notice, and a Google confirmation process when I clicked through to confirm if I wanted this service. I shut it off as best I could find and told them I wanted no part of any such deceptive marketing, even though I would perhaps have wanted a service that made it clear that I was being offered an option, not a fake reminder.
I saw nothing of it since, but then I'd shut it off, so I should have no data. So, it seems that Google has siloed projects that not all employees know about...
Seriously, multiple downvotes for an accurate recounting of an observed event contradicting a parent assertion, yet no one provides any new info? I don't expect any upvotes, but what am I missing here? Is there something against posting actual data and events?
What you're describing isn't an advertisement, so isn't at all relevant to the discussion.
It's well known (and not at all secret) that google will grab upcoming flight info from emails and put it into your calendar. But this info still isn't used for ad targeting.
However, I was describing the deceptive advertisement that came along with the flight->calendar populating feature (which is fine).
The advertisement that Google added to my calendar was an auto-populated fake hotel reservation.
It was setup to look as if I'd previously reserved rooms but had simply failed to confirm the final ccard info -- the click-thru literally went straight to the enter ccard info page with my "reservation" prepopulated.
This was trying to hijack any other reservation that I might have intended to make (of course it was really obvious to me since we were staying w/relatives).
Let's be clear:
Adding a "need a hotel?" link to my calendar might be ok.
Populating my calendar with an item for a business I've never even contacted is an advertisement.
Falsely claiming I've already made a reservation when I've never even contacted the business is a lie. It is a deceptive practice.
I was genuinely surprised to see this level of both advertising and deception from Google.
What you're describing is a bug. It may be due to something you did and didn't realize, it may be due to something the site you were reserving things on did and didn't realize, it may be due to a bug in how Calendar parses reservation emails. But its a bug somewhere along the process. It's not an intentional advertisement.
So you're being downvoted for claiming to have been the target of an unrealistic sounding advertisement campaign that, it seems, no one else has ever been targeted by. This, at Google scale, is pretty unlikely.
To give a recent similar example from HN, a user was complaining that google was incorrectly tracking their watch history on YouTube. In reality, they had been infected with malware that watched YouTube videos in the background. Your story gives me the same vibe, and that's likely why people are downvoting.
I had never made a reservation, never even searched for a hotel, as we were staying with relatives. So, there is zero chance that it had anything to do with other searches or reservation attempts, and the flights were made directly at the airline site (i.e., not through Google Flights or a travel aggregator).
The advert populated in the same time-frame as the flight; the airline's flight confirmation email to my Gmail acct had clearly been was parsed, and there was nothing in the email having to do with a hotel.
Moreover, when I clicked thru and found the menus, Google settings pages were there, offering to enable/disable the feature, along with describing that the "helpful" auxiliary reservation feature could not be turned off separately, and providing a feedback opportunity.
My recollection is that this advertisement itself also came with a google message about how this was a helpful new feature to complete my travel palns.
Obviously I turned it off, and slammed them in the feedback, in particular how I couldn't manage the sub-features (actual vs advertised reservations) separately.
Neither item ever happened again, which I would expect having turned off the feature. Which also indicates that this was something under Google's control as opposed to an infection or bug.
My guess is that this is a short-lived experiment, gathered a lot of the kind of feedback I gave, and was cut off by one of their smarter managers.
1) This occurred in exactly the same set of minutes since the email direct from the airline arrived and Google parsed it and posted the flight info to my calendar
2) The hotel locations and dates for the fake reservation were exactly matched to the airline travel dates.
It is also possible to win the lottery, but to post such an exactly targeted advertisement is for all practical purposes impossible without access to the email. Only three parties had access to it: The airline, me, and Google.
It is plausible, but I do have scripting & images shut off in my email client, use NoScript in the browser to prevent unknown scripts from executing (where I do sometimes read Gmail), and saw no such link. If Google is executing them on the server side, I've lost control.
The other indication against this is that this only appeared at the same time as the auto-populate for flights appeared, and never after I turned off that service. Yet, I've received many flight confirmations from the same airline.
So, it would have had to have been a short-lived shady link deal between the hotel & airline at exactly the time that Google started auto-populating the flights, then dropped. Possible, and important to consider, but...
I used to work at Google. Its a big company, composed of almost 10^5 employees today. In my experience, they can be trusted to adhere to the letter of their agreements; no more and no less. That's not as bad as it might sound. Plenty of companies consider their contracts to be far more mutable than Google did while I was there.
But it does mean that you can't take any individual employee's word about what they are doing today. Individuals change. Managers change. People come and go. A fundamental part of the company mythos is that you as an individual know more about what goes on in the company than you really do. So unless you are acting in an official capacity to speak for the gmail and calendar teams, you should hush.
If the letter of their agreements permits GOOG to use your data for <X> purpose, then as a user of the service you should assume that they are.
Thank you for the citation. Nevertheless, I hope that you can clearly see the difference between referring folks to an official company communication, and starting off with "I work at google..." IIRC, there's mandatory annual training about just this sort of thing.
This linked post doesn't say anything about Google Calendar.
It's not necessarily that I doubt you, it's just that the press release is pretty specific that consumer GMail isn't going to be used for ad customization from now on, and it seems like if it was everything in G-Suite that would be mentioned somewhere.
I don't suppose there's a list Google maintains anywhere online that shows which of their own products they use to aid with ad targeting?
That's exactly what makes me nervous. Even someone working at Google isn't entirely sure what user data is or isn't sent to other parts of the company for other uses.
Plenty of other tech companies have lied, changed their policies, or just plain screwed up. Privacy and security are hard, and everybody has their hands out. As a consumer, there's no way for me to verify anything.
As Woz said (sort of), "Never trust a computer you can't throw out a window." When I make a backup to an external FW disk on my desk, or sync my phone with my computer over a USB cable, I know exactly what data is going where. The whole point of The Cloud being a cloud is that we don't know what's happening in it. It's inherent to the architecture. "Trust but verify" minus the "verify".
I swapped out Horde for Roundcube as my default for customers (who didn't want/need groupware, those are M$ and Zimbra), much easier on the eyes. Just as functional IMO.
There is also Nextcloud which provides for most self-hosting needs in a single package (calendar, tasks, file storage, notes, music, photos, device tracking, etc.)
I'll add that Microsoft Outlook calendar works really well. or $69/year, Microsoft will sell you an Office 365 account, Outlook email and calendar, all their apps on-line, two desktop-installable licenses for their Office Suite, and 1 TB of OneDrive storage. It's a great deal.
While self-hosting is a good start, I believe that everyone should be end-to-end encrypting their data. Servers can be hacked and snooped on, and I cringe at the thought of leaving my personal information in the clear on a server, even if it's mine.
That's why I created EteSync[1] a secure, end-to-end encrypted, and privacy respecting sync solution that seamlessly integrates with existing apps and feels just like a Google account.
190 comments
[ 2.9 ms ] story [ 207 ms ] threadI'm sure they are fine, this product is not a personal page with guestbook for your 20 friends and gifs on shared hosting plan.
Here's their tweet about it: https://twitter.com/awscloud/status/836656664635846656
https://www.theregister.co.uk/2017/03/01/aws_s3_outage/
https://news.ycombinator.com/item?id=13756111
> ©2019 Google - Last updated: June 18, 2019 at 3:12:13 PM UTC+1
Looks like it was last updated before the outage.
And I'm not surprised, since actually reporting it as down has a lot of political blowback (not to mention contract blowback) within the company.
Not that this can't have been an obvious reason (deleting all the servers in a datacenter or similarly trivial but severe) but it's likely impossible to ensure status page accuracy.
And yet everybody agrees it's down.
https://www.google.com/appsstatus#hl=en&v=issue&sid=2&iid=cc...
> We're investigating reports of an issue with Google Calendar. We will provide more information shortly. The affected users are unable to access Google Calendar.
Previous one was only 15 days ago: https://news.ycombinator.com/item?id=20077421
My opinion is worth nothing but Google feels like a crumbling cookie now.. it used to be a cool addition to one's life.
Nothing but anecdote, but a lot of my friends at Facebook and Google are eyeing the exits.
They’ve made good money and can afford to go somewhere better aligned with their values. They’re also each remarkably talented.
These outages may be a reflection of that exodus. (Counterfactual: we started our careers at the same time and are nearing a natural switching point simultaneously.)
Hint: companies at Google's scale do not have a single point of failure where employees slowly trickling in or out can impact their infrastructure in this way. You would see many more failures in this case. The tenure for the average employee is ~2 years.
I've worked at big companies and small ones. Every place has a small pool of extraordinary technical talent (the 10x or 100x engineers or whatever). It isn't just that these people are geniuses (although some of them def are), its how much context they have around the systems that are critical to the functioning of the Company. They have that context + dedication to have learned about different failure scenarios. They probably have built the automation systems that deploy the services.
When such people leave, its not the end of the company, someone else (either a person or a group) usually are interested and step up to knowledge transfer before the person leaves and then learn the system.
However, if a critical mass of people leave at or around the same time, crucial knowledge that is necessary for the systems to operate correctly is lost. This may not surface immediately, but when something goes wrong, you will notice it.
I'm not saying this is what happened to Google, IDK. But its very much a possiblity, even at the largest companies. Especially the ones that have somewhat centralized systems, so outages tend to affect a whole bunch of seemingly unrelated services.
We worked hard and managed to keep it pretty high, but in all that time I think we only had two or three perfect weeks worldwide.
Google calendar sort of requires your personal data to function (just like gmail).
https://radicale.org/
I use it with Thunderbird, and DavDroid (now DavX) to handle all my events and contacts (SMS contacts even show up correctly in Hangouts on Android phones)
I realize GCal is rarely ever down, but they do harvest all your data to sell you shit.
Here's a basic Dockerfile for Radicale if you want to try it out:
https://github.com/sumdog/bee2/tree/master/dockerfiles/Radic...
One reason for using Gmail (which is generating ad money) is the integration with calendar. Getting travel bookings automatically in calendar is useful.
It's also a freemium product, because you can pay for gSuite.
[1] https://blog.google/products/gmail/g-suite-gains-traction-in...
What percentage of people actively use Google Calendar? Out of those what percentage put useful information in there? How much commercial value does this information have?
Enough to make it worth continuing to operate the service, clearly. Do you really want to go down that road?
If I put "Go fishing" into my calendar for next Monday, Google knows:
1) This person fishes.
2) This person probably buys fishing related products.
3) We know this person lives in X and fishes there, and business Y has a website selling similar products.
4) This person is going fishing on a Monday afternoon (for example) so they are probably off work on Mondays or afternoons.
Let's say, I also put in meetings. "Meet with Joe about building a website for his company."
1) This person builds websites
2) Uses hosting services
3) Buys domains
4) Probably codes
5) Must need project management software
6) And all other things typically related to people in this field...
Knowing someone's life this intimately is incredibly valuable.
You work for Google and are apparently "pretty sure" that Calendar isn't used for ad targeting but don't actually know, how is a consumer supposed to know this stuff?
But... why wouldn't they? Google already is heavily invested in doing this kind of parsing because a lot of their calendar entries are integrated with AI that figures out when you need to leave your house, how to add entries dictated over voice, how to auto-add new appointments based on emails without duplicating entries, and so on.
So Google already needs to know how to parse your calendar data in intelligent ways. And the way that data would be integrated into ads is not going to be all that different from the way Gmail data was processed.
Who cares if it's a small fraction of users? It's basically free data, being consumed by technologies that Google already has to build anyway -- and for the few users that do heavily use calendar, you're getting a ton of data on their everyday schedule.
Why build a set of tools that can do all of this parsing and then say, "nah, we're not going to deploy it everywhere."?
> What percentage of people actively use Google Calendar?
I don't know. My hunch is that it's not a fringe product though.
It's been a while since I checked, but I remember either Calendar or Keep being the default calendar app on Android. What percentage of Android owners use the built-in calendar on their phone to set reminders?
I want to believe you, I like the story you are telling. Unfortunately you like it too and are not being objective.
About 2yrs ago I booked a flight to my relatives for a Christmas trip and pasted the info into my Google calendar. Within hours, I got google calendar notices about not having confirmed my hotel reservation. It was VERY annoying as it was borderline deceptive, presented as if I'd already gone through the whole reservation process but failed to confirm.
It was definitely a Google notice, and a Google confirmation process when I clicked through to confirm if I wanted this service. I shut it off as best I could find and told them I wanted no part of any such deceptive marketing, even though I would perhaps have wanted a service that made it clear that I was being offered an option, not a fake reminder.
I saw nothing of it since, but then I'd shut it off, so I should have no data. So, it seems that Google has siloed projects that not all employees know about...
It's well known (and not at all secret) that google will grab upcoming flight info from emails and put it into your calendar. But this info still isn't used for ad targeting.
However, I was describing the deceptive advertisement that came along with the flight->calendar populating feature (which is fine).
The advertisement that Google added to my calendar was an auto-populated fake hotel reservation.
It was setup to look as if I'd previously reserved rooms but had simply failed to confirm the final ccard info -- the click-thru literally went straight to the enter ccard info page with my "reservation" prepopulated.
This was trying to hijack any other reservation that I might have intended to make (of course it was really obvious to me since we were staying w/relatives).
Let's be clear:
Adding a "need a hotel?" link to my calendar might be ok.
Populating my calendar with an item for a business I've never even contacted is an advertisement.
Falsely claiming I've already made a reservation when I've never even contacted the business is a lie. It is a deceptive practice.
I was genuinely surprised to see this level of both advertising and deception from Google.
So you're being downvoted for claiming to have been the target of an unrealistic sounding advertisement campaign that, it seems, no one else has ever been targeted by. This, at Google scale, is pretty unlikely.
To give a recent similar example from HN, a user was complaining that google was incorrectly tracking their watch history on YouTube. In reality, they had been infected with malware that watched YouTube videos in the background. Your story gives me the same vibe, and that's likely why people are downvoting.
I had never made a reservation, never even searched for a hotel, as we were staying with relatives. So, there is zero chance that it had anything to do with other searches or reservation attempts, and the flights were made directly at the airline site (i.e., not through Google Flights or a travel aggregator).
The advert populated in the same time-frame as the flight; the airline's flight confirmation email to my Gmail acct had clearly been was parsed, and there was nothing in the email having to do with a hotel.
Moreover, when I clicked thru and found the menus, Google settings pages were there, offering to enable/disable the feature, along with describing that the "helpful" auxiliary reservation feature could not be turned off separately, and providing a feedback opportunity.
My recollection is that this advertisement itself also came with a google message about how this was a helpful new feature to complete my travel palns.
Obviously I turned it off, and slammed them in the feedback, in particular how I couldn't manage the sub-features (actual vs advertised reservations) separately.
Neither item ever happened again, which I would expect having turned off the feature. Which also indicates that this was something under Google's control as opposed to an infection or bug.
My guess is that this is a short-lived experiment, gathered a lot of the kind of feedback I gave, and was cut off by one of their smarter managers.
https://www.wired.com/story/phishing-links-google-calendar-i...
Similar attacks have been directed at iCloud users, too.
1) This occurred in exactly the same set of minutes since the email direct from the airline arrived and Google parsed it and posted the flight info to my calendar
2) The hotel locations and dates for the fake reservation were exactly matched to the airline travel dates.
It is also possible to win the lottery, but to post such an exactly targeted advertisement is for all practical purposes impossible without access to the email. Only three parties had access to it: The airline, me, and Google.
https://support.google.com/calendar/forum/AAAAd3GaXpEb0_0F6e...
The other indication against this is that this only appeared at the same time as the auto-populate for flights appeared, and never after I turned off that service. Yet, I've received many flight confirmations from the same airline.
So, it would have had to have been a short-lived shady link deal between the hotel & airline at exactly the time that Google started auto-populating the flights, then dropped. Possible, and important to consider, but...
But it does mean that you can't take any individual employee's word about what they are doing today. Individuals change. Managers change. People come and go. A fundamental part of the company mythos is that you as an individual know more about what goes on in the company than you really do. So unless you are acting in an official capacity to speak for the gmail and calendar teams, you should hush.
If the letter of their agreements permits GOOG to use your data for <X> purpose, then as a user of the service you should assume that they are.
https://blog.google/products/gmail/g-suite-gains-traction-in...
It's not necessarily that I doubt you, it's just that the press release is pretty specific that consumer GMail isn't going to be used for ad customization from now on, and it seems like if it was everything in G-Suite that would be mentioned somewhere.
I don't suppose there's a list Google maintains anywhere online that shows which of their own products they use to aid with ad targeting?
It's so gross.
Plenty of other tech companies have lied, changed their policies, or just plain screwed up. Privacy and security are hard, and everybody has their hands out. As a consumer, there's no way for me to verify anything.
As Woz said (sort of), "Never trust a computer you can't throw out a window." When I make a backup to an external FW disk on my desk, or sync my phone with my computer over a USB cable, I know exactly what data is going where. The whole point of The Cloud being a cloud is that we don't know what's happening in it. It's inherent to the architecture. "Trust but verify" minus the "verify".
I synchronize my information between Thunderbird (Windows), Android (DavX and GMail app) and of course, they provide a nice web interface (Horde)
Beauty is in the eye of the beholder :)
https://www.horde.org/apps/webmail/screenshots
That's why I created EteSync[1] a secure, end-to-end encrypted, and privacy respecting sync solution that seamlessly integrates with existing apps and feels just like a Google account.
1: https://www.etesync.com