But the emoji search addin feature will be developed by a team in mountain view... So if emergency code changes need to be made for this bug HN has just found, they will still be woken up.
Any major product at Google has an on-call team with offices in multiple time zones. Typically, two engineers from different time zones will be on-call at any given time—they switch back and forth between primary and secondary, so nobody is primary on-call during the middle of the night, local time.
For example, you might have an SRE team in Mountain View, and an SRE team in Dublin. Maybe the engineer in Mountain View is primary on-call until midnight, and then it switches to the engineer in Dublin, who starts their shift at 8:00 AM.
If it’s getting code changes, it’s not getting code changes right now. The software engineers may be in Mountain View, but they won’t patch this and push out a new version during the night. Someone (read: an SRE) may change a configuration or push out a temporary fix now, and any code changes will only go out after significant testing. Generally speaking, you don’t hotfix high-profile production services. You rollback, you set up filters, you disable features, you turn off component services, you run in a degraded capacity—but if you want to actually change the code, you take your time and do it right. Rushing out a code change in the middle of the night is liable to make things worse, and nobody wants to do it.
For the right definition of "major". Search, Ads, GMail, Docs, absolutely. If you only have 50 million DAU then you may not even have an SRE let alone one in multiple time-zones. Most teams have an on-call rotation and you get a phone-call an hour after you fall asleep unless you recently did a push or go to bed early. (Or you just forget all the pages that happened during normal hours and only remember the annoying ones.)
Possibly, but probably not. Google tends to have follow-the-sun rotations staffed by SRE as the first line of defense, with an escalation path to devs if necessary.
SREs may not have intricate knowledge of the code but they have an array of tools that can mitigate problems, and they're software engineers too, so they can debug this themselves if need be. Their focus however won't be on fixing the bug, it will be on stopping the bleeding. Typically they'll check if this is an issue introduced in a recent rollout and roll that back. In the case of Search they also have an array of tactical tools - someone mentioned that a specific website was causing this, they probably have a way to quickly and temporarily delist this specific result.
The focus will be on recovering ASAP, figuring out the details and the long-term fix later. That later can be during business hours on Monday.
That depends, I don't think that there needs to be anything emoji specific (at the character level). It could be a special handler for inline results, marketing filters, etc that's gated on some variation of that string.
Might be worth seeing if there's another "how many X on iOS" that faults as well, because I would be that it reports an ISE on timeouts, and you could easily imagine some service making a follow on request that now times out.
It really depends on what is happening behind the scenes. Is it killing an entire server process or just an error in a single thread? Even if it kills an entire server process, Google has lots of these running and the ratio may not be high enough to page.
Why do you think so? Lots of important systems are developed in European offices at least, I wouldn't be surprised if some are in Asia as well. It isn't like Google search is only in MTV.
It's likely that there is some tool for blocking or rewriting certisn queries or results that trigger errors so that the code changes can be made with less rush, more testing and during business hours.
No. That is insane to even think about. Google practices blameless postmortem processes that focuses on avoiding repeat incidents and fixing processes. This is not just something we write down in books but something we do on a daily basis and believe in it.
The only way I can think of someone being punished is, of course, it was maliciously done.
Disclaimer: I am a Google SRE, opinions my own, not an official comment.
Hilariously, the third result on Kagi is a direct link to the google search that crashes, I wonder where it pulled this from: https://imgur.com/a/sCmeADy
Server Error
We're sorry but it appears that there has been an internal server error while processing your request. Our engineers have been notified and are working to resolve the issue.
Taking a while to load is probably indicative of the problem.... There's probably some O(n^3) memory or compute complexity going on somewhere of the number of emojis on the page.
For example, I imagine Google uses as a final ranking step some stuff based on the similarity of the pages about to be returned - to make sure you aren't about to show the user two pages that are practically identical. That logic might try to build similarity mappings between the pages, and has logic that fails badly for large numbers of emoji.
I would guess it's the time that is the problem - whatever service would produce an inline answer is presumably falling over, and the subsequent timeout is reported as an error.
My guess is this is related to instant answers section at the top of the search results. It's trying to figure out the count, but it's failing for one reason or another.
> We're sorry but it appears that there has been an internal server error while processing your request. Our engineers have been notified and are working to resolve the issue.
Would be definitely interesting to read a report on what triggers this error! I don’t think it is a tile error as presumably the backend that collects all the answers would be set up in a way that is resilient to upstream errors. So maybe the core results compiler?
I'm not sure exactly how it works but if you look at the network requests for "how many emojis ios" vs "how many emojis android" the html for the initial page contains an internal server error whereas the android version contains some more javascript which contains the search results.
The likeliest candidate is probably the final render step. Under the hood, search farms the query out to multiple backends, gathers partials from them, and then composites those partials into the final rendered page. For an actual 500 to surface to the user, that final compositing step is probably what fails; gathering and compositing are already robust against an individual backend failing to return a result.
I'm on board with other people's suggestions that asking a question like this is tripping over a site that surfaced an emoji in their site title that is choking the Unicode library that Google is relying upon.
Server Error
We're sorry but it appears that there has been an internal server error while processing your request. Our engineers have been notified and are working to resolve the issue.
Please try again later.
My guess is that it will be a problem with handling Unicode in some way. There will be a page on emojipedia with some Unicode emojis in the title or description that Google can’t handle while trying to extract the text to display a result.
158 comments
[ 48.3 ms ] story [ 3872 ms ] threadFor example, you might have an SRE team in Mountain View, and an SRE team in Dublin. Maybe the engineer in Mountain View is primary on-call until midnight, and then it switches to the engineer in Dublin, who starts their shift at 8:00 AM.
If it’s getting code changes, it’s not getting code changes right now. The software engineers may be in Mountain View, but they won’t patch this and push out a new version during the night. Someone (read: an SRE) may change a configuration or push out a temporary fix now, and any code changes will only go out after significant testing. Generally speaking, you don’t hotfix high-profile production services. You rollback, you set up filters, you disable features, you turn off component services, you run in a degraded capacity—but if you want to actually change the code, you take your time and do it right. Rushing out a code change in the middle of the night is liable to make things worse, and nobody wants to do it.
SREs may not have intricate knowledge of the code but they have an array of tools that can mitigate problems, and they're software engineers too, so they can debug this themselves if need be. Their focus however won't be on fixing the bug, it will be on stopping the bleeding. Typically they'll check if this is an issue introduced in a recent rollout and roll that back. In the case of Search they also have an array of tactical tools - someone mentioned that a specific website was causing this, they probably have a way to quickly and temporarily delist this specific result.
The focus will be on recovering ASAP, figuring out the details and the long-term fix later. That later can be during business hours on Monday.
Might be worth seeing if there's another "how many X on iOS" that faults as well, because I would be that it reports an ISE on timeouts, and you could easily imagine some service making a follow on request that now times out.
The only way I can think of someone being punished is, of course, it was maliciously done.
Disclaimer: I am a Google SRE, opinions my own, not an official comment.
The third sesrch result attempts to answer the question. The 1st and 2nd are related but not perfectly relevant.
the emoji list is the #1 match on brave
Hilariously, the third result on Kagi is a direct link to the google search that crashes, I wonder where it pulled this from: https://imgur.com/a/sCmeADy
I wonder if processing emoji characters has any different performance characteristics than other UTF characters.
https://www.google.com/search?q=how+many+emojis+on+ios&oq=ho...
Please try again later.
Maybe the quantity question is running into an error trying to make one of those cards? Incredible that it just crashes search though.
Instant answer crawler tries to answer that question by parsing some site, fails in an unexpected way.
That'd be my theory.
For example, I imagine Google uses as a final ranking step some stuff based on the similarity of the pages about to be returned - to make sure you aren't about to show the user two pages that are practically identical. That logic might try to build similarity mappings between the pages, and has logic that fails badly for large numbers of emoji.
> Server Error
> We're sorry but it appears that there has been an internal server error while processing your request. Our engineers have been notified and are working to resolve the issue.
> Please try again later.
this is a handled crash, but still a crash
how many emojis are there on iphone
edit: so does "how many emojis ios" and "how many emojis apple" but not "how many emojis android"
I'm on board with other people's suggestions that asking a question like this is tripping over a site that surfaced an emoji in their site title that is choking the Unicode library that Google is relying upon.
"how many emojis on ios" - error
"how many emojis on apple" - error
"how many emojis on windows" - error
"how many emojis on macos" - working
"how many emojis on lumia" - error
"how many emojis lumia" - error
"how many emojis on linux" - working
"how many ios emoji" - working (albeit slowly)
"how many emojis on messages ios" - working
"how many emojis on ipados" - working
"how many emojis in ios" - error
"how many emojis inside ios" - error
"ios number of emojis" - working
"how many emojis on i" - working
"how many emojis on ios has" - error
"how many emojis on ios has does" - error
"how many emojis on ios has does how has does how has does how" - error
I'd hazard that a specific web page appearing in the results is probably causing this error - I would be very curious to find out which page this is.
Edit: Yep, a specific .com site seems to be causing it:
"how many emojis on ios site:com" - error
"how many emojis on ios site:aero" - works
If you add a start date filter (even since 2000 years ago) it will work.
io, net, ru - ok
"how many emojis on ios -inurl:emojipedia.org"
trying to exclude other sites does not work:
how many emojis on ios -inurl:cnet.com
how many emojis on ios -inurl:wikipedia.org
https://www.google.com/search?q=emojipedia.com
At the same time I can reproduce the results from the grandparent comment.
is slow, but works, while
how many emojis on ios
still does not work.
After all (at least for me) it doesn't crash immediately but it more seems to throw an error after a timeout because it keeps loading without result.
how many emojis on ios -emojipedia
Check:
https://www.google.com/search?q=google%20before%3A1969-12-31
this one works:
https://www.google.com/search?q=google+before%3A2006-12-31
Not that we should... but curiosity does lead to some interesting finds sometimes.
relax with the disclaimers, I think we're all on the same side.
I was thinking more of accessing the Google search binary or source code.
or url, or some other edge case like https://daniel.haxx.se/blog/2022/10/14/there-is-a-tab-in-my-...
Wonder why "emojis" itself throws an error? I would expect the query understanding model to have the same result for the singular and plural forms
https://translate.google.com/translate?sl=de&tl=en&hl=en&u=h...
- same request with "s" removed from "emojis": works, but in 5.03 seconds (!)
- "how many emoji symbols on ios": 1.37 seconds
Other modifications further decrease the request time.
Something unusual about this query evidently triggered a bug in the somewhere. We will probably never know the specifics.