148 comments

[ 3.0 ms ] story [ 218 ms ] thread
Seeing these big sites with no ads is strange. CNN, NYPost, etc. This is going to be wildly expensive for Google.

Anyone internal know what's going on? We started seeing GAM not returning responses about an hour ago, yet their status page and accounts were silent till 10 minutes ago.

Does this mean every page is loading faster? Or do they just hang halfway trying to load ads?
Should be faster I think. Pihole just refuses to resolve ads servers and generally improves performance, right?
If the ad server is instead taking a long time to respond, it'll probably make pages load slower.
Only very very poorly built websites. So all the news sites.
There are definitely a thousand people here more qualified on this stuff (I write Fortran lol) but I thought JavaScript was all asynchronous now? Shouldn’t the actually desired content load independently of the ads?
Fortran is cool, what do you work on?
Parallel linear system solvers.

Grad student, so mostly I use Fortran so it’ll plug into my professor’s code more easily, haha.

No complaints, though. It is a fine language.

Requests for remote content don’t delay rendering of the page. Everything will be significantly faster.
Anecdotally it feels like all of these sites are super fast right now. I even reinstalled the heinous Weather Channel app and it’s working wonderfully. With ads that app is essentially unusable on an iPhone 14.
CNN is zippy without the chumboxes. Time to party like it's 1999.
Genuinely curious, do you not use an adblocker?
It always feels bizarre when someone on HN complains about lots of ads on web pages.
I used to browse without one when I worked on an ad-supported site and had to implement and make sure they were working correctly. Gotta assume it's a similar situation here?
I worked at one of the top ad networks for a decade and blocked them the entire time. Would usually just open a separately configured browser window for testing.
(comment deleted)
I have a "clean" (no addons at all) Chrome profile for web development purposes.
On personal machines, yes. Work involves ads so I keep it off the work machine so I understand what's going on.
Nobody on here is going to be authorized to talk about about it while it's going on.
Eh - worth a shot. HN surprises me sometimes with anons jumping in and giving a behind-the-curtain look.
> Seeing these big sites with no ads is strange.

I've been using ad blockers for many years, so for me it's the opposite: it would be extremely strange and unsettling seeing any ads on my own devices at all.

Same. When I help relatives with their tech woes, loading a website on a vanilla browser after all these years without ads is shocking.

All these flashing and moving gizmos and pop up videos. It's so jarring, like having a flashlight flickered in your face constantly.

I have honestly no idea how people manage to work the web without adblock nowadays.

yep... I've started loading pages I want to send to my mom with adblockers turned off just to check how bad they'll be because somehow despite turning them on every time I visit by the time I get back they have been turned back off...
just teach them to download adblock. It might be an annoying 20 minutes of your life but it'll do wonders for their mental health in the long run
I'm going to guess it's one or more of:

- Lack of experience or awareness that running an ad blocker isn't something weird or niche, but is in fact basic hygiene when on-line.

- Sites that ask, demand, beg or cajole visitors into turning your ad blocker off, many of which provide detailed instructions aimed at non-technical people; your mom may have needed something from one of such sites, and didn't bother or remember to turn the adblocker back on. If I were to bet, I'd pick a link to an interesting article on a news site that employs this technique.

- Perhaps your mom uses one of those cashback extensions to get some money back when shopping on-line? They're somewhat popular and regularly advertised by vloggers and influencers. Disabling ad blocker is AFAIK necessary - for technical reasons - for the service to award you cashback.

My wife uses one of those cashback services, but we did some research on their inner workings and business model first, to be sure she's taking advantage of it, instead of being taken advantage of herself. And she follows a strict routine: she disables uBlock before visiting a participating e-commerce store, does the purchase, and re-enables it immediately afterwards.

— There are so many ads on this page, it must be intolerable. Do you want me to install an ad blocker?

— Nah I'm fine.

haha, the same thing happened with my 10 year old nephew

I was watching live TV, and he just randomly started screaming at me

I was like "Why the hell are you screaming, kid?" and it turned out it was because the TV was so loud and obnoxious

Turned out he managed to go 10 years without watching ad-infested live TV!

Same experience. People universally hate ads, yet they're not too keen on blocking them. I'd be curious to learn why.
I think some people are so annoyed or afraid of tech that once they’ve got something that at least somewhat works they avoid making any changes in fear that it’ll break their workflow.
Ads lower the barrier of entry to the web by paying for content. By blocking them and encouraging others to as well that on-ramp becomes weaker, motivating producers to pack more ads for what audience still sees them.
Funny thing though, I worked in ad-tech for ten years at a small player (outbrain). They're present on most major news sites around the world, which means I got to dig into the source of many of these sires. Easily 90% of their source was dedicated to serving ads or tracking behavior for campaigns. And we're talking a ton of code. It's like they're in an arms race with themselves where their site is an expensive nightmare to maintain so they add ads which makes their site an expensive nightmare so they add more ads.
Damn I tried turning off my adblock just to see but it's probably been years since I've seen those sites _with_ ads so I have no way to relate to this experience
I trust that payola pre-written content is still going up as usual, though.
What’s payola?
back when radio stations played records, record companies would pay DJ's to play specific songs or albums [1]. I was a scandal at the time. It's almost quaint now.

1 https://en.wikipedia.org/wiki/Payola

Technically still illegal (IANAL), since advertising is supposed to be labeled as such. But widespread.
Good, keep it that way forever
would you pay subscriptions instead?..
(comment deleted)
Interesting question. I probably would not, which might mean I would spend less time on the internet reading useless stuff and more time outside on my mountain bike.
There can be subscriptions and mountain bike riding at the same time you know.
I do, and in some cases, I still get ads.
Would you offer the option to?
services likely do studies to decide do they want to be ads based, or visitors are willing to pay for subscription.
And then, historically, show ads to their subscribers.

Cable, streaming video, etc.

Because advertisers pay more to reach people they know are willing to pay for stuff. It's whatever the opposite of a virtuous cycle is... a race to the bottom?
The conventional wisdom is that offering paid subscribers ad free experiences poisons the well for advertisers. After all, it eliminates the people that have enough disposable income to pay a nominal fee!

Not sure how this translates to targeted ads. Would offering such a thing violate Google’s TOS?

There's a New Zealand Journalist, David Farrier (https://en.wikipedia.org/wiki/David_Farrier) and I actually subscribe and pay a monthly fee to be able to view what he writes. I think he's really awesome (some will disagree) but I just like him. He also has a TV series on Netflix called Dark Tourist if you're able to see that.

He has a new movie out at the moment that I really want to see, but his last movie Tickled was _crazy_ like... Tiger King type craziness.

So yeah - if I know someone who actually puts in the work to give me quality output I'd pay for it. Just like I do now.

Yes, can I subscribe to the internet?
They have the internet on computers now
Ads are an enabler of bad products. If things required a subscription, half the shit on the Internet would be gone and for good reason. Also, often you get ads anyway regardless of the subscription. This is a false dichotomy.
would you not serve me ads if I did?

All newspapers/magazines, VHS/DVDs, cable TVs etc. were are 'subscription' in the sense you had to pay for them. Ads were still there. If paying a subscription for websites was normalized then soon enough they would as well be filled with ads.

Yes. But I wouldn't subscribe to "content" that is used as a minimum skeleton to hand ads on, and I include the google search engine in that definition. I'm happy to pay for something I value. Most web content I value at $0 and may browse opportunistically but I would never pay for it, including by looking at an ad or providing my email
I've spent hundreds of hours of my life contributing to Wikipedia pages and may spend hundreds more on open source projects. Capitalism has made us label any not immediately self-serving as illogical. We are not homo economicus
There is this concept of supporting people you like without them having to charge people. The more you can read upfront, the more you can make an impartial judgement if you actually like them. A lot of people subconsciously convince themselves that they like something just because they spent money on it. If I like a project or open source software that I use then I go support the developers. A lot of other types of arrangements are basically signing a contract waiving your rights before you've even had a chance to form an opinion.
What's the difference? You already pay the ads on top of every product you buy making everything you buy more expensive, it's already a tax.

The only difference I see is all the money on salaries and expenses related to ads is 100% a pure economic loss in the ad model.

(Of course I'm only talking about the economics here, there's a lot of other bad side-effects of ads not covered here)

MindGeek ads are still working. Shows why it's important to diversify your ad delivery
They had outages far more than Google ads outages.

They would love to use Google if it wasn't against the TOS and nobody wanting to deal with them because ethical and morals clauses.

Source: Worked there in the past

...and it was at that moment that riotous celebration among all mankind swept the globe in unanimous rejoice.
In unrelated news, productivity around the globe went up 5%.
Is the link to a description of the problem, or it is just an example of a dead page? (Not sure if I’m getting a dead page because of their issue or because my ad blocker blocked it, haha).
I'd like to see some normalized numbers to see how the absence of ads affected sales, neg, pos or neutral.
World becoming a better place momentarily
Didn't even notice. Ad blockers work?
I can't load this link thanks to pihole :smug:
Wonder what the $ loss per second is…
From their recent 10Q for investors, they list a revenue of $7.8 billion for "Google Network" in revenue for the preceding 3 months before September 30, 2022 [1] There are 90 days in 3 months, so their revenue for a day is approximately

$7.8 billion / 90 days = ~$87 million/day

$87 million / 24 hours = ~$3.64 million/hour

$3.64 million / 60 minutes = ~$60,740 thousand/minute

$60,740 thousand / 60 seconds = ~$1,012/second

And, depending on their agreements with their advertising partners, they might be liable for some of the profits lost by their partners.

[1] https://abc.xyz/investor/static/pdf/20221025_alphabet_10Q.pd...

edit: It was initially using the wrong row, it should be correct now.

I don't think that's the right number to look at, given search ads aren't down. You probably want just the "Google Network" line item, which is about 1/6th of that number.
You are correct, I just misread it.

edit: It should be fixed now.

Exactly. Although this is likely one of the biggest days of the year for display and video advertising, so it'll be likely more than 1/6 of $25m/hr.
It doesn't look like it's affecting their search ads (or if it is, it's not fully down for them), so not the full advertising revenue of GOOG. But it's still likely tens of millions per hour.
I took one for the team and disabled my adblocker so I could get a screenshot of the page.

https://i.imgur.com/9n7HuLu.png

"Ad serving has now been restored for the affected users."

https://i.imgur.com/rnRiF6Z.png

How do I get to pull the actual image from the poisonous pile of garbage called imgur? It runs JS from 16 domains.
Use a PC. On mobile it's hopeless - on top of garbage scripts that sometimes lock up even a recent flagship phone, the service does aggressive lossy compression that's hard to work around. If someone uploads an image (whether PNG or JPG, doesn't matter) that has small details like text, forget about discerning them from the blurry mess imgur sends your way.
For single images, imgur will redirect from /<id>.png to /<id> which requires the ton of javascript (you see a message about javascript required to run). Just re-add the extension to the end of the url and you'll access the image directly.

This trick doesn't work for albums.

Interested to hear how many "hands on deck" this gets vs a Google Cloud or Google Workspace outage. Don't fuck with the money!
Does GCP not make Google money?
You wouldn’t think so based on the quality of their support team
No. GCP consistently loses money.

One of many sources: https://www.theregister.com/2022/07/27/alphabet_q2_2022/

> Google Cloud brought in $6.3 billion of revenue, a 35 percent year on year increase compared to Q2 2021’s $4.6 billion revenue. But the $858 million loss was 45 percent higher than the $591 million deficit recorded last year.

You might ask: "how is that possible, with those same margin-intensive prices that AWS and Azure use?" I have no clue. It makes no sense, given the prices they charge.

What is or isn't cloud is different at every company on top line but especially bottom line. For example how far down the stack are engineers still considered cloud when they are supporting both cloud and internal products? Investors are fine losing money on growing cloud and also love to see ad margins up? Then everyone is cloud.
I don't think salaries are the driving factor behind GCP's balance sheet, but that's just my opinion. I obviously can't wave a magic wand and find out for sure either, but my own back of the napkin math can't see that reasonably working out. They brought in billions of additional revenue, but their losses also increased.
Similar reasons to a company with 30 tps 8am - 6pm has a 3 h/a Kafka running on a k8s cluster, multi master MySQL cluster, all the trimmings

Built to scale that isn’t here today ? Resume Padding ? Data centers are expensive and slow unto build

GCP revenue losses from being down are approximately lower bounded by (downtime in seconds * revenue per second). Their costs don't decrease from being down.
Labor costs to keep all of these features/servers/infra running is expensive. Google is probably charging less due to competition from AWS/Azure/Oracle.

Similar to how Amazon.com is subsidized by AWS. GCP is subsidized by the ad biz

P0, page everybody. But their incident response planning and training is on point so there’s one incident commander and everyone doing individual tasks reports to them. I suspect not more than a dozen cooks are in the kitchen as it were. Any more and people would be getting in each other’s way.

I seem to recall that ads runs on one of the largest distributed MySQL instances on the planet. That’s probably the most complicated component at play here. Then again everything has to have active, secondary and tertiary so maybe it’s something upstream like dns dos protection or cdn.

Not a Google employee, just guessing.

I like to think that at a certain point, more “hands” doesn’t help solve the issue faster (maybe even counter productive), so that in both circumstances the outages get the same amount of hands — the maximum.
Finally iOS safari adblocker!
Might I recommend Firefox Focus for Safari ad blocking?
You’re joking but adblocking has been available on Safari iOS for several years
I didn't know! Mainly because I just got in a habit of avoiding the crappy ad-ridden sites when on mobile.
And the internet kept on working. In fact, better, faster than before.
You are ignoring the concept of debt. Everything doesn't immediately implode. Especially for a temporary outage.
Don't take for granted how many ad supported sites and services you use :)

I have a website with several writers that serves almost half a million monthly pageviews and we are entirely ad supported. The only reason my writers have jobs and we are able to provide the recipes and information that we provide is because of ads.

So many viewers, why not start asking for voluntary donations? You could dial back ads, and thereby tracking, to the same degree you receive donations. Over time see how far it goes.
This is naive. People simply don't donate, it doesn't work. It just doesn't
This is naive. If you don't try, you never know for sure : )

But that easy retort aside, there are many creators, who live off of donations. And as a matter of fact, it also cannot be correct, because I count myself as "people" too and donated to several projects/creators.

We aren't a "project" and we aren't individual creators. It works okay for certain types of audiences, but does not work for my type of audience. I know my business model well and there is a reason none of my competitors ask for donations either - it doesn't work.
This is one of those good outages
Google is A/B testing impact of switching to their new business model which will be paid search+AI ;)
Crazy to see websites load this fast without ad-block. This event must be eye-opening for non tech-people...
It's always weird when something like part of Google goes down. I just don't expect it because of how massive Google is and how much money they put into their site running reliably. I guess it's interesting to be reminded that Google can go down just like any other website, it's just a really big one.
A few weeks ago, no one would have predicted a major Google service going down before Twitter.
Ouch. You got me. I’d have bet hard against this.
I, for one, are caught red-handed on this. It is so weird that Google's money-maker has been down, especially that I never have remembered when was the last time it was down (maybe practically never?)
Google laid off the one guy that was keeping this application stack running
It's a nice reminder that even the big players can make mistakes.
(comment deleted)
Burn it before it lays more eggs.