Back when it was introduced as a replacement for Transfer Service 15 years ago or so, the mention of the Latin etymology was always accompanied by a wink...
I'm not sure ex Googlers are even necessary for ill suited technologies to be used. There definitely weren't enough ex-Googlers to explain the spread of microservices or NoSQL. Reading about it on Hacker News might be enough.
Microservices are just SUN RPC, CORBA, DCOM, EJBs, .NET Remoting, SOA, WCF, rebranded for a newer generation, which apparently hasn't learnt much about them.
Hence now the modular monolith hype, as they rediscover programming languages have modules, and placing a network in the middle doesn't sort out spaghetti dependencies!
Its applicable if the services have different teams and different infra requirements. Its hard to create a ton of sphaghetti when integration is atleast a network hop away. There's some friction to being dependent on too much stuff.
Well it definitely scales, but also has a constant cost of entry, which probably would make it unsuitable for "small startups" - all details are in the paper.
However, if you read through the bibliography, you might notice that every so called "hyperscaler" and many network operators of similar scale (e.g. academia) uses something similar.
Yeah, no, that's not going to happen on this site. Why would people waste their time actually reading the paper? That takes too much effort and that's time you could have spent getting internet upvote points. That's why it's much better to post idiotic dismissals which have zero substance or value (and have been regurgitated 1000 times, reminiscent of when a toddler learns a new word).
It does remind me bit of "Big Data is Dead", which I believe was written by someone who worked on BigQuery (external version of Dremel), and then realized it doesn't actually add much value for customers:
My opinion is that Google culture enforces scale because it can lead to doing something competitors can't, or if you want to put it less charitably, "monopoly"
For example, there are stories YouTube being a few months from falling over when it was acquired, and being saved by Google scale
At the time, I do think it was true that few other companies could scale to what YouTube users demanded, or at least absorb that cost
---
So the company has a culture of scale, for sure, because it has business value FOR GOOGLE
I remember Marissa Mayer sent out "product guidance", and said that technical advantage is one of the criteria, i.e. being able to do something that a competitor can't do
The market is very hostile -- profits are eroded away by competition. But yeah for most companies, like approximately all Google Cloud customers, scale is not the reason they exist. The product is the reason they exist
By definition, there can only be one company that uses scale as a competitive weapon :)
i.e. there is only one "most scalable" company
Of course that's not strictly true, because there are different dimensions of scale, but it's perhaps a useful way of thinking about why Google culture is obsessed with scale. And conversely, why Google seemingly ignores and sunsets products that don't scale.
Yeah, I think people just don't fully realise how powerful hardware has become. You don't need complicated tools if you need to transfer even significantly large amounts of data, since you can have hundreds of gigabits of network capacity on a single server, so unless you're truly large you probably can get away with using vanilla rsync for file transfers for example :).
Managed async bulk transfers as-a-service seems relevant to anyone writing lots of data doing multisite including DR. You can easily have this problem and not have yet hit any other scale problems that are addressed by the alluded-to tools people love to hate (k8s, tf, containers etc).
What is being managed here with Effingo is the relatively tiny links between sites, not the bandwidth on the fat local links. Bandwidth needs to be managed and prioritized across all the various copies that are happening across different services apps and teams simultaneously across the company, and across the systems themselves as aggregate for the links. Managing it gives you control on cost but also ensures optimal use within the constraints imposed and consistent and reliable handling of the copy use case.
1.2 EB is 1.288e9 GB. The lowest listed price for cross-region traffic at Google is 0.2c / GB. That’s almost $2.6M per day to run this program — actually, more, because a bunch of the traffic surely goes outside North America! Google can afford that, but most companies can’t.
I mean, possibly. Networking is usually counted in bits though.
I wish there was a way to signify "I know about EiB I do mean EB". If you write EiB, there is no chance you meant EB, but the other way around is always ambiguous...
I spell out "trillion / quadrillion / quintillion bytes" for base-10 terabyte, petabyte, exabyte when the distinction is important. they're also unlikely to be confused with long-scale.
David Ziegler was the original designer and engineer who actually built effingo with all kind of headwings. It seems rather unfortunate he is merely mentioned as one of many who worked on it. That's probably why google hasn't really done any new groundbreaking system for the past 10 years.
34 comments
[ 4.0 ms ] story [ 89.3 ms ] threadeffing go for me too
Hence now the modular monolith hype, as they rediscover programming languages have modules, and placing a network in the middle doesn't sort out spaghetti dependencies!
At least now Open Telemetry based tools can render those in beautiful ways.
You ... think? I think there's a promotion in that in some organizations...
However, if you read through the bibliography, you might notice that every so called "hyperscaler" and many network operators of similar scale (e.g. academia) uses something similar.
The cost of entry will be worth it just in case the startup turns into the next Google overnight :)
> However, if you read
Yeah, no, that's not going to happen on this site. Why would people waste their time actually reading the paper? That takes too much effort and that's time you could have spent getting internet upvote points. That's why it's much better to post idiotic dismissals which have zero substance or value (and have been regurgitated 1000 times, reminiscent of when a toddler learns a new word).
They're always designed for infinite scaling, they always have trade offs like the initial setup costs, they always don't belong in a small operation.
There are always people with a fetish for those solutions because it feels good to have the potential to handle "big data".
Big Data is Dead (2023) - https://news.ycombinator.com/item?id=34694926
My opinion is that Google culture enforces scale because it can lead to doing something competitors can't, or if you want to put it less charitably, "monopoly"
For example, there are stories YouTube being a few months from falling over when it was acquired, and being saved by Google scale
At the time, I do think it was true that few other companies could scale to what YouTube users demanded, or at least absorb that cost
---
So the company has a culture of scale, for sure, because it has business value FOR GOOGLE
I remember Marissa Mayer sent out "product guidance", and said that technical advantage is one of the criteria, i.e. being able to do something that a competitor can't do
The market is very hostile -- profits are eroded away by competition. But yeah for most companies, like approximately all Google Cloud customers, scale is not the reason they exist. The product is the reason they exist
By definition, there can only be one company that uses scale as a competitive weapon :)
i.e. there is only one "most scalable" company
Of course that's not strictly true, because there are different dimensions of scale, but it's perhaps a useful way of thinking about why Google culture is obsessed with scale. And conversely, why Google seemingly ignores and sunsets products that don't scale.
What is being managed here with Effingo is the relatively tiny links between sites, not the bandwidth on the fat local links. Bandwidth needs to be managed and prioritized across all the various copies that are happening across different services apps and teams simultaneously across the company, and across the systems themselves as aggregate for the links. Managing it gives you control on cost but also ensures optimal use within the constraints imposed and consistent and reliable handling of the copy use case.
It is open access, so you should be able to see the PDF.
There is a youtube video summarizing the main points: https://www.youtube.com/watch?v=hBME9Q6RPZ8.
I’m being sarcastic, if this wasn’t obvious.
I wish there was a way to signify "I know about EiB I do mean EB". If you write EiB, there is no chance you meant EB, but the other way around is always ambiguous...
Oh wait, maybe the endpoints are in the special list of Google endpoints to which traffic is zero rated. :)
So that's how much data they collect from their users every day. /s