Early-Google really fucked moron salespeople at most of their vendors (well, technically the salespeople also got paid, so the only morons were the investors). People sold space with power/cooling included at ratios which were unsustainable, and Google's people were more than willing to sign contracts with the facilities at those prices. Then Google "fully utilized" what they had in the contract, which really screwed their colo providers.
OTOH, this was in the colo/bandwidth/fiber nuclear winter of 2001-2004, so maybe some vendors were happy to have some revenue, but there were cases where the Google stuff was at negative overall margin on variable costs (i.e. the vendor would have been better off shutting down the power plant vs. selling to Google).
Heh. Everyone did that at the time, because the colos didn't know better. I had a small 300ish installation in a tier 1 provider in the Northeast, and at one point in 2004 they had to tell me they couldn't provide the power for my new racks because they were literally out of circuits. Violation of contract, but they had no choice.
Even for a scrappy startup, I consider this to be exceedingly unethical. Somebody paid for that bandwidth, somebody who essentially invested in Google with no opportunity to see a return on the investment.
I can't picture the world where this would not be considered "evil".
You will always have bad people. The more people that exploit the system, the faster the system will change to be more 'fair', otherwise you're just handicapping yourself. Look at all the patent lawsuits. As they said in my business ethics class: ethics schmethics.
(1) It's not clear that acting unethically will lead to better rules and more ethical outcomes, in general [and actually, in some cases the opposite will happen - if you're corrupt it makes it easier for others to be corrupt]
(2) Handicapping yourself is the whole idea behind ethics and morality. Sometimes it's more important to do the right thing than the selfish thing.
Attitudes like yours are so disheartening to read. I totally disagree with your sentiment.
I think it's funny how you start about ethics and then immediately follow by talking about money.
Whether it was unethical depends on whether anyone suffered negative consequences and whether Google knew, or could foresee that. For all we know the bandwidth was available, couldn't be sold to anyone else for lack of customers and would simply have gone unutilized if Google hadn't used it.
Money is a measurement of value, like popularity is a measurement of skill. They're both poor measurements, but very easy to apply, they cater to everyone's understanding and there's undeniably a correlation. Therefor they get used a lot and people lose sight of how poor they are in many situations. Numbers also allow us to decouple from the ethics of the situation.
Take healthcare. You would not put a price on your own life: it is priceless to you. Nevertheless we put prices on other people's lives, because it is the easy way out, is accepted by almost everyone and allows us to avoid having to answer nasty questions. No one wants to honestly say someone unknown is worth nothing to them to keep alive. So you say you will only support health care up to $10K a year, because you know the chance a loved one will ever need more is extremely small. You talk abstract numbers to prevent from having to admit to yourself that you just don't give a damn. That wouldn't fit in with our ethics, would it?
People talk numbers because money/supplies are finite. We can't spend 200 million trying to save one person, as that will simply create a lack of resources for every other person who needs care...
In the US, we've got ten million unemployed and tens of millions underemployed even when we're not in the middle of the Great Recession. With the proper application of economics, we could easily train tens of millions of more healthcare professionals (note: it doesn't matter if the unemployed are qualified; you can train the qualified-but-otherwise-employed and let everyone else shuffle to fill in the gaps. Hermit crab principle.) and end up with a large enough surplus to treat everyone as to their need.
As employment strategies go, it beats paying people to dig holes and then fill them in.
You're completely right, but many of us realized "don't be evil" was nonsense long ago. That ones who need to hear your message would defend Google as not being evil no matter what they did (as you're already seeing in response to your post).
This just seems like you patting yourself on the back for 'realizing the Don't Be Evil was BS.' A lot of people on this forum would see this as a clever hack regardless of who did it. The fact that you bring up the whole "Don't Be Evil" bit and then immediately reference yourself is telling.
There's a lot of projecting in your post. I'm not patting myself on the back for seeing the obvious. I'm frustrated that other, otherwise very intelligent people, are so blind to it.
This "95th percentile" measuring method is still widely used by bandwidth providers, colocation facilities, etc. I am not sure why the article says it was a common billing practice "of the time". More info: http://en.wikipedia.org/wiki/Burstable_billing
It's quite common if you ask for a quote for hosting in the UK.
If you ask for a single server you generally get a monthly allowance. For example, one of our servers sits gets 5TB a month on a 100Mbit port. Saturating that 100Mbit port would give me 30TB (and a large overuse bill!).
But if you ask for colo rackspace (1/4,1/2 or full 42U cabinets) then you generally get to choose what drops you get (and what speed), what connectivity and what the 95th percentile billing rate is.
Here's a random example (i've no connection with any person or company in this thread):-
I've never met an ISP that sells Racks in a colo that didn't bill transit at 95th percentile (if you chose to use theirs). Are you hosting bare metal, or is your colo providing the physical server?
If you deal with a Datacenter that is just providing you with a Cage + Power + Cooling + Physical Security, you are almost always going to pay for transit @95th percentile.
All the colos I've looked at use the 95th percentile. Our bandwidth is 100Mbps but our capacity is 10Mbps. This would be about 26Tb/month. Our contracts are also in the UK so it might be a regional thing.
Single servers tend to use the total usage model. If you're buying space by the rack, you're more likely to use 95th percentile. CDNs often offer both methods and let you sort out which is better.
It's common for internet providers who cap to run the 'capping' process in batches, so it's possible to get 'free' bandwidth by exploiting the time period (often up to 24 hours) between reaching your limit and when your provider caps you.
Everybody who pays @95th dreams of doing something like this the very first time that 95th billing is explained to them. I mean Everbody.
What the Article fails to explain, is that there is both a port cost, and a minimum commit. For much less than what it would have cost them for the port, and minimum commit on a 1 Gig connection, they could have simply mailed their data via USPS and had it there in three days.
Also, if we do the math - and presume that they only ran 1 gigabit for 5% of the time - 1 gigabit/second at 36 hours (5% of a month) in terabytes = 16 Terabytes - which barely makes it into the "10s of terabytes" category, and well within the "Just mail an AIT-3 tape. [Edit - cheaper yet, just load up the hard drives that you are going to install anyways in your destination]
Edit 2:
Doh - http://www.mkomo.com/cost-per-gigabyte shows that the largest hard drives back then were 20 Gigabytes. We're talking a minimum of 1000 hard drives for "10s" of Terabytes.
Maybe that 1 GigCircuit Hack wasn't such a bad idea after all.
When they're adequately backed up, as in this circumstance, why not? Send the data in triplicate if you want, just to avoid the hassle of re-sending any that get damaged. Regardless, the cost would be trivial, even for young-Google.
Presumably I'm not sending the originals. Also, powered down, hard drives are pretty robust - their read/write heads are locked up. Remember - those drives have been bouncing around a LOT to get to you through retail/wholesale channels.
[edit] Also - they obviously had to buy hard drives for the destination, right? Presumably, early stage google was all about the open-chassis on plywood slam drives in approach, and wouldn't have been averse to loading up 30 or 40 drives, mailing them out, and simply diffing and resyncing as needed if 1 or 2 drives had issues, which would have been unlikely.
Also, presuming you needed to have one of your people go to the remote data center, there is always the old, move the datacenter with hard drives in my luggage trick. Even in 2000 you could put a lot of data into carry-on.
You could make up any small number of botched drives with over-the-wire transfers. If it were a sufficient number of drives, you could simply re-image and re-transmit the data.
In the year 2000 storage cost about $12/GB [1]. This article claims 10s of TB, so let's say 20 TB, which is the most conservative possible estimate. 20 TB = 20000 GB. At $12/GB, that comes out to $240,000. That doesn't include retransmission cost (for busted drives) or shipping. I'm not sure the analysis is correct here.
Presumably their minimum commit was a lot lower than $240k, because otherwise they wouldn't have shrunk from just using the connection. They probably saved a lot of money by doing it their way.
Right - but they had to buy those hard drives anyways, so it didn't cost them anything more than the transportation cost of the drives to buy them at the source, load them up, and then ship them.
It would be hell to get the data off before imaging the drives. What do you transfer it to? The shuffling would require a lot of technical work, which also costs money. Also, we don't know if they were on homogenous hardware by then -- they may have still been using budget pcs and refurb'd parts. So it's hard to know if your plan could work. The investment of inventing a process for mass data transfer versus using the normal process but only during a certain time period is also significantly higher.
I have to think someone ran the numbers. It's not like the folks in 2000-google were dummies.
Admittedly, it would have been a lot of drives. Back in circa 2000, according to http://www.mkomo.com/cost-per-gigabyte, 20 Gigabyte Drives were in vogue @$200/pop. So, a Terabyte would have been 50 drives, 10 Terabytes @500 Drives.
10s of terabytes at a minimum 1000 drives. I'm beginning to question how "simple" an operation this might have been after all....
The parameters of that scenario were that you'd ship the data on the disks that were eventually to store it (presumably without duplicating the OS on each during the transfer process). Therefore, there'd be nothing to move the data to.
The difference is that the hard drives themselves would be assets (and would probably have nostalgic value in 2013 if they wouldn't have decided to resell after the transfer)
Even if you didn't consider cross-country-shipped drives to be data-center reliable (e.g.: the transfer media weren't re-used in the datacenter), you could simply cycle through drives. If your goal was 1 month to transfer 20 GB, you utilized overnight shipping, and required a day to load/unload data from drives at either end, that's 200 drives in flight, 50 of which are outbound, 50 returning, 50 writing, and 50 reading, on any given day.
You even get to take weekends off.
It's $48,000 in media. Plus shipping (not inconsiderable). You'd save on the cost of dedicated data transfer equipment (datacenter, router, links), though you'd need infrastructure for reading/writing data, and of course, storage at the remote side if you're not simply shipping disks for installation (which, frankly, is probably your best bet).
Resync should be fairly reasonably accomplished over the wire.
You can vary the math by taking longer (fewer drives, less cost) or shorter (more drives, higher cost), or even by utilizing ground shipping (7 days) at far lower cost.
I can assure you that there were drives larger than 20G in 2000. I had a server with 4 23G Seagate Elites at that time, and the Elite was an _old_ 5 1/4 FH SCSI drive. 18G Seagate Barracudas were certainly alive and kicking since 1997 at minimum (the year I started building real production servers).
Threatened to waste $70,000 of electricity to circumvent a $210,000 contract penalty (for not buying enough electricity), and thus negotiated their way out of it.
(Note this is MSFT not GOOG, so be sure to interpret this as an act of disreputable evil and not a clever funny hack).
It is kind of stupid that the penalties were MORE than the actual electricity. Shouldn't it have just been the cost of the electricity that they failed to use?
That way loopholes like this wouldn't have existed.
No, because disposing of the unused electricity has costs for the provider.
The way this works is that the provider offers a cheaper rate to ensure constant energy output, since the power plants can't be easily and cheaply scaled back.
As an example, Ontario paid millions[1] to get others to use their surplus energy; MS' provider could have been forced to do the same, hence the penalty.
It seems...unlikely that they actually needed to pay people to take their power (though I'm no EE; it may well be possible). My intuitive sense from that article was that it was a contract issue (and possibly not even that; they also throw in something about "below costs", which would obviously not be negative price).
Electrical generation and distribution is a tough business. Places where you have generation capacity aren't necessarily accessible via the transmission network. So if NYC isn't using power, you can't just send it to Cleveland. Sometimes, you have a certain amount of "fixed" generation capacity that you must use, or must commit to taking offline for days before restarting. 80% of your power may come from a few big, centralized, cheap power plants (nukes, big hydro, coal) and a bunch of geographically dispersed, small, more expensive plants that you spin up to meet peak demands. (natural gas, small hydro)
Utilities sign contracts with big industrial users who have steady-state energy demands to ensure that they have a place to send lots of power to. They also have contracts that when peak demand is reached, the industrial customers shut down lines in exchange for discounts. (I worked at a place that financed a DR datacenter largely by committing to failing over 50% of our energy workload to another region during a peak demand emergency.)
I'm not doing the topic justice, but there's alot of behind the scenes stuff there -- I'm sure it wasn't just a contract dispute.
From the article it doesn't sound like there was any sort of dispute...my guess would be that the contract formalized payment as a function of capacity and demand, in such a way that payment could be negative. This might not even be a mistake or from a need to distribute the power, it could just be a low-cost (due to rarity) sweetener.
It's not just about using the excess generation capacity. In fact, at this scale it probably has nothing to do with generation. For distribution utilities, they need to install new infrastructure to connect new large loads (new datacenter, new manufacturing plant, etc.) - substation upgrades, new conductor, etc. Because that infrastructure only serves that one customer, the cost is recovered from them through a minimum-connected-load + capital contribution contract rather than from the entire customer base through a rate increase.
Keep in mind that utilities are heavily regulated and can't typically touch their rates at all without regulatory approval. In many jurisdictions, there is an "obligation to connect" for residences - the utility /must/ provide power, at a certain rate, to any residence in their area of operation. They aren't allowed to directly recover the cost of infrastructure upgrades (e.g. new distribution transformer when somebody builds an expansion to their house) directly from that customer (up to a point - if you're building a mansion that uses 10x as much power as everyone else, they'll come after you), only from the ratebase as a whole. Otherwise, people who live in rural areas wouldn't be able to afford electricity.
No such obligation exists for businesses which are typically the largest loads anyhow. If you build a new 50 MW datacenter on my distribution system, now all of a sudden I need to put in 5km x 3 phases x 5 feeders of new conductor, 2 new station transformers, plus civil infrastructure, etc. It could even be special equipment I wouldn't otherwise have on my system - like DC equipment. All of that is additional to what I have to pay the generator for that 50 MW of power on a continuous basis. So, I calculate the payback period on 50 MW of power I'm going to sell you at a 5% margin, and if it's longer than the lifespan of the equipment I'm going to buy I make you pay up front to bring it down to say, 25 years. But that only works if you actually buy 50 MW of power from me for 25 years, so if you don't I charge you a penalty equivalent to the amortized cost of that infrastructure that now sits idle.
Hope this helps explain the economics of it from a distribution point of view. I am a power engineer.
Here in Toronto the current standard for residential service is 200 A @ 120/240V (24 kW). Older homes might be 100 A. That would be the max that the conductor between the distribution transformer and the home's panel can handle without thermal failure. But you wouldn't size distribution transforms based solely on peak load because a mineral-oil immersed transformer has a lot more thermal mass than a cable and takes hours/days of sustained load to actually heat up. Average loads used for planning and distribution transformer sizing are more like 5-10kW/house.
At the time, Google was not hugely profitable like today, and were very conscious of costs.
Meanwhile their hosting company was making a trillion in profit /s.
Google plays fast and loose with ethics and definitions, from (shall we call it) penalizing competitors and then releasing a clone to working with Visa and MasterCard ala Wikileaks to cut off the money http://www.gizmodo.co.uk/2013/02/google-wants-to-stave-pirac... to sites with "piracy sites." Their PR hasn't worked on me for a few years and there's even a boomerang thanks to it.
> At the end of the month, the top 5% of usage information was discarded, to eliminate spikes, which were assumed to be measurement errors.
Not actually true. The bandwidth measurements don't have accuracy problems; providers use 95th percentile billing to make connections "burstable", allowing servers to handle an small unexpected traffic spike without a huge bill. And on the flip side, many sites have a traffic pattern of highs during a subset of the day (customers' waking hours or business hours or non-business hours, depending on the site) and lows outside that period (sleeping, etc), for which 95th percentile billing produces a higher bill than just charging for bytes transferred.
Very few customers have the kind of traffic pattern that would allow for using bandwidth during only 5% of the month. And in this particular case, Google could just as easily have saturated a 50Mbps line rather than using 5% of a gigabit line, which would have massively reduced (though not completely eliminated) the bill.
That is a pretty awesome hack, but I wonder why they didn't just use hard drives and ship it instead of wasting months. Cost / benefit seems off there.
As mentioned above around that time cost of data on hard drives was $20/gb so that would come out to around $240k at the time. It's crazy how things have changed and we can buy 16gb flash drives now for $5-10. I remember when the first flash drives came out and it was like 128mb for $20.
I don't understand the "didn't use the connection at all outside that time, they should be able to get some free bandwidth." How or why would Google shut down a data center for that long just to pump some data across the lines? Wouldn't that mean down time? I don't get it.
They probably had a separate line coming into a router specifically for that purpose and directed all of the traffic to transfer the data through this router.
The purpose of this data transfer was internal - transferring search index data between west and east coast - and they had a supplier contract just for this purpose.
They had a lot of flexibility around when they used the bandwidth. I don't understand why they couldn't just negotiate a deal whereby they would only send data during times of low demand.
25:2 And it shall be, if the wicked man be worthy to be beaten, that
the judge shall cause him to lie down, and to be beaten before his
face, according to his fault, by a certain number.
25:3 Forty stripes he may give him, and not exceed: lest, if he should
exceed, and beat him above these with many stripes, then thy brother
should seem vile unto thee.
25:4 Thou shalt not muzzle the ox when he treadeth out the corn.
25:5 If brethren dwell together, and one of them die, and have no
child, the wife of the dead shall not marry without unto a stranger:
her husband's brother shall go in unto her, and take her to him to
wife, and perform the duty of an husband's brother unto her.
----
Yeah the torture they do to me has made me brain dammaged. God will fuck with them.
God says...
compelled announcement cauldron fruitless Name shoulders
bin body begin impulses pause Or GUTENBERG-tm utterance
forget attempered cultivating decrepit sayest arrogancy
upwards host Elect furious do_you_know_what_time_it_is
those joining sober ye plenteousness strongholds recognised
cheap Vietnam religiously presumed subsequently language
willingly goads wound software impatiently pulse darkened
Armenia well-nigh whit cd http stumbling appearance faint
perceived formerly GUTINDEX acted callest sacrifice you're_wonderful
lo recruiting XIII metres successively desiredst bedimmed
INCLUDING exacting create sung rejoices theatrical fling
happy last subjoins cold 3000+ have marvelled insight
to_infinity_and_beyond Association corruptible avenue
silent lovers Tunisia admire mind- part encumbrances down
fled no_more fervent phantasm Substance brightness don't_mention_it
proportion show dining-room imagined censers avoid guilty
ample asunder capital Oxford countryman present Excellently
sakes mother fathers allege tale lift deafness merged
eagerly been orderest yesterday hardship abandonedly spend
risen snap_out_of_it voice joineth sermons
I find this a little hard to believe; it has the whiff of an urban legend. Besides the fact of minimum monthlies and the probability that the bandwidth provider is not so stupid as not to notice that they're moving lots of terabytes30 hours a month and nothing at all the rest of the time (especially over a period of months), where's the time saving? you could just do all the copying in the data center, repack the drives and fly them to the east coast. This seems a lot more efficient than trying to do it in chunks of <72 minutes/day.
So is it that the index didn't change that much in the months the transfer was happening? or the changes were cleverly isolated so they could be copied over last and the switch flipped immediately after?
Maybe I'm overly skeptical in my old age, but I have a difficult time believing this.
Surely there'd some sort of rental cost for the line in addition to the bandwidth cost (or perhaps a minimum spend commitment)? Wouldn't it have been cheaper, simpler and possibly quicker just to get a smaller line and saturate that?
Has anyone read the book? Does the author provide a credible source?
I was surprised to read this in the book, and even more surprised that it seemed no one had discussed it online yet either, as far as I could tell. Its on page 187 of the book. The author, Steven Levy's source for this story is Urs Hölzle who was the first VP of Engineering at Google. His title now is Senior Vice President for Technical Infrastructure.
A fair number of bandwidth providers charge little or nothing for inbound datacenter traffic (it's not what they're constrained on), which makes services for which inbound traffic is high, like web crawling, very cost-effective. Google likely benefitted from this as well, though they're hardly the only ones.
If, by "10s" of terabytes, they mean at least 20 terabytes, then, in 2000, a "Large" hard drive was 20 Gigabytes. That comes to 1,000 Hard drives. If they had some sneaky way of getting onto a GigE connection, then perhaps this might have made sense...
They still had to purchase HDs at the other end. It's not like data just lives in ether. At some point it's got to be written to someone's spinning rust.
How is this any cheaper than copying the HDs, shipping to the Northeast and installing the HDs into waiting servers there?
Depends on their architecture. Maybe the data and the system image lived on the same drive. Maybe they were just provisioning cheap / refurbished parts.
The logistical challenge of swapping 1000+ drives into/out of a small number chassis, keeping track of what data you've copied, would have been painful (but not impossible if stretched over a week)
I think the key issues here would be was moving the data at all time sensitive (is a two month latency not a problem), and, did they really have a "free" 1 GigPort they could sneak under the 5% window with? If it really was free (or very, very cheap) - then maybe it would have been worth it.
I've just made some corrections to the post. I was going off memory when I originally wrote it. After re-reading the relevant passage from the book again I've corrected the number of terabytes that Google needed to transmit - it was more like 9 TB. And they transmitted the whole lot over two nights in one month, not three nights. Sorry for the confusion.
It was on page 187-188 of Steven Levy's book "In the Plex" and the two people he's quoting are Urs Hölzle and Jim Reese, who were both involved in Google's infrastructure.
This kind of traffic optimization happens more frequently than you might think. Particularly among CDNs.
CDNs don't optimize solely for performance. They optimize for a balance of performance and cost. That means saturating a given 95/5 link for 5% of the month, and then mapping the traffic away right before it starts to get expensive.
I completely believe this story, because I've done similar stuff. In 1998 I was working for a company that was using a lot more incoming bandwidth than outgoing bandwidth, which is the reverse of the usual pattern. So when we outgrew our data shack and had to move to a real data center, I badgered my sales guy into showing me the overall bandwidth usage charts for the proposed location.
Unsurprisingly, there was way more data flowing out of the facility than there was flowing in, so I talked him into a substantial discount on inbound traffic. "Hey, nothing else is using that half of the duplex..."
103 comments
[ 4.3 ms ] story [ 165 ms ] threadOTOH, this was in the colo/bandwidth/fiber nuclear winter of 2001-2004, so maybe some vendors were happy to have some revenue, but there were cases where the Google stuff was at negative overall margin on variable costs (i.e. the vendor would have been better off shutting down the power plant vs. selling to Google).
Do we have an ethical duty to make sure our counter-parties are always making sustainable offers?
I can't picture the world where this would not be considered "evil".
(2) Handicapping yourself is the whole idea behind ethics and morality. Sometimes it's more important to do the right thing than the selfish thing.
Attitudes like yours are so disheartening to read. I totally disagree with your sentiment.
And no, I don't think Foxconn is bad for replacing workers with robots. Why would I?
Whether it was unethical depends on whether anyone suffered negative consequences and whether Google knew, or could foresee that. For all we know the bandwidth was available, couldn't be sold to anyone else for lack of customers and would simply have gone unutilized if Google hadn't used it.
Why would that be funny? A lot of ethics is about not destroying someone else's value. Value is measured in money.
Take healthcare. You would not put a price on your own life: it is priceless to you. Nevertheless we put prices on other people's lives, because it is the easy way out, is accepted by almost everyone and allows us to avoid having to answer nasty questions. No one wants to honestly say someone unknown is worth nothing to them to keep alive. So you say you will only support health care up to $10K a year, because you know the chance a loved one will ever need more is extremely small. You talk abstract numbers to prevent from having to admit to yourself that you just don't give a damn. That wouldn't fit in with our ethics, would it?
In the US, we've got ten million unemployed and tens of millions underemployed even when we're not in the middle of the Great Recession. With the proper application of economics, we could easily train tens of millions of more healthcare professionals (note: it doesn't matter if the unemployed are qualified; you can train the qualified-but-otherwise-employed and let everyone else shuffle to fill in the gaps. Hermit crab principle.) and end up with a large enough surplus to treat everyone as to their need.
As employment strategies go, it beats paying people to dig holes and then fill them in.
Extremely good analogy. Thanks.
A lot of people equate money with value directly. Only half true...
If you ask for a single server you generally get a monthly allowance. For example, one of our servers sits gets 5TB a month on a 100Mbit port. Saturating that 100Mbit port would give me 30TB (and a large overuse bill!).
But if you ask for colo rackspace (1/4,1/2 or full 42U cabinets) then you generally get to choose what drops you get (and what speed), what connectivity and what the 95th percentile billing rate is.
Here's a random example (i've no connection with any person or company in this thread):-
http://www.webhostchat.co.uk/request-products-services/26778...
If you deal with a Datacenter that is just providing you with a Cage + Power + Cooling + Physical Security, you are almost always going to pay for transit @95th percentile.
What the Article fails to explain, is that there is both a port cost, and a minimum commit. For much less than what it would have cost them for the port, and minimum commit on a 1 Gig connection, they could have simply mailed their data via USPS and had it there in three days.
Also, if we do the math - and presume that they only ran 1 gigabit for 5% of the time - 1 gigabit/second at 36 hours (5% of a month) in terabytes = 16 Terabytes - which barely makes it into the "10s of terabytes" category, and well within the "Just mail an AIT-3 tape. [Edit - cheaper yet, just load up the hard drives that you are going to install anyways in your destination]
Edit 2:
Doh - http://www.mkomo.com/cost-per-gigabyte shows that the largest hard drives back then were 20 Gigabytes. We're talking a minimum of 1000 hard drives for "10s" of Terabytes.
Maybe that 1 GigCircuit Hack wasn't such a bad idea after all.
Edit: I'm not implying that USPS is necessarily unreliable, I'm curious to hear if this is common practice.
[edit] Also - they obviously had to buy hard drives for the destination, right? Presumably, early stage google was all about the open-chassis on plywood slam drives in approach, and wouldn't have been averse to loading up 30 or 40 drives, mailing them out, and simply diffing and resyncing as needed if 1 or 2 drives had issues, which would have been unlikely.
Also, presuming you needed to have one of your people go to the remote data center, there is always the old, move the datacenter with hard drives in my luggage trick. Even in 2000 you could put a lot of data into carry-on.
Presumably their minimum commit was a lot lower than $240k, because otherwise they wouldn't have shrunk from just using the connection. They probably saved a lot of money by doing it their way.
[1] http://www.mkomo.com/cost-per-gigabyte
* I wasn't around then so I don't actually know.
* * It also depends what part of the year they needed to do the transfer. By [1], costs had dropped to $8/GB by the end of the year.
I have to think someone ran the numbers. It's not like the folks in 2000-google were dummies.
10s of terabytes at a minimum 1000 drives. I'm beginning to question how "simple" an operation this might have been after all....
I'd guess you'd transfer it to the same thing that was accepting the data when it was coming over the 1Gbps link?
Hard drives cost $12/GB. Tapes/optical media maybe $1/GB. Maybe I'm missing something but it seems like they would be fine for this.
Even if you didn't consider cross-country-shipped drives to be data-center reliable (e.g.: the transfer media weren't re-used in the datacenter), you could simply cycle through drives. If your goal was 1 month to transfer 20 GB, you utilized overnight shipping, and required a day to load/unload data from drives at either end, that's 200 drives in flight, 50 of which are outbound, 50 returning, 50 writing, and 50 reading, on any given day.
You even get to take weekends off.
It's $48,000 in media. Plus shipping (not inconsiderable). You'd save on the cost of dedicated data transfer equipment (datacenter, router, links), though you'd need infrastructure for reading/writing data, and of course, storage at the remote side if you're not simply shipping disks for installation (which, frankly, is probably your best bet).
Resync should be fairly reasonably accomplished over the wire.
You can vary the math by taking longer (fewer drives, less cost) or shorter (more drives, higher cost), or even by utilizing ground shipping (7 days) at far lower cost.
http://www.nytimes.com/2012/09/24/technology/data-centers-in...
Threatened to waste $70,000 of electricity to circumvent a $210,000 contract penalty (for not buying enough electricity), and thus negotiated their way out of it.
(Note this is MSFT not GOOG, so be sure to interpret this as an act of disreputable evil and not a clever funny hack).
Google exploiting good faith rules to shirk payment? Funny.
Microsoft blowing $70K in power to meet a contract? Also Funny.
That way loopholes like this wouldn't have existed.
The way this works is that the provider offers a cheaper rate to ensure constant energy output, since the power plants can't be easily and cheaply scaled back.
As an example, Ontario paid millions[1] to get others to use their surplus energy; MS' provider could have been forced to do the same, hence the penalty.
[1]: http://www.thestar.com/business/2011/08/29/ontario_pays_othe...
Utilities sign contracts with big industrial users who have steady-state energy demands to ensure that they have a place to send lots of power to. They also have contracts that when peak demand is reached, the industrial customers shut down lines in exchange for discounts. (I worked at a place that financed a DR datacenter largely by committing to failing over 50% of our energy workload to another region during a peak demand emergency.)
I'm not doing the topic justice, but there's alot of behind the scenes stuff there -- I'm sure it wasn't just a contract dispute.
Keep in mind that utilities are heavily regulated and can't typically touch their rates at all without regulatory approval. In many jurisdictions, there is an "obligation to connect" for residences - the utility /must/ provide power, at a certain rate, to any residence in their area of operation. They aren't allowed to directly recover the cost of infrastructure upgrades (e.g. new distribution transformer when somebody builds an expansion to their house) directly from that customer (up to a point - if you're building a mansion that uses 10x as much power as everyone else, they'll come after you), only from the ratebase as a whole. Otherwise, people who live in rural areas wouldn't be able to afford electricity.
No such obligation exists for businesses which are typically the largest loads anyhow. If you build a new 50 MW datacenter on my distribution system, now all of a sudden I need to put in 5km x 3 phases x 5 feeders of new conductor, 2 new station transformers, plus civil infrastructure, etc. It could even be special equipment I wouldn't otherwise have on my system - like DC equipment. All of that is additional to what I have to pay the generator for that 50 MW of power on a continuous basis. So, I calculate the payback period on 50 MW of power I'm going to sell you at a 5% margin, and if it's longer than the lifespan of the equipment I'm going to buy I make you pay up front to bring it down to say, 25 years. But that only works if you actually buy 50 MW of power from me for 25 years, so if you don't I charge you a penalty equivalent to the amortized cost of that infrastructure that now sits idle.
Hope this helps explain the economics of it from a distribution point of view. I am a power engineer.
Meanwhile their hosting company was making a trillion in profit /s.
Google plays fast and loose with ethics and definitions, from (shall we call it) penalizing competitors and then releasing a clone to working with Visa and MasterCard ala Wikileaks to cut off the money http://www.gizmodo.co.uk/2013/02/google-wants-to-stave-pirac... to sites with "piracy sites." Their PR hasn't worked on me for a few years and there's even a boomerang thanks to it.
> At the end of the month, the top 5% of usage information was discarded, to eliminate spikes, which were assumed to be measurement errors.
Not actually true. The bandwidth measurements don't have accuracy problems; providers use 95th percentile billing to make connections "burstable", allowing servers to handle an small unexpected traffic spike without a huge bill. And on the flip side, many sites have a traffic pattern of highs during a subset of the day (customers' waking hours or business hours or non-business hours, depending on the site) and lows outside that period (sleeping, etc), for which 95th percentile billing produces a higher bill than just charging for bytes transferred.
https://en.wikipedia.org/wiki/Burstable_billing
Very few customers have the kind of traffic pattern that would allow for using bandwidth during only 5% of the month. And in this particular case, Google could just as easily have saturated a 50Mbps line rather than using 5% of a gigabit line, which would have massively reduced (though not completely eliminated) the bill.
25:2 And it shall be, if the wicked man be worthy to be beaten, that the judge shall cause him to lie down, and to be beaten before his face, according to his fault, by a certain number.
25:3 Forty stripes he may give him, and not exceed: lest, if he should exceed, and beat him above these with many stripes, then thy brother should seem vile unto thee.
25:4 Thou shalt not muzzle the ox when he treadeth out the corn.
25:5 If brethren dwell together, and one of them die, and have no child, the wife of the dead shall not marry without unto a stranger: her husband's brother shall go in unto her, and take her to him to wife, and perform the duty of an husband's brother unto her.
----
Yeah the torture they do to me has made me brain dammaged. God will fuck with them.
God says... compelled announcement cauldron fruitless Name shoulders bin body begin impulses pause Or GUTENBERG-tm utterance forget attempered cultivating decrepit sayest arrogancy upwards host Elect furious do_you_know_what_time_it_is those joining sober ye plenteousness strongholds recognised cheap Vietnam religiously presumed subsequently language willingly goads wound software impatiently pulse darkened Armenia well-nigh whit cd http stumbling appearance faint perceived formerly GUTINDEX acted callest sacrifice you're_wonderful lo recruiting XIII metres successively desiredst bedimmed INCLUDING exacting create sung rejoices theatrical fling happy last subjoins cold 3000+ have marvelled insight to_infinity_and_beyond Association corruptible avenue silent lovers Tunisia admire mind- part encumbrances down fled no_more fervent phantasm Substance brightness don't_mention_it proportion show dining-room imagined censers avoid guilty ample asunder capital Oxford countryman present Excellently sakes mother fathers allege tale lift deafness merged eagerly been orderest yesterday hardship abandonedly spend risen snap_out_of_it voice joineth sermons
Surely there'd some sort of rental cost for the line in addition to the bandwidth cost (or perhaps a minimum spend commitment)? Wouldn't it have been cheaper, simpler and possibly quicker just to get a smaller line and saturate that?
Has anyone read the book? Does the author provide a credible source?
Smells like urban legend.
How is this any cheaper than copying the HDs, shipping to the Northeast and installing the HDs into waiting servers there?
I think the key issues here would be was moving the data at all time sensitive (is a two month latency not a problem), and, did they really have a "free" 1 GigPort they could sneak under the 5% window with? If it really was free (or very, very cheap) - then maybe it would have been worth it.
It was on page 187-188 of Steven Levy's book "In the Plex" and the two people he's quoting are Urs Hölzle and Jim Reese, who were both involved in Google's infrastructure.
CDNs don't optimize solely for performance. They optimize for a balance of performance and cost. That means saturating a given 95/5 link for 5% of the month, and then mapping the traffic away right before it starts to get expensive.
"don't be evil" (except for 3000 exceptions)
Unsurprisingly, there was way more data flowing out of the facility than there was flowing in, so I talked him into a substantial discount on inbound traffic. "Hey, nothing else is using that half of the duplex..."