I'm 99% sure they are just his style. If you click through his articles on his personal site, you'll see a theme. [0] I think he just likes random 3D generated graphics.
> You’ve reached the end of your free member preview for this month. Become a member now for $5/month to read this story and get unlimited access to all of the best stories on Medium.
Ugh. I didn't realise that Medium had become another paywall.
They have a link to a study of the major internet routes in the US and the name of the PDF is 'tubes_final.pdf'
That made me laugh.
"Long-haul fiber-optic cabling. Red squares mark where cables connect in nodes, many in major population centers. University of Wisconsin and ACM SIGCOMM, 2015. http://pages.cs.wisc.edu/~pb/tubes_final.pdf "
Is there some astroturfing happening with this link? It's at the top of HN with 60 points, only two actual comments about the content, and I personally couldn't find the point of the article.
Was is just a survey of interesting datacenter phenomena? I'm still not sure.
Edit: Amusingly this post had a bunch of points and now they're gone. Maybe people don't find this comment interesting, or maybe the people who were manipulating the vote didn't like getting called out. One shall never know I suppose.
As for their humidity problem, the article says little about how their badly designed chiller-less system got into a condensing situation. The original article in the Register does.[1]
This was an HVAC configuration error. The facility used evaporative cooling, which dumps heat into water by running air through a water spray or some wet surface. This raises the humidity. They somehow got the system configured so that it was recirculating air through the evaporative cooler, increasing the humidity on each pass until it hit saturation. The problem only lasted for a few hours until somebody figured out what was wrong.
They probably didn't have a union-trained stationary engineer on site. This sort of thing is covered in the apprenticeship training for a stationary engineer.
Fans, dampers, and controls can all fail, which can create situations like this.
Stationary engineers deal with those problems. At large data center scale, you may need one.
Regardless of how it happened I'm glad, because now we can say that for a while millions of Facebook users had their data literally stored "in the cloud".
Thank you, the article was really annoying me by talking about everything but the actual problem. Really does sound like somebody tried to cheap out by not hiring the person(s) they needed. I know people that never got past the 8th grade that wouldn't have made such a mistake. They should of at least been monitoring humidity levels when they turned it on and noticed the problem before the entire place got to 100% humidity.
Initially I was a bit confused about the transition from where they were talking about the rain cloud to where they started talking about the history of the Internet, but I enjoyed it nonetheless and I don’t get why the story was flagkilled.
To me it seems that actually the story about the cloud was meant to be an introduction to the remainder of the text.
The story is about how “the cloud” where we put our data is an abstract concept and that even those of us that know about data centers and servers and so on, most of us know very little about the concrete physical locations of every data center and where the individual fiber lines go in the US.
The way the introduction was done however made it seem like the point of the article was to talk about the neurological cloud that had formed in that Facebook data center. And of course they made the matters worse by then picking the title to be about that as well.
But like I said, I think it was an interesting read.
TL;Dr version - author takes an article on data centers, an article on long haul fiber, and an article on the structure of the Internet and puts it into a blender.
It is kind of too bad since two of the three articles haven't had a lot of good coverage. One being alternate cooling strategies for data centers, and the other being disrupting the moats that long haul fiber imposes.
This seems like an egregious oversight to me, no one thought about controlling the humidity levels in this carefully climate controlled server farm beforehand? They used evaporation cooling to lower energy costs which ideally would saturate the air up to near 100% humidity in order to work best, how would you not know taking that air directly would result in water and humidity problems? They likely had multiple engineers working on climate control by itself, someone should have foreseen it.
Of course the article isn't about the actual cloud in the actual data center.
Pipkin is a code artist, this is essentially a philosophical musing, but of the continental kind, with importance given to phenomenology, with thought on metaphilosophy and suchlike, with philosophy given as first science.
This kind of author specializes in deeper critique in that continental style. Most such works are technically worthless, but that is not the point.
It isn't just technically worthless, it is technically wrong. I know that none of it is even the point, but that's where the article loses itself: there is no point but to make an appealing critique of nothing in particular.
The much better (and probably worthy of a modern spiritual successor, which this article is not) version of this theme is Neal Stephenson's excellent "Mother Earth Mother Board" from a 1996 issue of /Wired/:
This reminds me of something that happened in our on-site datacenter at a former employer of mine. It wasn't raining inside, they were just doing routine maintenance on the cooling system. Somehow, they managed to dump all of the cooling water into the offices (which were not ours) in the 2-3 immediate floors below. Needless to say, there was an emergency shutdown as the datacenter overheated. It was definitely an all-hands scenario as we got ready for market open the next day. I think I crashed around 4am-5am after my servers were back up and checked out.
I had my small basement (shared with the print department) computer room flooded in London - this was the second room the one before was flooded and was repurposed as a printer room - it still had the flood marks on the wall.
> Furthermore, much of this energy consumption is generated by coal — even ‘green companies’ will often buy carbon offset credits rather than invest in the energy storage required for 24/7 solar or wind power.
Plain bullshit. One of the reasons Facebook and Google have datacenters in Oregon (FB at Prineville and Google at The Dalles) is the availability of cheap renewable energy from the Columbia River.
And the companies don't buy carbon offset. The author has no idea how electricity markets work. They buy renewable electricity contracts, for example:
I was at Google and they did buy offsets from 2007 until after I left. Not for Oregon, sure, but definitely for places like the Carolinas (our good old friends at Duke Energy...). It was only in 2017 that they covered 100% of corporate/datacenter electricity with actual contracts.
Speechless (occurrence year 2011). Was it really just lack of appropriate personnel or design flaw. I just can't get that information out of the article. Someone cut corners.
Now, the article lacks focus. It expands unecessarily on every tangent (it reached ARPANET -- drops mic). 2 or 3 pages could have been 2-3 paragraphs or references. This article lacks an editor. (Or they are payed per page -- I think it is word count actually.)
The author could state: they pick remote locations, because it is cheap and away from publicity (power hungry) and less likely to become a target. Then argue and clarify with some details. Then go to the destructive events perhaps.
Like many others, I am not sure of the goal of the article to be honest.
I'm always amazed when I think of a big data center in Prineville. I'm currently reading a book on the history of the area: https://amzn.to/2KLxqyM , since I live down the road in Bend. Pretty much your basic small western town, with native americans, range wars, and a murderous group of vigilantes in its past and a popular rodeo in this day and age.
Nice place to visit if you ever get the chance, on the way to the Painted Hills for instance.
It definitely follows the tracks. Using existing tracks provided to major advantages: the railroads already had the right-of-way, and it was relatively affordable to lay the fiber by equipping a locomotive with a special trenching attachment. (A fiber optic rail plow).
Just noticed on the fiber map contained in the article that there isn't any fiber in Michigan's upper peninsula.
How is that even remotely possible? It appears that it runs up to to Mackinac City and then stops short of the bridge. The map doesn't even show the U.P.
I was at a tech conference once that had attendees states on their badges. A lady came up to me and said how cute you've got a deer jumping over the state of Michigan on your badge. Told her that's not a deer, it's part of the state ;<).
> like a thunderstorm formed from two opposing weather fronts, the physicality of outside Prineville air met the everywhere-ness of inside space in an impossible, non-Euclidean intersection
Not every word is a pretty one you can throw around, you know: what exactly about this metaphorical meeting was "non-Euclidean"?
The metric that connects data centers is not Euclidean. There are some names of that metric but for that story the name "railroad metric" might fit best.
54 comments
[ 4.0 ms ] story [ 53.0 ms ] thread[0] http://everest-pipkin.com/
Ugh. I didn't realise that Medium had become another paywall.
:)
That made me laugh.
"Long-haul fiber-optic cabling. Red squares mark where cables connect in nodes, many in major population centers. University of Wisconsin and ACM SIGCOMM, 2015. http://pages.cs.wisc.edu/~pb/tubes_final.pdf "
http://pages.cs.wisc.edu/~pb/tubes_final.pdf
http://pages.cs.wisc.edu/~pb/tubes_final.pdf
Was is just a survey of interesting datacenter phenomena? I'm still not sure.
Edit: Amusingly this post had a bunch of points and now they're gone. Maybe people don't find this comment interesting, or maybe the people who were manipulating the vote didn't like getting called out. One shall never know I suppose.
As for their humidity problem, the article says little about how their badly designed chiller-less system got into a condensing situation. The original article in the Register does.[1]
This was an HVAC configuration error. The facility used evaporative cooling, which dumps heat into water by running air through a water spray or some wet surface. This raises the humidity. They somehow got the system configured so that it was recirculating air through the evaporative cooler, increasing the humidity on each pass until it hit saturation. The problem only lasted for a few hours until somebody figured out what was wrong.
They probably didn't have a union-trained stationary engineer on site. This sort of thing is covered in the apprenticeship training for a stationary engineer. Fans, dampers, and controls can all fail, which can create situations like this. Stationary engineers deal with those problems. At large data center scale, you may need one.
[1] http://www.theregister.co.uk/2013/06/08/facebook_cloud_versu...
Edit: It's a mediocre poem.
To me it seems that actually the story about the cloud was meant to be an introduction to the remainder of the text.
The story is about how “the cloud” where we put our data is an abstract concept and that even those of us that know about data centers and servers and so on, most of us know very little about the concrete physical locations of every data center and where the individual fiber lines go in the US.
The way the introduction was done however made it seem like the point of the article was to talk about the neurological cloud that had formed in that Facebook data center. And of course they made the matters worse by then picking the title to be about that as well.
But like I said, I think it was an interesting read.
It is kind of too bad since two of the three articles haven't had a lot of good coverage. One being alternate cooling strategies for data centers, and the other being disrupting the moats that long haul fiber imposes.
Pipkin is a code artist, this is essentially a philosophical musing, but of the continental kind, with importance given to phenomenology, with thought on metaphilosophy and suchlike, with philosophy given as first science.
This kind of author specializes in deeper critique in that continental style. Most such works are technically worthless, but that is not the point.
It isn't even wrong.
Also, the graphics are really bad, and completely distract from the already questionable piece.
https://www.wired.com/1996/12/ffglass/
Plain bullshit. One of the reasons Facebook and Google have datacenters in Oregon (FB at Prineville and Google at The Dalles) is the availability of cheap renewable energy from the Columbia River.
And the companies don't buy carbon offset. The author has no idea how electricity markets work. They buy renewable electricity contracts, for example:
http://www.datacenterdynamics.com/content-tracks/power-cooli...
The author seems more preoccupied with the sesquipedalian loquaciousness than with actual facts.
Now, the article lacks focus. It expands unecessarily on every tangent (it reached ARPANET -- drops mic). 2 or 3 pages could have been 2-3 paragraphs or references. This article lacks an editor. (Or they are payed per page -- I think it is word count actually.)
The author could state: they pick remote locations, because it is cheap and away from publicity (power hungry) and less likely to become a target. Then argue and clarify with some details. Then go to the destructive events perhaps.
Like many others, I am not sure of the goal of the article to be honest.
Nice place to visit if you ever get the chance, on the way to the Painted Hills for instance.
In the UK there are canals trains tracks and motorway running next door as the geography suits.
How is that even remotely possible? It appears that it runs up to to Mackinac City and then stops short of the bridge. The map doesn't even show the U.P.
I was at a tech conference once that had attendees states on their badges. A lady came up to me and said how cute you've got a deer jumping over the state of Michigan on your badge. Told her that's not a deer, it's part of the state ;<).
Actually, it honestly reads like a prequel to Asimov's short story, "The Last Question". [1]
[1] http://www.multivax.com/last_question.html
Not every word is a pretty one you can throw around, you know: what exactly about this metaphorical meeting was "non-Euclidean"?