Everything is dual redundancy. We run RAID so if a drive fails it's fine; alerting will page oncall which will trigger remote hands onsite, where we have spares for everything in each datacenter
We built some internal tooling to help manage the hosts. Once a host is onboarded onto it, it's a few button clicks on an internal dashboard to provision a QEMU VM. We made a custom ansible inventory plugin so we can manage these VMs the same as we do machines on GCP.
The host runs a custom daemon that programs FRR (an OSS routing stack), so that it advertises addresses assigned to a VM to the rest of the cluster via BGP. So zero config of network switches, etc... required after initial setup.
We'll blog about this system at some point in the coming months.
I thought it was an interesting post, so I tried to add Railway's blog to my RSS reader... but it didn't work. I tried searching the page source for RSS and also found nothing. Eventually, I noticed the RSS icon in the top right, but it's some kind of special button that I can't right click and copy the link from, and Safari prevents me from knowing what the URL is... so I had to open that from Firefox to find it.
How did you select the hardware?
Did you do a bake off/poc with different vendors?
With the intention of being in different countries, are you going to leverage the same hardware at every DC?
What level of support SLA did you go with for your hardware vendors and the colo facilities?
And my favorite, how are your finances changing (plus pros cons) by going capex vs opex?
The requirements end up being pretty specific, based on workloads/power draw/supply chain
So, while we could have bought something off the shelf, that would have been suboptimal from a specs perspective. Plus then we'd have to source supply chain etc.
By owning not just the servers but the whole supply chain, we have redundancy at every layer, from the machine, to the parts on site (for failures), to the supply chain (refilling those spare parts/expanding capacity/etc)
Kinda. It's like if you had everything from an infra stack but didn't need to manage it (Kubernetes for resilience, Argo for rollouts, Terraform for safely evolving infrastructure, DataDog for observability)
If you've heard of serverless, this is one step farther; infraless
Give us your code, we will spin it up, keep it up, automate rollouts service discovery, cluster scaling, monitoring, etc
I've been using railway since 2022 and it's been great. I host all my personal projects there and I can go from code to a url by copy-pasting my single dockerfile around.
We didn't find many good up-to-date resources online on the hardware side of things - kinda why we wanted to write about it. The networking aspect was the most mystical - I highly recommend "BGP in the datacenter" by Dinesh Dutt on that (I think it's available for free via NVidia). Our design is heavily influenced by the ideas discussed there.
We talked to a few, I think they're called MSPs? We weren't super impressed. We decided to YOLO it. There are probably great outfits out there, but it's hard to find them through the noise. We're mostly software and systems folks, but Railway is a infrastructure company so we need to own stuff down to the cage-nut - we owe it to our users. All engineering, project management and procurement is in-house.
We're lucky to have a few great distributors/manufacturers who help us pick the right gear. But we learnt a lot.
We've found a lot of value in getting a broker in to source our transit though.
My personal (and potentially misguided) hot take is that most of the baremetal world is stuck in the early 2000's, and the only companies doing anything interesting here the likes of AWS,Google and Meta. So the only way to innovate is to stumble around, escape the norms and experiment.
We're blessed with some kickass investors. They gave us just the right level of scrutiny. We were super clear about why we wanted to do this, we did it, and then they invested more money shortly after the first workloads starting running on metal
If you're looking for great partners, who actually have the gal to back innovation, you'd be hard pressed to do better than Redpoint (Shoutout Erica and Jordan!)
Why would you call colocation "building your own data center"? You could call it "colocation" or "renting space in a data center". What are you building? You're racking. Can you say what you mean?
Dealing with power at that scale, arranging your own ISPs, seems a bit beyond your normal colocation project, but I haven’t bee in the data center space in a very long time.
One of the many reasons we went with Switch for our DC is because they have a service to handle all of that for you. Having stumbled on doing this ourselves before, it can be pretty tricky to negotiate everything.
We had one provider give us a great price and then bait and switch at the last moment to tell us that there is some other massive installation charge that they didn't realize we had to pay.
Switch Connect/Core is based off the old Enron business that Rob (CEO) bought...
I have to second this. While it takes mich effort and in-depth knowledge do build up from an “empty” cage it’s still far from dealing with everything from building permits, to plan and realize a data center to code including redundant power lines, AC and fibre.
Still kudos going this path in the cloud-centric time we live in.
> Yes, the second is much more work, orders of magnitude at least.
I feel it's important to stress that the difficulty level of collocating something, let alone actually building a data center, is exactly what makes cloud computing so enticing and popular.
Everyone focuses on trivia items like OpEx vs CapEx and dynamic scaling, but the massive task of actually plugging in the hardware in a secure setting and get it to work reliably is a massive undertaking.
I just honestly don't agree with that at all. That's the easy bit, the bit I don't enjoy is organising backups and storage in general. But it's not 'hard'.
One of the better was the dead possum in the drain during a thunderstorm.
>So do we throw the main switch before we get electroduced? Or do we try to poke enough holes in it that it gets flushed out? And what about the half million in servers that are going to get ruined?
Sign up to my patreon to find out how the story ended.
While it is more complex to actually build out the center , a lot of that is specific to the regional you are doing it.
Thy will vary by country, by state or even county , setting up a DC in the Bay Area and say one in Ohio or Utah is a very different endeavor with different design considerations.
>Thy will vary by country, by state or even county , setting up a DC in the Bay Area and say one in Ohio or Utah is a very different endeavor with different design considerations.
What point are you trying to make? It does not matter where you are in the world, or what local laws exist or permits are required, racking up servers in a cage is much less difficult than physically building a data center (of which racking up servers is a part).
I meant that the learning from doing actual build outs aren't going to translate in other geographies and regulatory climates, not that the work is less difficult or not interesting and important.
Also people doing the build outs of a DC aren't likely keen on talking about permits and confidential agreements in the industry quite publicly.
Yes the title is click baity, but that is par of the course these days.
Sure, every business has confidential agreements which are usually kept secret but there are even on youtube a few people/companies who gave deep insides in the bits and bytes of building a data center from ground up across multiple hours of documentation. And the confidential business agreements in the data center world are up to a certain level the same as any other businesses.
> Thy will vary by country, by state or even county , setting up a DC in the Bay Area and say one in Ohio or Utah is a very different endeavor with different design considerations.
Regarding data centers that cost 9 figures and up:
For the largest players, there’s not a ton of variation. A combination of evaporative cooling towers and chillers are used to reject heat. This is a consequence of evaporative open loop cooling being 2-3x more efficient than a closed-loop system.
There will be multiple medium-voltage electrical services, usually from different utilities or substations, with backup generators and UPSes and paralleling switchgear to handle failover between normal, emergency, and critical power sources.
There’s not a lot of variation since the two main needs of a data center are reliable electricity and the ability to remove heat from the space, and those are well-solved problems in mature engineering disciplines (ME and EE). The huge players are plopping these all across the country and repeatability/reliability is more important than tailoring the build to the local climate.
FWIW my employer has done billions of dollars of data center construction work for some of the largest tech companies (members of Mag7) and I’ve reviewed construction plans for multiple data centers.
You've got more experience there than me, and I've only seen the plans for a single center.
I'll point out that some of the key thermal and power stuff in those plans you saw may have come from the hyperscalers themselves - our experience a dozen years or so ago was that we couldn't just put it out to bid, as the typical big construction players knew how to build old data centers, not new ones, and we had to hire a (very small) engineering team to design it ourselves.
Heat removal is well-solved in theory. Heat removal from a large office building is well-solved in practice - lots of people know exactly what equipment is needed, how to size, install, and control it, what building features are needed for it, etc. Take some expert MEs without prior experience at this, toss them a few product catalogs, and ask them to design a solution from first principles using the systems available and it wouldn't be so easy.
There are people for whom data center heat removal is a solved problem in practice, although maybe not in the same way because the goalposts keep moving (e.g. watts per rack). Things may be different now, but a while back very few of those people were employed by companies who would be willing to work on datacenters they didn't own themselves.
Finally I'd add that "9 figures" seems excessive for building+power+cooling, unless you're talking crazy sizes (100MW?). If you're including the contents, then of course they're insanely expensive.
Issues in building your own physical data center (based on a 15MW location some people I know built):
1 - thermal. To get your PUE down below say 1.2 you need to do things like hot aisle containment or better yet water cooling - the hotter your heat, the cheaper it is to get rid of.[]
2 - power distribution. How much power do you waste getting it to your machines? Can you run them on 220v, so their power supplies are more efficient?
3 - power. You don't just call your utility company and as them to run 10+MW from the street to your building.
4 - networking. You'll probably need redundant dark fiber running somewhere.
1 and 2 are independent of regulatory domain. 3 involves utilities, not governments, and is probably a clusterfck anywhere; 4 isn't as bad (anywhere in the US; not sure elsewhere) because it's not a monopoly, and you can probably find someone to say "yes" for a high enough price.
There are people everywhere who are experts in site acquisition, permits, etc. Not so many who know how to build the thermals and power, and who aren't employed by hyperscalers who don't let them moonlight. And depending on your geographic location, getting those megawatts from your utility may be flat out impossible.
This assumes a new build. Retrofitting an existing building probably ranges from difficult to impossible, unless you're really lucky in your choice of building.
[*] hmm, the one geographic issue I can think of is water availability. If you can't get enough water to run evaporative coolers, that might be a problem - e.g. dumping 10MW into the air requires boiling off I think somewhere around 100K gallons of water a day.
> Why would you call colocation "building your own data center"?
The cynic in me says this was written by sales/marketing people targeted specifically at a whole new generation of people who've never laid hands on the bare metal or racked a piece of equipment or done low voltage cabling, fiber cabling, and "plug this into A and B power AC power" cabling.
By this, I mean people who've never done anything that isn't GCP, Azure, AWS, etc. Many terminologies related to bare metal infrastructure are misused by people who haven't been around in the industry long enough to have been required to DIY all their own infrastructure on their own bare metal.
I really don't mean any insult to people reading this who've only ever touched the software side, but if a document is describing the general concept of hot aisles and cold aisles to an audience in such a way that it assumes they don't know what those are, it's at a very introductory/beginner level of understanding the OSI layer 1 infrastructure.
I think that's my fault BTW (Railway Founder here). I asked Charith to cut down a bit on the details to make sure it was approachable to a wider audience (And most people have only done Cloud)
I wanted to start off with the 101 content to see if people found it approachable/interesting. He's got like reams and reams of 201, 301, 401
Sitting on the front page of HN with a good read, and what is ultimately company promo and a careers link seems like a job well done. It made me read/click.
Yes, building a physical DC is much wider scope than colo. This is one part of that, which is also still interesting. The world is built on many, many layers of abstraction which can all take lifetimes to explore. There are non-devs who enjoy learning about software, web-devs who dabble in compilers, systems programmers curious about silicon, EE's that are aspiring physicists, who in turn peek into the universe of pure path (cue yes, that xkcd you're thinking of).
A 'full stack' overview of a standalone DC build still has to set a bound somewhere. This was an approachable intro and look forward to reading more from the layers you operate.
I mean the more people realize the the cloud is now a bad deal the better.
When the original aws instance came out it would take you about two years or on demand to pay for the same hardware on prem. Now its between two weeks for ml heavy instances to six months for medium CPU instances.
It just doesn't make sence to use the cloud for anything past prototyping unless you want Bazos to have a bigger yacth.
> You could call it "colocation" or "renting space in a data center". What are you building? You're racking. Can you say what you mean?
TFA explain what they're doing, they literally write this:
"In general you have three main choices: Greenfield buildout (...), Cage Colocation (getting a private space inside a provider's datacenter enclosed by mesh walls), or Rack colocation...
The title is "So you want to build your own data center" and the article is about something else. Its nice that they say that up front, but its valid to criticize the title.
Only one of those options is ‘building your own data center’, and I’ll give you three guesses as to which one it is. I’ll even give you a hint: ‘greenfield’ is in the correct answer.
It seems a bit disingenuous but it’s common practice. Even the hyperscalers, who do have their own datacenters, include their colocation servers in the term “datacenter.” Good luck finding the actual, physical location of a server in GCP europe-west2-a (“London”). Maybe it’s in a real Google datacenter in London! Or it could be in an Equinix datacenter in Slough, one room away from AWS eu-west-1.
Cloudflare has also historically used “datacenter” to refer to their rack deployments.
All that said, for the purpose of the blog post, “building your own datacenter” is misleading.
The hyperscalers are absolutely not colo-ing their general purpose compute at Equinix! A cage for routers and direct connect, maybe some limited Edge CDN/compute at most.
Even where they do lease wholesale space, you'd be hard pushed to find examples of more than one in a single building. If you count them as Microsoft, Google, AWS then I'm not sure I can think of a single example off the top of my head. Only really possible if you start including players like IBM or Oracle in that list.
Google doesn't put GCP compute inside Equinx Slough. I could perhaps believe if they have a cage of routers and perhaps even CDN boxes/Edge, but no general cloud compute.
Google and AWS will put routers inside Equinx Slough sure, but that's literally written on the tin, and the only way a carrier hotel could work.
Security reasons, I presume? Otherwise it would be trivial for an adversary to map out their resources by sampling VM rentals over a moderate time-period.
The best part about adamantly making such a claim is that anybody who knows better also knows better than to break NDA and pull a Warthunder to prove that the CSPs do use colo facilities, so you're not going to get anyone who knows better to disagree with you and say AWS S3 or GCP compute is colo-ed at a specific colo provider.
You're correct, there are multiple flavors of Google Cloud Locations. The "Google concrete" ones are listed at google.com/datacenters and London isn't on that list, today.
cloud.google.com/about/locations lists all the locations that GCE offers service, which is a super set of the large facilities that someone would call a "Google Datacenter". I liked to mostly refer to the distinction as Google concrete (we built the building) or not. Ultimately, even in locations that are shared colo spaces, or rented, it's still Google putting custom racks there, integrating into the network and services, etc. So from a customer perspective, you should pick the right location for you. If that happens to be in a facility where Google poured the concrete, great! If not, it's not the end of the world.
> These data centers might be owned by Google and listed on the Google Cloud locations page, or they might be leased from third-party data center providers. For the full list of data center locations for Google Cloud, see our ISO/IEC 27001 certificate. Regardless of whether the data center is owned or leased, Google Cloud selects data centers and designs its infrastructure to provide a uniform level of performance, security, and reliability.
So someone can probably use web.archive.org to get the ISO-27001 certificate PDF from whenever the last time it was still up.
> P.S., I swear the certification PDFs used to include this information (e.g., https://cloud.google.com/security/compliance/iso-27018?hl=en) but now these are all behind "Contact Sales" and some new Certification Manager page in the console.
This is not good, I can't think of any actual reason to hide those certificates.
For comparison, AWS makes their ISO-27001 certificate available at https://aws.amazon.com/compliance/iso-27001-faqs/ and also cites the certifying agent, most of which have a search page from where you can find all the certificates they've issued.
> Which makes you a subletter, and the one with the highest fee of the whole chain…
I don't know what point you tried to make. Any business in the whole world survives because they sell things for more money than what it takes to keep their business running. Is it surprising that they charge their customers more than their infrastructure costs?
> It seems a bit disingenuous but it’s common practice. Even the hyperscalers, who do have their own datacenters, include their colocation servers in the term “datacenter.”
I think you're conflating things.
Those hypothetical hyperscalers can advertise their availability zones and deployment regions, but they do not claim they built the data centers. They provide a service, but they do not make broad claims on how they built infrastructure.
They usually don’t say they are building their own datacenter, though. It is different to say something like, “our website runs in our datacenter” than saying you built a datacenter. You would still say, “at our office buildings”, even if you are only renting a few offices in an office park.
Don't the hyperscalers outsource datacenter construction and operation? Maybe it's not clear where to draw the line because the datacenters are owned or operated by disposable shell companies for various reasons.
Come to my office and tell me how it’s not actually my office because it’s leased by my company from the investment vehicle for institutional investors that owns the building that stands on land owned by someone else again that was stolen by the British anyway and therefore calling it “my office” makes me a fool and a liar and I should just “say what I mean”.
When you invite a girl/guy over, do you say "let's meet at my place" or "let's meet at the place I'm renting"? The possessive pronoun does not necessarily express ownership, it can just as well express occupancy.
I wouldn't oppose telling a client "we can meet at your data centre". I would not tell my wife "we need to discuss building our apartment complex" when we are planning interior decorations in our flat.
If I said to my wife, “let’s build a home together”, she would be halfway done with engaging a promising firm of radical young architects and negotiating for April delivery of pre-stressed concrete, Italian art glass, and Japanese tatami mats by close of business.
> Come to my office and tell me how it’s not actually my office (...)
I think you're failing to understand the meaning and the point of "building your own datacenter".
Yes, you can talk about your office all you'd like. Much like OP can talk about there server farm and their backend infrastructure.
What you cannot talk about is your own office center. You do not own it. You rent office space. You only have a small fraction of the work required to operate an office, because you effectively offloaded the hard part to your landlord.
I think the word GP is objecting to isn't "your own" but rather "build".
For people who have taken empty lots and constructed new data centers (ie, the whole building) on them from scratch, the phrase "building a datacenter" involves a nonzero amount of concrete.
OP seems to have built out a data hall - which is still a cool thing in its own right! - but for someone like me who's interested in "baking an apple pie from scratch", the mismatch between the title and the content was slightly disappointing.
It doesn't matter which word. Which I should confess makes my remark above appear, in retrospect, to be something of a trap; because when parsing ambiguity, it's a matter of simple courtesy and wisdom to choose the interpretation that best illustrates the point rather than complaining about the ones that don't.
I say this not merely to be a pompous smartass but also because it illustrates and echoes the very same problem the top-level comment embodies, viz. that some folks struggle with vernacular, nonliteral, imprecise, and nonlinear language constructs. Yet grasping this thistle to glark one's grok remains parcel-and-part of comprehension and complaining about it won't remeaningify the barb'd disapprehensible.
Your disappointment, nevertheless, seems reasonable, because the outcome was, after all, a bait-and-route.
It would be nice to have a lot more detail. The WTF sections are the best part. Sounds like your gear needs "this side towards enemy" sign and/or the right affordances so it only goes in one way.
Did you standardize on layout at the rack level? What poke-yoke processes did you put into place to prevent mistakes?
What does your metal->boot stack look like?
Having worked for two different cloud providers and built my own internal clouds with PXE booted hosts, I too find this stuff fascinating.
Also take utmost advantage of a new DC when you are booting it to try out all the failure scenarios you can think of and the ones you can't through randomized fault injection.
I'm going to save this for when I'm asked to cut the three paras on power circuit types.
Re: standardising layout at the rack level; we do now! we only figured this out after site #2. It makes everything so much easier to verify. And yeah, validation is hard - manually doing it thus far; want to play around with scraping LLDP data but our switch software stack has a bug :/. It's an evolving process, the more we work with different contractors, the more edge cases we unearth and account for. The biggest improvement is that we have built a internal DCIM that templates a rack design and exports a interactive "cabling explorer" for the site techs - including detailed annotated diagrams of equipment showing port names, etc... The screenshot of the elevation is a screenshot of part of that tool.
> What does your metal->boot stack look like?
We've hacked together something on top of https://github.com/danderson/netboot/tree/main/pixiecore that serves a debian netboot + preseed file. We have some custom temporal workers to connect to Redfish APIs on the BMCs to puppeteer the contraption. Then a custom host agent to provision QEMU VMs and advertise assigned IPs via BGP (using FRR) from the host.
Re: new DCs for failure scenarios, yeah we've already blown breakers etc... testing stuff (that's how we figured out our phase balancing was off). Went in with a thermal camera on another. A site in AMS is coming up next week and the goal for that is to see how far we can push a fully loaded switch fabric.
The edge cases are the gold btw, collect the whole set and keep them in a human and machine readable format.
I'd also go through and using a color coded set of cables, insert bad cables (one at a time at first) while the system is doing an aggressive all to all workload and see how quickly you can identify faults.
It is the gray failures that will bring the system down, often multiple as a single failure will go undetected for months and then finally tip over an inflection point at a later time.
Are you workloads ephemeral and/or do they live migrate? Or will physical hosts have long uptimes? It is nice to be able to rebaseline the hardware before and after host kernel upgrades so you can detect any anomalies.
You would be surprised about how larger of a systemic performance degradation that major cloud providers have been able to see over months because "all machines are the same", high precision but low absolute accuracy. It is nice to run the same benchmarks on bare metal and then again under virtualization.
I am sure you know, but you are running a multivariate longitudinal experiment, science the shit out of it.
Long running hosts at the moment, but we can drain most workloads off a specific host/rack if required and reschedule it pretty fast. We have the advantage of having a custom scheduler/orchestrator we've been working on for years, so we have a lot of control on that layer than with Kube or Nomad.
Re: Live Migration
We're working on adding Live Migration support to our orchestrator atm. We aim to have it running this quarter. That'll makes things super seamless.
Re: kernels
We've already seen some perf improvements somewhere between 6.0 and 6.5 (I forget the exact reason/version) - but it was some fix specific to the Sapphire Rapids cpus we had. But I wish we had more time to science on it, it's really fun playing with all the knobs and benchmarking stuff. Some of the telemetry on the new CPUs is also crazy - there's stuff like Intel PCM that can pull super fine-grained telemetry direct from the CPU/chipset https://github.com/intel/pcm. Only used it to confirm that we got NUMA affinity right so far - nothing crazy.
You will need a way to coordinate LM with users due them being sensitive to LM blackouts. Not many workloads are, but the ones that are are the kinds of things that customers will just leave over.
If you are draining a host, make sure new VMs are on hosts that can be guaranteed to be maintenance free for the next x-days. This allows customers to restart their workloads on their schedule and have a guarantee that they won't be impacted. It also encourages good hygiene.
Allow customers to trigger migration.
Charge extra for a long running maintenance free host.
It is good you are hooked into the PCM already. You will experience accidentally antagonistic workloads and the PCM will really help debug those issues.
If I were building a DC, I put as many NICs into a host as possible and use SR-VIO to pass the nics into the guests. The switches should be sized to allow for full speed on all nics. I know it sounds crazy but if you design for a typical crud serving tree, you are a saving a buck but making your software problem 100x harder.
Everything should have enough headroom so it never hits a knee of a contention curve.
We have a distributor we work with - just because it makes import/export a lot easier. But we get to interface directly with Supermicro for the technical/design stuff, and they're super awesome. If you're looking in the US, reach out to their eStore - really great fuss-free turnaround and all direct.
oh yes we want to; I even priced a couple out. Most of the SKUs I found were pretty old, and we couldn't find anything compelling to risk deploying at the scale we wanted.
It's on the wishlist, and if the right hardware comes along; we'll rack it up even as a bet. We maintain Nixpacks (https://nixpacks.com/docs/getting-started), so for most of our users we could rebuild most their apps for ARM seamlessly - infact we mostly develop our build systems on ARM (because macbooks). One day.
I remember talking to Jake a couple of years ago when they were looking for someone with a storage background. Cool dude, and cool set of people. Really chuffed to see them doing what they believe in.
This is a pretty decent write up. One thing that comes to mind is why would you write your own internal tooling for managing a rack when Netbox exists? Netbox is fantastic and I wish I had this back in the mid 2000s when I was managing 50+ racks.
we evaluated a lot of commercial and oss offerings before we decided do go build it ourselves - we still have a deploy of netbox somewhere. But our custom tool (Railyard) works so well because it integrates deeply into the our full software, hardware and orchestration stack. The problem with the OSS stuff is that it's almost too generic - you shape the problem to fit its data model vs. solve the problem. We're likely going to fold our tool into Railway itself eventually - want to go on-prem; button click hardware design, commission, deploy and devex. Sorta like what Oxide is doing, but approaching the problem from the opposite side.
It is not that difficult to build it into your app, if you're already storing information about hosts, networking etc. All you're really doing is expanding the scope, netbox is a fine starting point if you're willing to start there and build your systems around it, but if you've already got a system (or you need to do anything that doesn't fit netbox logic) you're probably better off just extending it.
In this case railway will need to care about a lot of extra information beyond just racks, IP addresses and physical servers.
correct; I think the first version of our tool sprung up in the space of a couple of weekends. It wasn't planned, my colleague Pierre who wrote it just had a lot of fun building it.
There's a fork called nautobot that tries to add-in automation. Most things we wanted to do with either meant we had to go writing django plugins and trying to interface with their APIs (and fight with the libs). Overall just hammering together a small custom service ended up being way faster/simpler.
Note how they want to be "NetBox functions as the source of truth for your network infrastructure."
Your individual situation dictates what is important, but had netbox targeted being a central repository vs insisting on not allow other systems to be truthful for certain items it could be a different story.
We have learned that trying to centralize complexity and control doesn't work, heck we knew that almost immediately after the Clinger Cohen Act passed and even ITIL and TOGAF fully call this out now and I expect this to be targeted by consultants over the next few years.
You need a central constant way to find state, to remove any questions or doubt regarding where to find the authoritative information, but generally if you aspire to scale and grow or adapt to new changes you really need to avoid having some centralized, god-box, and prescriptive system like this.
Netbox is crap unless you are trying to manage a small but very heterogeneous environment. For anything big, very homogeneous etc you really don't want it.
It feels more like an OSS tool for managing university campus scale infra, which is completely fine if that is the problem you have but for commercial scale infrastructure unfortunately there isn't a good OOTB DCIM option right now.
I would be super interested to know how this stuff scales physically - how much hardware ended up in that cage (maybe in Cloud-equivalent terms), and how much does it cost to run now that it's set up?
More to learn from the failures than the blog haha. It tells you what the risks are with a colocation facility. There really isn't any text on how to do this stuff. The last time I wanted to build out a rack there aren't even any instructions on how to do cable management well. It's sort of learned by apprenticeship and practice.
Reminds me of the old Rackspace days! Boy we had some war stories:
- Some EMC guys came to install a storage device for us to test... and tripped over each other and knocked out an entire Rack of servers like a comedy skit. (They uh... didn't win the contract.)
- Some poor guy driving a truck had a heart attack and the crash took our DFW datecenter offline. (There were ballards to prevent this sort of scenario, but the cement hadn't been poured in them yet.)
- At one point we temporarily laser-beamed bandwidth across the street to another building
- There was one day we knocked out windows and purchased box fans because servers were literally catching on fire.
Data center science has... well improved since the earlier days. We worked with Facebook on the OpenCompute Project that had some very forward looking infra concepts at the time.
The datacenters I've been in with emergency cooling fans in the walls all exhaust out, not in. Easier to get portable CRACs inside and get a good draft going.
When I was in college I’d call up datacenters pretending to be a prospective customer and schedule a tour. I was totally fascinated by them and knew enough to sound legit, it was like going to an amusement park for me.
We had a bird land on a transformer up on a pole and blew fuses. A couple years later, I toured the facility and the fried carcass was still there on the ground below it.
> Data center science has... well improved since the earlier days
You say that, but...
> There was one day we knocked out windows and purchased box fans because servers were literally catching on fire
This happened to Equinix's CH1 datacenter in Chicago Jan24 (not the literal fire part). Took down Azure ExpressRoute.
Apparently it got too cold and the CRACs couldn't take it? I'm told they had all the doors and windows open trying to keep things cold enough, but alas. As the CRAC goes, so goes the servers
running European ISPs in summer we’d nick desk fans off the telco folks to cool down our walls of USR Sportsters, distracting them first with snarky remarks about ATM overhead
Manufacturing is always about 25 years behind the times. I made good scratch in the '00s helping manufactures with their DEC PDP-11 and DG Novas (from the 70s).
Many years ago I had a BlackDiamond dropped on my foot during installation at INTX LON1 for LINX, disabling me for hours. The switch in question was evidently cursed: later that week a spanning tree misconfiguration on the same unit then disabled LINX for hours, throwing half of Britain's ISP peering into temporary chaos, and everyone else involved in that project was dead within two years.
In the bad old days I had a server at blue host in Dallas. Went to the dc once and there extension cords accross the racks suspended about 1ft off the ground that I had to step over to get to my server. Hey at least it was cheap :)
Once worked in a "DC" in a converted cow shed in the English countryside. Hot takes that align with your experiences:
- A key microwave link kept going down with intermittent packet errors way down in the data link layer. A short investigation discovered that a tree across the road had come into leaf, and a branch was blowing into the line of sight of the kit on our building. A step-ladder, a saw and 10 minutes later we restored connectivity
- Our main (BGP-ified) router out of the DC - no, there wasn't a redundant device - kept rebooting. A quick check showed the temp in the DC was so high, cooling so poor, that the *inlet* fan had an air temp of over 60C. We pointed some fans at it as a temporary measure.
- In a similar vein, a few weeks later the air con in another room just gave up and started spewing water over the Nortel DMS-100 (we were a dial-in ISP with our own switch). Wasn't too happy to be asked to help mop it up (thought the water could potentially be live), but what to do?
After that experience I spent time on a small, remote island where main link to the internet was a 1MB/sec link vis GS satellite (ping times > 500ms), and where the locals dialled in over a microwave phone network rated to 9600 baud, but somehow 56k modems worked... One fix I realised I needed was a Solaris box was missing a critical .so, there were no local backups or install media and so I phoned my mate back in the UK and asked him to whack up a copy on an FTP server for me to get the box back online.
And a few years after that I also got to commission a laser beam link over Manchester's Oxford Road (at the time, the busiest bus route in Europe), to link up an office to a University campus. Fun times.
It was all terrific fun, but I'm so glad I now only really do software.
> It was all terrific fun, but I'm so glad I now only really do software.
I don't blame you, a lot of us had to do things outside the box. Could be worse though, I saw a post on r/sysadmin yesterday where a poor guy got a support ticket to spray fox urine outside near the generators.
> Data center science has... well improved since the earlier days. We worked with Facebook on the OpenCompute Project that had some very forward looking infra concepts at the time.
Am a bit surprised Meta doesn't offer a cloud provider yet to compete with AWS/GCP. Especially considering how much R&D they've put into their infra.
Pro: even more opportunities to spy on every user in the world
Con: interacting with internal stakeholders is waaaaay different from doing the same for the general public paying you. See also: every mention of GCP that ever shows up in these threads
Was really hoping this was was actually about building your own data center. Our town doesn't have a data center, we need to go an hour south or an hour north. The building that a past failed data center was in (which doesn't bode well for a data center in town, eh?), is up for lease and I'm tempted.
But, I'd need to start off small, probably per-cabinet UPSes and transfer switches, smaller generators. I've built up cabinets and cages before, but never built up the exterior infrastructure.
If it turns out to be any of “location, location, location” then getting a partially kitted out building may not help you.
Did they get independent data into the building via different routes? How’s the power?
Could be the data was coming in through a route that sees frequent construction. I knew a guy who ran the IT dept for a university and he discovered that the excavation crews found it was cheaper to maybe have to pay a fine for cutting data lines than it was to wait for them to be marked accurately. He spent a lot of time being stressed out.
I agree that one of the first steps would be to take someone from the previous facility out for a meal, which I can probably arrange fairly easily. I don't exactly know why it failed, it was run by the biggest local ISP. I can speculate about why they failed (DSL speeds are severely limited in our town, so really Xfinity was it, they tried providing fiber in some locations, but found it hard to keep up with fiber locate calls). The Colocation side of the business was never very big, but it's not clear if that is because there's not demand or that they just never really pushed it.
Location is fairly good, as far as data centers go. It's got relatively good network connectivity, I believe, but I don't have specifics about entrances and diversity. It is close to one of the big fiber rings around the city, I believe the ring is pulled into the facility. I don't know if they had telco fiber in, or backhauled it via the fiber ring.
Power is probably good, but not great -- I'd doubt it's fed from multiple substations. There was, at one point, some generator bays.
While I could use data center space in town, it'd be hard to convince my work to move, partly as we just signed a 3 year agreement for hosting 60 miles away, partly just because of the cost of a move. It probably should remain a pipe dream.
The date and time durations given seem a bit confusing to me...
"we kicked off a Railway Metal project last year. Nine months later we were live with the first site in California".
seems inconsistent with:
"From kicking off the Railway Metal project in October last-year, it took us five long months to get the first servers plugged in"
The article was posted today (Jan 2025), was it maybe originally written last year and the project has been going on for more than a year, and they mean that the Railway Metal project actually started in 2023?
ah that's my bad - I wrote this in Dec, we only published in Jan. Obv. missed updating that.
Timeline wise;
- we decided to go for it and spend the $$$ in Oct '23
- Convos/planning started ~ Jan '24
- Picked the vendors we wanted by ~ Feb/Mar '24
- Lead-times, etc... meant everything was ready for us to go fit the first gear by mostly ourselves at the start of May (that's the 5mo)
- We did the "proper" re-install around June, followed closely by the second site in ~ Sep, around when we started letting our users on it as a open beta
- Sep-Dec we just doubled down on refining software/automation and process while building out successive installs
Lead times can be mind numbing. We have certain switches from Arista that have a 3-6 mo leadtime. Servers are build to order, so again 2+ months depending on stock. And obv. holidays mean a lot of stuff shuts down around December.
Sometimes you can swap stuff around to get better lead-times, but then the operational complexity explodes because you have this slightly different component at this one site.
I used to be a EEE, and I thought supply chain there was bad. But with DCs I think it's sometimes worse because you don't directly control some parts of your BoM/supply chain (especially with build-to-order servers).
From working at a cloud, and speaking with capacity folks regularly when I was in certain roles, the supply chain strikes me as one of the biggest nightmares. Even at scale when vendors really, really want (or want to keep) your business.
At times it almost seems like someone sneezes somewhere and whoops, there goes your hardware delivery timelines.
The advantage at cloud scale is a lot of constant signal around capacity delivery, demand etc. so you can build mathematical models to best work out when to start placing orders, and for what.
If you’re using 7280-SR3 switches, they’re certainly a fine choice. However, have you considered the 7280-CR3(K) range? They're much better $/Gbps and more relevant edge interfaces.
At this scale, why did you opt for a spine-and-leaf design with 25G switches and a dedicated 32×100G spine? Did you explore just collapsing it and using 1-2 32×100G switches per rack, then employing 100G>4×25G AOC breakout cables and direct 100G links for inter-switch connections and storage servers?
All valid points - and our ideas for Gen 2 sound directionally similar - but those are at crayon drawing stage.
When we started, we didn't have much of an idea about what the rack needs to look like. So we chose a combination of things we thought we could pull this off. We're mostly software and systems folks, and there's a dearth of information out there on what to do. Vendors tend to gravitate towards selling BGP+EVPN+VXLAN or whatever "enterprise" reference designs; so we kinda YOLO'ed the Gen 1. We decided to spend extra money if we could get to a working setup sooner. When the clock is in cloud spend, there's uh... lots of opportunity cost :D.
A lot of the chipset and switch choices were bets and we had to pick and choose what we gambled on - and what we could get our hands on. The main bets this round were eBGP to the hosts with BGP unnumbered, SONiC switches - this lets us do a lot of networking with our existing IPv6/Wireguard/eBPF overlay and a debian based switch OS + FRR (so fewer things to learn). And ofc. figuring out how to operationalise the install process and get stuff running on the hardware as soon as possible.
Now we've got a working design, we'll start iterating a bit more on the hardware choice and network design. I'd love for us to write about it when we get through it. Plus I think we owe the internet a rant on networking in general.
Edit: Also we don't use UniFi Pro / Uniquity gear anywhere?
This is how you build a dominant company. Good for you ignoring the whiny conventional wisdom that keeps people stuck in the hyperscalers.
You’re an infrastructure company. You gotta own the metal that you sell or you’re just a middleman for the cloud, and always at risk of being undercut by a competitor on bare metal with $0 egress fees.
Colocation and peering for $0 egress is why Cloudflare has a free tier, and why new entrants could never compete with them by reselling cloud services.
In fact, for hyperscalers, bandwidth price gouging isn’t just a profit center; it’s a moat. It ensures you can’t build the next AWS on AWS, and creates an entirely new (and strategically weaker) market segment of “PaaS” on top of “IaaS.”
in my experience (Eu, datacenter/ISP space) connectivity is either sold based on 99percentile commitments (aka, you pay for sending us this amount of traffic, anything over this gets you billed extra) or based on a minimum commitment for the traffic you send. (atleast X amount of bandwith) or it is based on a flat free principle where you pay an upfront cost for the setup and the rest is a base price for X amount of bits per seconds.
It depends a lot on what kind of connection you require, and things like oversubscription and congestion control also come into play.
Peering ports with IXP's are usually flat rate, while ports in datacenters to end customers usually have more complex constructs.
Hyperscaler bandwith is notoriously expensive.
for instance, a 100Gbps port on the AMS-IX is 2500$[1].
Now, you need to account for extra costs to actually use this port (some IP space, ASN number etc) but even with all that added up i think you will not get much more expensive then 400$ per month averaged in total over a year.
Now what makes comparing difficult is that hyperscalers are not transparant when it comes to connectivity costs. Looking at egress fees for example:
AWS seems to charge 1 cent per Gigabyte transferred for egress fees.
If we send data at line rate across our 100Gbps for an entire month we get the following:
100gbps = 12.5Gigabytes per second.
12.5 * 2 629 743 83 (number of seconds in a month) = 32871797875 Gigabytes
32871797875 / 0,001$ = 3.287.179,7875
thats 3,2 million dollars... compared to roughly 4000!
AWS also seems to offer "dedicated connections". at roughly 22 dollars per hour [3] (no clue if this is even comparble to an IXP port, but the comparison would still be fun to make).
22$ x 720 (hours per month = 15.840$, or roughly 3times the IXP port price.
In both cases, you are getting absolutely shafted by egress prices at cloud providers compared to doing it yourself.
My first colo box came courtesy of a friend of a friend that worked for one of the companies that did that (leaving out names to protect the innocent). It was a true frankenputer built out of whatever spare parts he had laying around. He let me come visit it, and it was an art project as much as a webserver. The mainboard was hung on the wall with some zip ties, the PSU was on the desk top, the hard drive was suspended as well. Eventually, the system was upgraded to newer hardware, put in an actual case, and then racked with an upgraded 100base-t connection. We were screaming in 1999.
It looked interesting, until I got to the egress cost. Ouch. $100 per TB is way too much if you're using bandwidth-intensive apps.
Meta-comment: it's getting really hard to find hosting services that provide true unlimited bandwidth. I want to do video upload/download in our app, and I'm struggling to find providers of managed servers that would be willing to provide me with fixed price for 10/100GB ports.
The later. I was expecting local boot because pxe introduces a rather big dependency for potentially many machines. Issues with network or issues with pxe server and nothing boots
Booting through the IPMI with virtual media isos over http is dog slow in my experience.
Using PXE to bootstrap an installer kernel (only few MB) over TFTP that fetches the rest of the OS over HTTP is quick and you can pressed/kickstart a machine in minutes.
285 comments
[ 2.9 ms ] story [ 272 ms ] threadThe host runs a custom daemon that programs FRR (an OSS routing stack), so that it advertises addresses assigned to a VM to the rest of the cluster via BGP. So zero config of network switches, etc... required after initial setup.
We'll blog about this system at some point in the coming months.
Could be worth adding a <meta> tag to the <head> so that RSS readers can autodiscover the feed. A random link I found on Google: https://www.petefreitag.com/blog/rss-autodiscovery/
Tons of Colocation available nearly everywhere in the US, and in the KCMO area, there are even a few dark datacenters available for sale!
cool project none-the-less. Bit jealous actually :P
So, while we could have bought something off the shelf, that would have been suboptimal from a specs perspective. Plus then we'd have to source supply chain etc.
By owning not just the servers but the whole supply chain, we have redundancy at every layer, from the machine, to the parts on site (for failures), to the supply chain (refilling those spare parts/expanding capacity/etc)
Like, if Terraform had a nice UI?
If you've heard of serverless, this is one step farther; infraless
Give us your code, we will spin it up, keep it up, automate rollouts service discovery, cluster scaling, monitoring, etc
I've been using railway since 2022 and it's been great. I host all my personal projects there and I can go from code to a url by copy-pasting my single dockerfile around.
Like Vercel but not just for front end
We're lucky to have a few great distributors/manufacturers who help us pick the right gear. But we learnt a lot.
We've found a lot of value in getting a broker in to source our transit though.
My personal (and potentially misguided) hot take is that most of the baremetal world is stuck in the early 2000's, and the only companies doing anything interesting here the likes of AWS,Google and Meta. So the only way to innovate is to stumble around, escape the norms and experiment.
If you're looking for great partners, who actually have the gal to back innovation, you'd be hard pressed to do better than Redpoint (Shoutout Erica and Jordan!)
We had one provider give us a great price and then bait and switch at the last moment to tell us that there is some other massive installation charge that they didn't realize we had to pay.
Switch Connect/Core is based off the old Enron business that Rob (CEO) bought...
https://www.switch.com/switch-connect/ https://www.switch.com/the-core-cooperative/
Still kudos going this path in the cloud-centric time we live in.
I feel it's important to stress that the difficulty level of collocating something, let alone actually building a data center, is exactly what makes cloud computing so enticing and popular.
Everyone focuses on trivia items like OpEx vs CapEx and dynamic scaling, but the massive task of actually plugging in the hardware in a secure setting and get it to work reliably is a massive undertaking.
One of the better was the dead possum in the drain during a thunderstorm.
>So do we throw the main switch before we get electroduced? Or do we try to poke enough holes in it that it gets flushed out? And what about the half million in servers that are going to get ruined?
Sign up to my patreon to find out how the story ended.
Thy will vary by country, by state or even county , setting up a DC in the Bay Area and say one in Ohio or Utah is a very different endeavor with different design considerations.
What point are you trying to make? It does not matter where you are in the world, or what local laws exist or permits are required, racking up servers in a cage is much less difficult than physically building a data center (of which racking up servers is a part).
Also people doing the build outs of a DC aren't likely keen on talking about permits and confidential agreements in the industry quite publicly.
Yes the title is click baity, but that is par of the course these days.
Regarding data centers that cost 9 figures and up:
For the largest players, there’s not a ton of variation. A combination of evaporative cooling towers and chillers are used to reject heat. This is a consequence of evaporative open loop cooling being 2-3x more efficient than a closed-loop system.
There will be multiple medium-voltage electrical services, usually from different utilities or substations, with backup generators and UPSes and paralleling switchgear to handle failover between normal, emergency, and critical power sources.
There’s not a lot of variation since the two main needs of a data center are reliable electricity and the ability to remove heat from the space, and those are well-solved problems in mature engineering disciplines (ME and EE). The huge players are plopping these all across the country and repeatability/reliability is more important than tailoring the build to the local climate.
FWIW my employer has done billions of dollars of data center construction work for some of the largest tech companies (members of Mag7) and I’ve reviewed construction plans for multiple data centers.
I'll point out that some of the key thermal and power stuff in those plans you saw may have come from the hyperscalers themselves - our experience a dozen years or so ago was that we couldn't just put it out to bid, as the typical big construction players knew how to build old data centers, not new ones, and we had to hire a (very small) engineering team to design it ourselves.
Heat removal is well-solved in theory. Heat removal from a large office building is well-solved in practice - lots of people know exactly what equipment is needed, how to size, install, and control it, what building features are needed for it, etc. Take some expert MEs without prior experience at this, toss them a few product catalogs, and ask them to design a solution from first principles using the systems available and it wouldn't be so easy.
There are people for whom data center heat removal is a solved problem in practice, although maybe not in the same way because the goalposts keep moving (e.g. watts per rack). Things may be different now, but a while back very few of those people were employed by companies who would be willing to work on datacenters they didn't own themselves.
Finally I'd add that "9 figures" seems excessive for building+power+cooling, unless you're talking crazy sizes (100MW?). If you're including the contents, then of course they're insanely expensive.
1 and 2 are independent of regulatory domain. 3 involves utilities, not governments, and is probably a clusterfck anywhere; 4 isn't as bad (anywhere in the US; not sure elsewhere) because it's not a monopoly, and you can probably find someone to say "yes" for a high enough price.
There are people everywhere who are experts in site acquisition, permits, etc. Not so many who know how to build the thermals and power, and who aren't employed by hyperscalers who don't let them moonlight. And depending on your geographic location, getting those megawatts from your utility may be flat out impossible.
This assumes a new build. Retrofitting an existing building probably ranges from difficult to impossible, unless you're really lucky in your choice of building.
[*] hmm, the one geographic issue I can think of is water availability. If you can't get enough water to run evaporative coolers, that might be a problem - e.g. dumping 10MW into the air requires boiling off I think somewhere around 100K gallons of water a day.
The cynic in me says this was written by sales/marketing people targeted specifically at a whole new generation of people who've never laid hands on the bare metal or racked a piece of equipment or done low voltage cabling, fiber cabling, and "plug this into A and B power AC power" cabling.
By this, I mean people who've never done anything that isn't GCP, Azure, AWS, etc. Many terminologies related to bare metal infrastructure are misused by people who haven't been around in the industry long enough to have been required to DIY all their own infrastructure on their own bare metal.
I really don't mean any insult to people reading this who've only ever touched the software side, but if a document is describing the general concept of hot aisles and cold aisles to an audience in such a way that it assumes they don't know what those are, it's at a very introductory/beginner level of understanding the OSI layer 1 infrastructure.
I wanted to start off with the 101 content to see if people found it approachable/interesting. He's got like reams and reams of 201, 301, 401
Next time I'll stay out of the writing room!
Fair tbh
We will indeed write more on this so this is great feedback for next time!
Yes, building a physical DC is much wider scope than colo. This is one part of that, which is also still interesting. The world is built on many, many layers of abstraction which can all take lifetimes to explore. There are non-devs who enjoy learning about software, web-devs who dabble in compilers, systems programmers curious about silicon, EE's that are aspiring physicists, who in turn peek into the universe of pure path (cue yes, that xkcd you're thinking of).
A 'full stack' overview of a standalone DC build still has to set a bound somewhere. This was an approachable intro and look forward to reading more from the layers you operate.
When the original aws instance came out it would take you about two years or on demand to pay for the same hardware on prem. Now its between two weeks for ml heavy instances to six months for medium CPU instances.
It just doesn't make sence to use the cloud for anything past prototyping unless you want Bazos to have a bigger yacth.
TFA explain what they're doing, they literally write this:
"In general you have three main choices: Greenfield buildout (...), Cage Colocation (getting a private space inside a provider's datacenter enclosed by mesh walls), or Rack colocation...
We chose the second option"
I don't know how much clearer they can be.
Cloudflare has also historically used “datacenter” to refer to their rack deployments.
All that said, for the purpose of the blog post, “building your own datacenter” is misleading.
Even where they do lease wholesale space, you'd be hard pushed to find examples of more than one in a single building. If you count them as Microsoft, Google, AWS then I'm not sure I can think of a single example off the top of my head. Only really possible if you start including players like IBM or Oracle in that list.
I can’t dig up the source atm but IIRC some Equinix website was bragging about it (and it wasn’t just about direct connect to GCP).
Google and AWS will put routers inside Equinx Slough sure, but that's literally written on the tin, and the only way a carrier hotel could work.
Equinx is retail, with only a couple of exceptions, although I know they're trying to grow the wholesale side.
That being said, this page amusingly mentions colocation in reference to media destruction: https://learn.microsoft.com/en-us/compliance/assurance/assur...
cloud.google.com/about/locations lists all the locations that GCE offers service, which is a super set of the large facilities that someone would call a "Google Datacenter". I liked to mostly refer to the distinction as Google concrete (we built the building) or not. Ultimately, even in locations that are shared colo spaces, or rented, it's still Google putting custom racks there, integrating into the network and services, etc. So from a customer perspective, you should pick the right location for you. If that happens to be in a facility where Google poured the concrete, great! If not, it's not the end of the world.
P.S., I swear the certification PDFs used to include this information (e.g., https://cloud.google.com/security/compliance/iso-27018?hl=en) but now these are all behind "Contact Sales" and some new Certification Manager page in the console.
Edit: Yes! https://cloud.google.com/docs/geography-and-regions still says:
> These data centers might be owned by Google and listed on the Google Cloud locations page, or they might be leased from third-party data center providers. For the full list of data center locations for Google Cloud, see our ISO/IEC 27001 certificate. Regardless of whether the data center is owned or leased, Google Cloud selects data centers and designs its infrastructure to provide a uniform level of performance, security, and reliability.
So someone can probably use web.archive.org to get the ISO-27001 certificate PDF from whenever the last time it was still up.
I cannot believe that they never had a UK DC until recently. I guess they (originally) chose Ireland instead.
This is not good, I can't think of any actual reason to hide those certificates.
For comparison, AWS makes their ISO-27001 certificate available at https://aws.amazon.com/compliance/iso-27001-faqs/ and also cites the certifying agent, most of which have a search page from where you can find all the certificates they've issued.
I don't know what point you tried to make. Any business in the whole world survives because they sell things for more money than what it takes to keep their business running. Is it surprising that they charge their customers more than their infrastructure costs?
I think you're conflating things.
Those hypothetical hyperscalers can advertise their availability zones and deployment regions, but they do not claim they built the data centers. They provide a service, but they do not make broad claims on how they built infrastructure.
Step 1: sign a lease at an apartment
HN people are smart
I think you're failing to understand the meaning and the point of "building your own datacenter".
Yes, you can talk about your office all you'd like. Much like OP can talk about there server farm and their backend infrastructure.
What you cannot talk about is your own office center. You do not own it. You rent office space. You only have a small fraction of the work required to operate an office, because you effectively offloaded the hard part to your landlord.
For people who have taken empty lots and constructed new data centers (ie, the whole building) on them from scratch, the phrase "building a datacenter" involves a nonzero amount of concrete.
OP seems to have built out a data hall - which is still a cool thing in its own right! - but for someone like me who's interested in "baking an apple pie from scratch", the mismatch between the title and the content was slightly disappointing.
I say this not merely to be a pompous smartass but also because it illustrates and echoes the very same problem the top-level comment embodies, viz. that some folks struggle with vernacular, nonliteral, imprecise, and nonlinear language constructs. Yet grasping this thistle to glark one's grok remains parcel-and-part of comprehension and complaining about it won't remeaningify the barb'd disapprehensible.
Your disappointment, nevertheless, seems reasonable, because the outcome was, after all, a bait-and-route.
It was my first job out of university. I will never forget the awesome experience of walking into the datacenter and start plugging cables and stuff
Did you standardize on layout at the rack level? What poke-yoke processes did you put into place to prevent mistakes?
What does your metal->boot stack look like?
Having worked for two different cloud providers and built my own internal clouds with PXE booted hosts, I too find this stuff fascinating.
Also take utmost advantage of a new DC when you are booting it to try out all the failure scenarios you can think of and the ones you can't through randomized fault injection.
I'm going to save this for when I'm asked to cut the three paras on power circuit types.
Re: standardising layout at the rack level; we do now! we only figured this out after site #2. It makes everything so much easier to verify. And yeah, validation is hard - manually doing it thus far; want to play around with scraping LLDP data but our switch software stack has a bug :/. It's an evolving process, the more we work with different contractors, the more edge cases we unearth and account for. The biggest improvement is that we have built a internal DCIM that templates a rack design and exports a interactive "cabling explorer" for the site techs - including detailed annotated diagrams of equipment showing port names, etc... The screenshot of the elevation is a screenshot of part of that tool.
> What does your metal->boot stack look like?
We've hacked together something on top of https://github.com/danderson/netboot/tree/main/pixiecore that serves a debian netboot + preseed file. We have some custom temporal workers to connect to Redfish APIs on the BMCs to puppeteer the contraption. Then a custom host agent to provision QEMU VMs and advertise assigned IPs via BGP (using FRR) from the host.
Re: new DCs for failure scenarios, yeah we've already blown breakers etc... testing stuff (that's how we figured out our phase balancing was off). Went in with a thermal camera on another. A site in AMS is coming up next week and the goal for that is to see how far we can push a fully loaded switch fabric.
The edge cases are the gold btw, collect the whole set and keep them in a human and machine readable format.
I'd also go through and using a color coded set of cables, insert bad cables (one at a time at first) while the system is doing an aggressive all to all workload and see how quickly you can identify faults.
It is the gray failures that will bring the system down, often multiple as a single failure will go undetected for months and then finally tip over an inflection point at a later time.
Are you workloads ephemeral and/or do they live migrate? Or will physical hosts have long uptimes? It is nice to be able to rebaseline the hardware before and after host kernel upgrades so you can detect any anomalies.
You would be surprised about how larger of a systemic performance degradation that major cloud providers have been able to see over months because "all machines are the same", high precision but low absolute accuracy. It is nice to run the same benchmarks on bare metal and then again under virtualization.
I am sure you know, but you are running a multivariate longitudinal experiment, science the shit out of it.
Re: Live Migration We're working on adding Live Migration support to our orchestrator atm. We aim to have it running this quarter. That'll makes things super seamless.
Re: kernels We've already seen some perf improvements somewhere between 6.0 and 6.5 (I forget the exact reason/version) - but it was some fix specific to the Sapphire Rapids cpus we had. But I wish we had more time to science on it, it's really fun playing with all the knobs and benchmarking stuff. Some of the telemetry on the new CPUs is also crazy - there's stuff like Intel PCM that can pull super fine-grained telemetry direct from the CPU/chipset https://github.com/intel/pcm. Only used it to confirm that we got NUMA affinity right so far - nothing crazy.
You will need a way to coordinate LM with users due them being sensitive to LM blackouts. Not many workloads are, but the ones that are are the kinds of things that customers will just leave over.
If you are draining a host, make sure new VMs are on hosts that can be guaranteed to be maintenance free for the next x-days. This allows customers to restart their workloads on their schedule and have a guarantee that they won't be impacted. It also encourages good hygiene.
Allow customers to trigger migration.
Charge extra for a long running maintenance free host.
It is good you are hooked into the PCM already. You will experience accidentally antagonistic workloads and the PCM will really help debug those issues.
If I were building a DC, I put as many NICs into a host as possible and use SR-VIO to pass the nics into the guests. The switches should be sized to allow for full speed on all nics. I know it sounds crazy but if you design for a typical crud serving tree, you are a saving a buck but making your software problem 100x harder.
Everything should have enough headroom so it never hits a knee of a contention curve.
It's written for Cumulus Linux, but it should be adaptable to other NOSes with some work: https://github.com/CumulusNetworks/ptm
You give it a graphviz dot file, and it uses LLDP to ensure that reality matches that file.
I _knew_ Railway sounded familiar.
Out of curiosity: is nix used to deploy the servers?
Looking forward to more blogposts!
https://schematix.com/video/racks
https://github.com/netbox-community/netbox
In this case railway will need to care about a lot of extra information beyond just racks, IP addresses and physical servers.
https://github.com/netbox-community/netbox/issues?q=is%3Aiss...
Note how they want to be "NetBox functions as the source of truth for your network infrastructure."
Your individual situation dictates what is important, but had netbox targeted being a central repository vs insisting on not allow other systems to be truthful for certain items it could be a different story.
We have learned that trying to centralize complexity and control doesn't work, heck we knew that almost immediately after the Clinger Cohen Act passed and even ITIL and TOGAF fully call this out now and I expect this to be targeted by consultants over the next few years.
You need a central constant way to find state, to remove any questions or doubt regarding where to find the authoritative information, but generally if you aspire to scale and grow or adapt to new changes you really need to avoid having some centralized, god-box, and prescriptive system like this.
This is the usual case of "We need X and Y does X", but ignoring that Y also does Z,M,Q and washes dishes and you really don't need those things.
Sometimes building what you need is the easiest solution, specially when what you need is CRUD infront of a DB...
It feels more like an OSS tool for managing university campus scale infra, which is completely fine if that is the problem you have but for commercial scale infrastructure unfortunately there isn't a good OOTB DCIM option right now.
Pointing the fans in or out?
You say that, but...
> There was one day we knocked out windows and purchased box fans because servers were literally catching on fire
This happened to Equinix's CH1 datacenter in Chicago Jan24 (not the literal fire part). Took down Azure ExpressRoute.
Apparently it got too cold and the CRACs couldn't take it? I'm told they had all the doors and windows open trying to keep things cold enough, but alas. As the CRAC goes, so goes the servers
absolutely do not miss those days
It was also 115 degrees ambient temp inside CH1. Techs were dipping in and out 5-10 minutes at a time to avoid heat stroke
Ah yes, or a collection of R2D2 portable air conditioners, with the tails draped out through the window.
Or a coolant leak that no one noticed until the sub-floor was completely full and the floor panels started to float!
Yikes, that escalated quickly. I'm glad you escaped the Switch Grim Reaper and my condolences to the families of the rest :(
wait, what?
One of the stories was learning that stuff on top gets hotter than stuff on bottom.
This is, like, basic stuff here, guys. I've never understood the hiring practices in these projects.
And a few years after that I also got to commission a laser beam link over Manchester's Oxford Road (at the time, the busiest bus route in Europe), to link up an office to a University campus. Fun times.
It was all terrific fun, but I'm so glad I now only really do software.
I don't blame you, a lot of us had to do things outside the box. Could be worse though, I saw a post on r/sysadmin yesterday where a poor guy got a support ticket to spray fox urine outside near the generators.
Am a bit surprised Meta doesn't offer a cloud provider yet to compete with AWS/GCP. Especially considering how much R&D they've put into their infra.
Con: interacting with internal stakeholders is waaaaay different from doing the same for the general public paying you. See also: every mention of GCP that ever shows up in these threads
Plus all their SDKs would be written in php :-P
But, I'd need to start off small, probably per-cabinet UPSes and transfer switches, smaller generators. I've built up cabinets and cages before, but never built up the exterior infrastructure.
If it turns out to be any of “location, location, location” then getting a partially kitted out building may not help you.
Did they get independent data into the building via different routes? How’s the power?
Could be the data was coming in through a route that sees frequent construction. I knew a guy who ran the IT dept for a university and he discovered that the excavation crews found it was cheaper to maybe have to pay a fine for cutting data lines than it was to wait for them to be marked accurately. He spent a lot of time being stressed out.
Location is fairly good, as far as data centers go. It's got relatively good network connectivity, I believe, but I don't have specifics about entrances and diversity. It is close to one of the big fiber rings around the city, I believe the ring is pulled into the facility. I don't know if they had telco fiber in, or backhauled it via the fiber ring.
Power is probably good, but not great -- I'd doubt it's fed from multiple substations. There was, at one point, some generator bays.
While I could use data center space in town, it'd be hard to convince my work to move, partly as we just signed a 3 year agreement for hosting 60 miles away, partly just because of the cost of a move. It probably should remain a pipe dream.
"we kicked off a Railway Metal project last year. Nine months later we were live with the first site in California".
seems inconsistent with:
"From kicking off the Railway Metal project in October last-year, it took us five long months to get the first servers plugged in"
The article was posted today (Jan 2025), was it maybe originally written last year and the project has been going on for more than a year, and they mean that the Railway Metal project actually started in 2023?
Timeline wise; - we decided to go for it and spend the $$$ in Oct '23 - Convos/planning started ~ Jan '24 - Picked the vendors we wanted by ~ Feb/Mar '24 - Lead-times, etc... meant everything was ready for us to go fit the first gear by mostly ourselves at the start of May (that's the 5mo) - We did the "proper" re-install around June, followed closely by the second site in ~ Sep, around when we started letting our users on it as a open beta - Sep-Dec we just doubled down on refining software/automation and process while building out successive installs
Lead times can be mind numbing. We have certain switches from Arista that have a 3-6 mo leadtime. Servers are build to order, so again 2+ months depending on stock. And obv. holidays mean a lot of stuff shuts down around December.
Sometimes you can swap stuff around to get better lead-times, but then the operational complexity explodes because you have this slightly different component at this one site.
I used to be a EEE, and I thought supply chain there was bad. But with DCs I think it's sometimes worse because you don't directly control some parts of your BoM/supply chain (especially with build-to-order servers).
The advantage at cloud scale is a lot of constant signal around capacity delivery, demand etc. so you can build mathematical models to best work out when to start placing orders, and for what.
If we have to we can “burst” into the cloud
At this scale, why did you opt for a spine-and-leaf design with 25G switches and a dedicated 32×100G spine? Did you explore just collapsing it and using 1-2 32×100G switches per rack, then employing 100G>4×25G AOC breakout cables and direct 100G links for inter-switch connections and storage servers?
Have you also thought about creating a record on PeeringDB?https://www.peeringdb.com/net/400940.
By the way, I’m not convinced I’d recommend a UniFi Pro for anything, even for out-of-band management.
When we started, we didn't have much of an idea about what the rack needs to look like. So we chose a combination of things we thought we could pull this off. We're mostly software and systems folks, and there's a dearth of information out there on what to do. Vendors tend to gravitate towards selling BGP+EVPN+VXLAN or whatever "enterprise" reference designs; so we kinda YOLO'ed the Gen 1. We decided to spend extra money if we could get to a working setup sooner. When the clock is in cloud spend, there's uh... lots of opportunity cost :D.
A lot of the chipset and switch choices were bets and we had to pick and choose what we gambled on - and what we could get our hands on. The main bets this round were eBGP to the hosts with BGP unnumbered, SONiC switches - this lets us do a lot of networking with our existing IPv6/Wireguard/eBPF overlay and a debian based switch OS + FRR (so fewer things to learn). And ofc. figuring out how to operationalise the install process and get stuff running on the hardware as soon as possible.
Now we've got a working design, we'll start iterating a bit more on the hardware choice and network design. I'd love for us to write about it when we get through it. Plus I think we owe the internet a rant on networking in general.
Edit: Also we don't use UniFi Pro / Uniquity gear anywhere?
You’re an infrastructure company. You gotta own the metal that you sell or you’re just a middleman for the cloud, and always at risk of being undercut by a competitor on bare metal with $0 egress fees.
Colocation and peering for $0 egress is why Cloudflare has a free tier, and why new entrants could never compete with them by reselling cloud services.
In fact, for hyperscalers, bandwidth price gouging isn’t just a profit center; it’s a moat. It ensures you can’t build the next AWS on AWS, and creates an entirely new (and strategically weaker) market segment of “PaaS” on top of “IaaS.”
With this, it'll mean we can slash that in half, lower storage costs, remove "per seat" pricing, etc
Super exciting
It depends a lot on what kind of connection you require, and things like oversubscription and congestion control also come into play.
Peering ports with IXP's are usually flat rate, while ports in datacenters to end customers usually have more complex constructs.
Hyperscaler bandwith is notoriously expensive. for instance, a 100Gbps port on the AMS-IX is 2500$[1].
Now, you need to account for extra costs to actually use this port (some IP space, ASN number etc) but even with all that added up i think you will not get much more expensive then 400$ per month averaged in total over a year.
Now what makes comparing difficult is that hyperscalers are not transparant when it comes to connectivity costs. Looking at egress fees for example:
AWS seems to charge 1 cent per Gigabyte transferred for egress fees.
If we send data at line rate across our 100Gbps for an entire month we get the following:
100gbps = 12.5Gigabytes per second. 12.5 * 2 629 743 83 (number of seconds in a month) = 32871797875 Gigabytes 32871797875 / 0,001$ = 3.287.179,7875
thats 3,2 million dollars... compared to roughly 4000!
AWS also seems to offer "dedicated connections". at roughly 22 dollars per hour [3] (no clue if this is even comparble to an IXP port, but the comparison would still be fun to make).
22$ x 720 (hours per month = 15.840$, or roughly 3times the IXP port price.
In both cases, you are getting absolutely shafted by egress prices at cloud providers compared to doing it yourself.
[1] https://www.ams-ix.net/ams/pricing-us [2] https://aws.amazon.com/ec2/pricing/on-demand/ [3] https://aws.amazon.com/directconnect/pricing/
I am seeing 5-9x that?
Meta-comment: it's getting really hard to find hosting services that provide true unlimited bandwidth. I want to do video upload/download in our app, and I'm struggling to find providers of managed servers that would be willing to provide me with fixed price for 10/100GB ports.
A 10G port should be in the range of $2k per month, I believe? I don't mind paying that much.
Using PXE to bootstrap an installer kernel (only few MB) over TFTP that fetches the rest of the OS over HTTP is quick and you can pressed/kickstart a machine in minutes.