Is this a big deal? I clicked on this skeptically, but...it actually seems like kind of a big deal? It is at least technically impressive, since it seems like they're eliminated a ton of datacenter pain points with the mechanical design.
It sort of reminds me of a Dell blade chassis that I pulled out of a DC last year - it was pretty impressive tech that I hadn't dealt with. 6 power supplies that I think could take 110v or 220v, but you needed 220v for full capacity. All of them were hot-swappable and as long as you had 3 powered it would keep running. The compute sleds were modular and it also had integrated networking as mentioned in this article. All obviously crafted to a very high standard; even the plastic fan cages felt premium (and I'm told it cost accordingly back in 2010 or so).
Also a former employer who was a bit below the hyperscaler level also had what they called "roll-in racks", though I believe they were just taking standard servers and networking gear and integrating them somewhere (presumably with cheap labor) and then bringing them into the DC as needed.
So I don't think it's a particularly new concept, but I agree it looks like it has some potential as a product.
Oxide is a bit of a darling child on HN - I think a lot of people (myself included) find it interesting that they are trying to re-engineer something somewhat commoditized and boring (servers) into something cool
It seems like they have managed to fit 32 1P servers in a 42U rack. That's pretty impressive, I have to admit. I don't think I've seen that level of density before.
32 1P servers in a 42U rack is impressive? If you use 1U servers, they would only use up 32 units, leaving with 10 units for storage and networking.
Also, I remember for example HP BladeCenter BL20p systems. There you would have up to 8 or 16 servers per 6 rack units IIRC. So about 56 to 112 servers in 42U.
It appears to be an all in one, preconfigured/partially-or-completely assembled rack server for on-prem hosting, complete with pre-configured software.
Just my perception given the front page. I agree it's a bit vague.
Absolutely. The point of cloud computing is that I don't have to care where something is running and that I (theoretically) have infinite elasticity and can scale in and out as fast and much as I need to.
This server is for cloud operators - i.e. for organizations that are building their own cloud infrastructure. That's why Oxide is drawing comparisons to hyperscalers.
Nonsense. The operative bit of "cloud" is "provision and de-provision instantly via an API, without much concern for what's going on inderneath", not "lives in someone else's datacenter".
It’s called cloud because it’s not in your own data center. Usually cloud symbols were used in network diagrams to depict systems/networks outside of our concern.
Also you could don’t really get elasticity with a system like this. If anything, that would be the operative bit for me.
That was the case ~15+ years ago. Private cloud has been a term for the majority of that time to evoke the elasticity and virtualisation without the "not in your own data center" bit, because to most users of a cloud the operative bit is that they don't have to worry about where the computer is or talk to someone to provision one, not whether it sits in the corporate data centre or off at Amazon.
Hybrid clouds even means devs might not know whether it sits in the corporate data centre or a public cloud, because it could be either/or depending on current capacity.
> Also you could don’t really get elasticity with a system like this. If anything, that would be the operative bit for me.
"You" as in "the organisation as a whole" don't get elasticity. "You" as in "your department" or "you as an individual" do get elasticity.
> "You" as in "the organisation as a whole" don't get elasticity. "You" as in "your department" or "you as an individual" do get elasticity.
Right, but to the degree that you get elasticity, it starts to look more and more like "someone else's computer", no? If multiple people/departments/etc are provisioning virtual instances on one shared cloud infra, with nobody who's using the provisioning API caring about the underlying capacity (and capacity is planned indirectly by forecasting, etc), then it really starts to sound like "someone else's computer" to me. That "someone else" just happens to be another org within the same company.
With apologies to everyone who has a "The cloud is someone else's computer" T-shirt, things have changed, and the language as evolved as it is wont to do.
I've spent the last decade building on-premises systems very like what Oxide is doing, but I've had to build them out of stacks of servers, switches, storage appliances and VMWare licenses. And the network cabling, and fan noise, and the number of power cables, and.. oh man, I can't wait to install one of these things myself. Having a single point of responsibility for the whole thing shouldn't be underestimated either - I've spent far too long trying to resolve problems with vendors on both sides blaming each other.
It's worth mentioning too that building something equivalent to this would be across more than one rack, and easily cost in excess of $1M.
> It’s called cloud because it’s not in your own data center. Usually cloud symbols were used in network diagrams to depict systems/networks outside of our concern.
That might be true historically, because the only way you could get resources provisioned on-demand via an API from someone else who'd built it. You had to run in someone else's datacenter to get the capability which you actually wanted.
Times have changed. Now, businesses think about "Cloud compute" as being synonymous with "on-demand", "elastic" etc. Where the actual silicon lives is merely an after-thought.
> Also you could don’t really get elasticity with a system like this. If anything, that would be the operative bit for me.
> Where the actual silicon lives is merely an after-thought.
If you have to buy the silicon and plan capacity for it (as in the case with Oxide for example), then it cannot be an afterthought. Which is exactly why I would not consider it cloud computing.
The point is the application team doesn't have to do any of that.
Someone's always got to buy the tin and manage that. Some people are big enough that they might get a benefit from doing that themselves, rather than having Jeff Bezos do it for them.
From the application team's perspective, call API then container go whirr.
> you could don’t really get elasticity with a system like this
Of course you do, right up until the point where you’ve used all available capacity. Just like with public clouds (ask anyone using meaningful amounts of {G,T}PUs). Elasticity doesn’t imply infinitely elastic, that would be ridiculous.
But that is literally not possible with hardware you purchase yourself.
Sure, you can buy X amount of hardware, and provision up to X amount of virtual hardware via an API, but then what? You can't provision any more until you go and buy more hardware. This is why "cloud, but local" is a contradiction IMO. You can only be "cloud-like" if you're under-provisioning. The moment you want to actually use all of the capacity you already paid for, you're not a cloud any more, because you've provisioned all of it.
> But that is literally not possible with hardware you purchase yourself.
Sure it is. I think TFA is talking about a company selling you exactly that capability.
> You can't provision any more until you go and buy more hardware.
But this is also true of AWS etc. When their estate gets full, they need to go buy more hardware. Regardless of who owns the tin, someone's doing a capacity plan and buying hardware to meet demand.
The point of 'cloud' is that you move that function out of the domain who are actually using resources to solve business problems, which is where it traditionally sat. Historically, if you wanted to run a service, you had to go buy some hardware and hire someone to manage it for you.
A cloud-like model means that the application engineers no longer care about servers, disks and switches. Instead, they just use some APIs to request some resources and then deploy a workload onto them. The details of what hardware, where and how is fuzzy and abstracted. Or cloud-like.
> You can only be "cloud-like" if you're under-provisioning
Everyone under-provisions. Nobody runs at 100% utilisation.
There are many “you”s in most enterprises. If your platform engineering team builds their own cloud, and offers an experience similar to other cloud providers (or even better and more targeted), this could be a clear win.
Actually, the definition of cloud is "a visible mass of particles of condensed vapor (such as water or ice) suspended in the atmosphere of a planet (such as the earth) or moon".
I understand that’s your definition and I’m saying that there are many companies where the cloud experience (of not worrying about physical infrastructure but having flexibility and elasticity) is offered to product teams by an internal platform team. That gives you an articulation point where you could migrate from AWS to Oxide racks, and yes, lose some functionality and some guarantees, but also gain more control and potentially make huge savings.
"Private cloud" has been a term for at least a decade to refer to situations where people are building their own cloud infrastructure to provide a "cloud feature set" even though the servers are self hosted.
You may not like it, but it's been a long time since "cloud" has exclusively referred to "someone in another organisation entirely runs the computers" as opposed to virtualised allocation of resources.
"Private cloud" as a term still kinda makes me itch, but I don't recall encountering an alternative that was more obvious and it's clearly the usual term of art at this point.
I suspect that while I do appreciate how some posters upthread find the website a tad on the vague side, the target customers-in-potentia will understand it fine.
> Absolutely. The point of cloud computing is that I don't have to care where something is running and that I (theoretically) have infinite elasticity and can scale in and out as fast and much as I need to.
Define "I"?
I-as-developer can call an VMware/OpenStack API with an on-prem/private cloud and get a new instance just as easily as calling an AWS API. I-as-developer does not have to worry about elasticity if the IT hardware folks have the capacity.
As an engineer, an Oxide system works like any other cloud provider. You’re just interacting with its API and tooling like you would with Google Cloud or AWS.
To someone on the IT/Operations side, obviously there are differences but theres SIGNIFICANTLY less labor required to build-out and operate an Oxide system vs a rack full of servers. The biggest difference for these people is that there’s actual hardware vs a Cloud Provider, but also costs are fixed so there’s likely no monthly or quarterly meetings with finance arguing over the cloud bill, tying people up to try and shave a few thousand off the bill every month.
In finance/accounting, Oxide is probably the most different: now compute is CapEx rather than OpEx. Depending on your company’s stage that can be a wonderful thing for the bean counters.
> To someone on the IT/Operations side, obviously there are differences but theres SIGNIFICANTLY less labor required to build-out and operate an Oxide system vs a rack full of servers.
But it’s also gonna be much more restricted. So I guess one could see as kind of “Apple for data centers”? Have a nice appliance and be happy as long as it runs as it should (but hope it never stops working as it should).
Every "cloud" computer is on-prem for someone, and "private cloud" has been a term for at least a decade to refer to "API-based provisioning of resources like with a cloud but in your own data centre/office" that may not be "technically" a cloud but carries the meaning clearly for anyone in the business.
What they're selling is building blocks for private/hybrid/public clusters that may or may not fit your definition of cloud depending on where they happen to be located but where what the term signifies is that it is built to be a building block of a cloud setup that includes features above and beyond a "regular" server to provide what most people tend to associate with a cloud. That is, you're getting APIs to spawn virtualized compute and storage, rather than having to install a hypervisor and management APIs etc. and combine the resources into a cohesive whole yourself.
My guess is that most of them will end up being sold to either hosting providers to provide cloud services to their customers or companies large enough to operate their own data centres where the IT department will use them to offer cloud services to other parts of the business, where the term really is not misleading anyone.
It's too big even for most "private clouds" in smaller companies, because they tend to be too small to be able to order by the rack even when there are APIs etc. offered to developers to provision compute and storage (e.g. years ago I used to operate a hybrid cloud setup with a ~1000 or so VM instances across several countries, and while we had several racks worth of gear we physically owned in aggregate, we didn't have any full racks in any individual location).
Weird there’s no mention of GPU at all when you’d think that’s what would prick up the ears of the hosting companies ….. stack a pile of GPUs in a rack, surely you can sell time on that.
I don't think they're trying to be AWS. They're trying to sell a product that greatly simplifies doing on-site cloud for companies. So they sell the physical hardware/software bundle and not time.
> “Oxide is a strong believer in the need for open-source software at the lowest layers of the stack -- including silicon initialization and platform enablement. With the availability of AMD openSIL, AMD is showing that they share this vision. We believe that the ultimate beneficiaries of open-source silicon initialization -- as it has been for open-source revolutions elsewhere in the stack -- will be customers and end-users, and we applaud AMD for taking this important and inspiring step!”
Oxide's goal is that they, and by extension their customers, have as much visibility and control over the software stack in these racks as possible, and that includes firmware. They started developing these systems before the current wave of interest in machine learning led by ChatGPU and Stable Diffusion really got underway.
Nvidia GPU drivers are very proprietary, which means that admins and developers have limited visibility into them if they misbehave in any way. This goes against Oxide's philosophy of full visibility into a system that you purchase.
Nvidia's CUDA software has a significant lead ahead of AMD and Intel GPUs, and they're not going to open source it any time soon. But this is a rapidly changing landscape, and AMD and Intel and others are pouring an enormous amount of research into getting their hardware and software to match what Nvidia has going. Nvidia is in pole position, but they're not guaranteed to stay there.
There's still a large market for the CPU workloads that Oxide is offering. For now, Oxide will be concentrating on meeting this traditional compute demand. But you're right to point out that in 2023, the absence of a top tier GPU in these racks is noticeable. I suspect Oxide will want to include some form of GPU or TPU into the next version of their system, but they won't just grab whatever hardware happens to be in fashion. It needs to work with their system as a whole.
Now to see how their development timeframes and synchronization efforts with big hardware companies go.
This is where they will enter the real “hard” part of hardware, with an exec team from software. Can they respond to the market while making hardware?
They seem to have presented as generally thoughtful about their approach. If they can release major variants about annually or even sub-annually that is what I think will enable them to win.
I'll second that, they should make it clear what they're selling and what they have to offer. Every company should have this in their main/landing page... If I can't understand what you're selling/offering in 30sec, then I'm not interested :-)
If you take some time and read around their site you'll see that they're offering a ready to run (turnkey) server. They have everything packed together, they've integrated everything that is needed (CPU, disk, networking...) into a nice looking cabinet, with not too many wires and they're selling it as a complete package.
If you're in need of a server (cloud computer) and you don't want to but separate components and unpack them and connect them yourself, then this looks like a possible solution.
Yes, but... what is a "Cloud Computer"? Is it a "computer in the cloud", like e.g. an AWS EC2 instance? Or is it a fancy name for a good old fashioned server rack (on which you will probably want to run your own cloud, because everybody's doing that nowadays, hence the name), like in this case? And if it's a server rack, how come you don't need any cables? And what do you do with this "cloud computer"? Do they host it in their data center or deliver it to you? So there is some potential for confusion - nothing that can't be mostly cleared up by reading a bit further, but still...
> like e.g. an AWS EC2 instance? Or is it a fancy name for a good old fashioned server rack
I mean, the former is just the latter with some of the setup done for you no? Anyways, it’s a full server rack, with tightly vertically integrated hardware and software. Not sure if you’ve poked around the rest of their site, but it seems like their whole software stack is designed with some really nice usability and integration in mind: there’s a little half-snippet there suggesting that provisioning bare-metal VM’s out of the underlying hardware could as trivial as provisioning an EC2 with Terraform, and if that’s the case, that’s _massive_.
> And if it's a server rack, how come you don't need any cables?
Because they’ve gone to great lengths and care to design it to not need anything extraneous IIUC. I think the compute sleds all automatically mount into some automatic backplane that presumably gives
you power, cooling and networking, and then, as above, you presumably configure all that via software, as you would your AWS setup. Not an expert here though, happy to be corrected by anyone who actually knows better.
> Do they host it in their data center or deliver it to you?
Presumably the latter, given they’re a hardware company, but if their software is even a 10th as good as it seems, I fully believe there’ll be a massive market for renting bare-metal capacity from them.
That's what some of us are saying, it's not crystal clear what they sell.
You use words like: it seems, I think, presumably, I believe...
This is what we're arguing. A company that has raised $44 million Series A for sure can afford to clearly write what they offer.
I understand, you can't have all the people happy and no matter what you do there will always be "weirdos" that don't like your page/design/wording, but hey at least recognize it :-)
> there’s a little half-snippet there suggesting that provisioning bare-metal VM’s out of the underlying hardware could as trivial as provisioning an EC2 with Terraform,
> Or is it a fancy name for a good old fashioned server rack
In a sense. Yes, it is a rack of servers. You're buying a computer. But we've designed the rack of servers as a full rack of servers, rather than an individual 1U. Comes with software to manage the rack like you would a cloud; you don't think of an oxide rack as individual compute sleds, you think of it as a pool of capacity.
> And if it's a server rack, how come you don't need any cables?
Because you are buying an entire rack. The sleds are blind mated. You plug in power, you plug in networking, you're good to go. You're not cabling up a bunch of individual servers when you're installing.
> Do they host it in their data center or deliver it to you?
Customers get them delivered to their data center.
Designing a hardware-software combination to allow for the managing of compute (and networking and storage) to occur at the rack-level rather at the individual-device (server, switch, etc) level, so that large(r)-scale operators can manage cattle more easily (rather than herding cats/pets).
It's a rack of servers you can buy for a lot of money and put in a datacenter. And once you plug it in and turn it on and do whatever setup is required, you're supposed to be able to spin up vms on it.
The site is really hard to navigate. I eventually looked at the footer and found a link to the technical specifications https://oxide.computer/product/specifications They give an idea of the beast, especially the dimensions and weight
Dimensions H x W x D
2354mm (92.7") x 600mm (23.7”) x 1060mm (41.8")
Weight
Up to ~2,518 lbs (~1,145 kg)
It's a rack.
BTW, they are following some no pictures policy. I found only a few pictures of boards but no picture of the product as a whole.
Indeed, which demonstrates that an explicit link to Home in a visible place is never a bad idea. I didn't click on 0xide in the top left even if that's a common shortcut to the home and I didn't notice the link in the footer, which is where you bury the least interesting stuff. I kept clicking on the text links in the top bar.
Also this. Looks technically great but how do you sell this to business? Whats the price range? Is this supposed to be an acceptable solution between on-prem and cloud as we start to realise the true costs of being hooked on cloud when your a large org and not a startup?
They've reinvented the IBM Mainframe. Big rack-sized box with lots of redundant hardware that serves guest VMs. This is basically a zSeries in drag.
The key difference is the price structure -- IBM leases the hardware wherever they can get away with it, and uses a license manager to control how many of the machine's resources you can access (based on how much you can bear to pay). This, however, is like a mainframe you own.
Mainframes also do lots of other things that this doesn't really do. Like allowing you to pretend its a bunch of multi-core machines, or even a single one.
In a data center you have racks of computers performing all of the workloads. At this point these racks are fairly standardized in terms of sizing and ancillary features. These are built-out to solve the following:
* Physical space - The servers themselves require a certain amount of room and depending on the workloads assigned will need different dimensions. These are specified in rack "units" (U) as the height dimension. The width is fixed and depths can vary but are within a standard limit. A rack might have something like 44U of total vertical space and each server can take anywhere from 1-4U generally. Some equipment may even go up to 6U or 8U (or more).
* Power - All rack equipment will require power so there are generally looms or wiring schemes to run all cabling and outlets for all powered devices in the rack. For the most part this can be run on or in the post rails and remains hidden other than the outlet receptacles and mounted power strips. This might also include added battery and power conditioning systems which will eat into your total vertical U budget. Total rack power consumption is a vital figure.
* Cooling - Most rack equipment will require some minimum amount of airflow or temperature range to operate properly. Servers have fans but there will also be a need for airflow within the rack itself and you might have to solve unexpected issues such as temperature gradients from the floor to the ceiling of the rack. Net heat output from workloads is a vital figure.
* Networking - Since most rack equipment will be networked there are standard ways of cabling and patching in networks built into many racks. This will include things such as bays for switches, some of which may eat into the vertical U budget. These devices typically aggregate all rack traffic into a single higher-throughput network backplane that interconnects multiple racks into the broader network topology.
* Storage - Depending on the workloads involved storage may be a major consideration and can require significant space (vertical Us), power, and cooling. You will also need to take into account the bus interconnects between storage devices and servers. This may also be delegated out into a SAN topology similar to a network where you have dedicated switches to connect to external storage networks.
These are some of the major challenges with rack-mounted computing in a data center among many others. What's not really illustrated here is that since all of this has become so standardized we can now fully integrate these components directly rather than buying them piece-meal and installing them in a rack.
This is what Oxide has to offer. They have built essentially an entire rack that solves the physical space, power, cooling, networking, and storage issues by simply giving you a turn-key box you plant in your data center and hook power and interconnects to. In addition it is a fully integrated solution so they can capture a lot of efficiencies that would be hard or impossible in traditional design.
As someone with a lot of data center experience I am very excited to see this. It is built by people with the correct attitude toward compute, imo.
The way I interpret it: an integrated stack of compute, storage and networking hardware and monitoring software that you can plug in to your data center (owned or colo) and can e.g. deploy Kubernetes to, and then use as a deployment target for your services.
The main USP: you own it, you're not renting it. You're not beholden to big cloud's pricing strategies and gotchas.
Secondary USP, why you buy this rather than DIY / rack computer vendor: it's a vertically integrated experience, it's preassembled, it's plug and play, compatibility is sorted out, there's no weird closed third party dependencies. Basically, the Apple experience rather than the PC experience.
The Oxide has everything you need to stand up your own private cloud, and everything works together down to the software. It's a great alternative to buying and managing servers, switches, storage, power management, KVMs, all of the software in between, and the tens of professional services contracts required to glue it all together.
This thing is the definition of vendor lock in. It's a rack full of non-commodity hardware using non-standard connectors and a bunch of custom software.
The industry moved to commodity x86 for compute for a reason.
I wonder how effective AMD 7840 would be as a cloud CPU. 8 cores, fast GPU, AV1 video encoder. Tiny SBC machines …. you could probably stack a bunch of them in a rack.
"Microserver" designs have failed several times. AMD Epyc is basically twelve 7840s in an even smaller space and you can partition it any way you want.
They sell servers, but as a finished product. Not as a cobbled together mess of third party stuff where the vendor keeps shrugging if there is an integration problem. They integrated it. It comes with all the features they expect you to want if you wanted to build your own cloud.
Also, they wrote the software. And it's all open source. So no "sorry but the third party vendor dropped support for the bios". You get the source code. Even if Oxide goes bust, you can still salvage things in a pinch.
Ironically this looks like the realization of Richard Stallman's dream where users can help each other if something doesn't work.
They sell rack-as-compute.[0] Their minimum order is one rack: You plug in power and network, connect to the built-in management software (API), and start spinning up VMs.
Yeah including networking and storage together with virtualization is what makes hyperconverged infrastructure hyperconverged. Otherwise it's usually just called converged infrastructure.
It's not a new concept, it's a new product. Ideas do not become uninteresting the moment the first person instantiates the idea. Further iterations on a possibly-good idea are also interesting.
"Real" mainframes have RAS (Reliability, Availability, Servicing) features such as hotswapping for all hardware components and automated HA/workload migration across physical racks. They can also do SSI (single system image), i.e. run a single workload across physical nodes/racks as if it was just multiple 'cores' in a single shared-memory computer. Oxide computers will probably end up doing at least some of this (namely workload migration across racks for HA) but saying that it can comprehensively replace mainframe hardware as-is is a bit of a stretch. In terms of existing hardware it's closer to a midrange computer.
The Oxide and Friends podcast had an episode on virtualizing time, specifically for the purpose of live-migrating a container from rack to rack without the VM being aware, and allowing operators to take the rack offline on their schedule. Otherwise, apparently, you end up having
to leave racks running because you cannot evacuate all of the containers currently running on it. (e.g. perhaps your contracts or SLAs are such that you cannot afford even the few seconds of downtime a shut-down-here-and-spin-up-elsewhere would cause)
I believe the episode name was "Virtualizing Time"
Obviously different marketing copy speaks to different people. But that is referring to how, when you buy a rack from us, you don't need to put everything together and cable it all up: you pull it out of the box, plug in networking and power, boot the thing up, and you're good to go. Installation time is hours, not days or weeks, which is the norm.
At this stage of the company, everyone gets a white-glove installation process. I suspect that will change over time but I don't work on that part of things, so I don't personally know the details.
Sorry to be slightly obtuse, which details are you referring to here? Help upon installation? At the moment, we are helping customers individually, yeah. But we do have a documented process we are following https://docs.oxide.computer/guides/system/rack-installation-... (and more on other pages there)
Ah yeah, so the "facilities" section of https://oxide.computer/product/specifications has some of these things, probably the closest we have to publicly publishing that in a general sense.
Yes I understand, but will your included service actually verify that everything is set up correctly, meets advertised parameters, and sign off on it? (Such that the customer can start using it immediately afterwards.)
Or does the customer need to take on some risk and hazard associated with installation, configuration, initial boot up, etc.?
e.g. If someone buys with the intention of using it up to X FLOPS, and the machine only delivers Y FLOPS once it's all said and done, what happens?
It’s not the area of the company I personally work on, so I don’t know those details, to be honest. We certainly make sure that everything is working properly.
I mean, we absolutely sell support. I just don't know anything about the details personally. You shouldn't take my lack of knowledge as a "no," just a "steve doesn't personally know."
I wouldn't be dismissive of people telling you that the product description can be improved. My opinion is that the description of the product in this thread will outperform your site 10 to 1.
I'll try to explain, not in the spirit of being argumentative, but with the hope of being useful.
The comment you replied to was not questioning the value of integrated cabling. It was pointing out that the product description on the site does not make sense.
"Cloud computer" sounds like a server you rent from AWS. It's kind of like calling Rust "cloud compiler."
If you choose to use words that your audience doesn't understand, or even worse understands to mean the opposite of what you want them to mean, it's a good idea to explain these words immediately using conventional words with conventional meaning. The comments by throw0101a did that.
The product seems really cool, but there is no way I would've understood what it was from the website.
I understand that's what you're saying, and I understand what the parent is saying. I chose to explain what that alluded to, in case anyone in this conversation is also finding it hard to understand what is meant by that specific copy. That doesn't mean I don't understand the broader point, or that I think the website copy is perfect.
Perhaps if you don't understand what the copy means, then that is a sign that you are not the target audience, rather than that the copy is bad? From what I've gathered from reading other comments in this thread, that copy will make perfect sense to Oxide's target audience, as it uses words in a way that will be very familiar and make perfect sense to the kind of person who might make a purchasing decision for a system like this.
And for what it's worth, I don't think you need to explain what's happening to Steve, it seems to me that he understands perfectly well. To me you come across as being rather condescending and in my opinion Steve is being commendably polite in response.
To be super clear about it, this is referring to not needing to cable up all of the individual sleds to the rack upon installation. It doesn't mean that we recommend connecting a rack of compute to your data center via wifi.
This is pretty big, as someone who has deployed servers to datacenters before. Remote hands are very good at plugging in the network uplink and the PDUs. Doing a complete leaf-spine 25GbE network with full redundancy is something they are pretty much guaranteed to screw up at some point.
"no cables no assembly just cloud" is completely misleading to any kind of people - tech or marketing or not.
When people hear cloud, it means that aspects such as electricity costs, electricity stability, Internet, bandwidth, fire protection, safety, etc etc are abstracted away.
Oxide IS on-premise, right? The website is very vague and wishy-washy.
It is on premises. You interact with the rack the same way you interact with the public cloud: as a pool of resources. The specifics are abstracted away. “Private cloud” is pretty well established terminology in this space, and that’s what we’re doing.
The first iMac famously made it easy to connect to the Internet; The 'i' in iMac was for "Internet". Its setup manual was a couple of pages long, mostly pictures and IIRC, just 37 words.
It would be interesting to sell a data center in a container. Cooling, power supply, compute, storage, and network, all in a box. You supply power, a big network pipe, and the piping to external heat exchangers.
There are whitebox Windows laptops, OpenWRT routers, Arduino boards, ArduPilot drones, etc. It almost sounds strange that there are no prepopulated 12U racks intended for OpenStack(is that still a thing?)
IIJ has a project like this, data center in a container, just add power. They build it all up in Japan, ship it to rural areas across the world to basically jumpstart a local data center (I imagine mostly for industrial sites). They had a fun project where they had a half rack, powered by solar and connected to the net via Starlink.
No idea how they do things today, but v0.1 of Azure (before it was called Azure) was a bunch of containers in a field. I remember seeing an aerial photo at the time.
I’m actually extremely impressed. I want one. I haven’t worked in a data center in years, but I’d be tempted to do it again just to get my hands on one.
Same here. I really want to work on one of these. I got in the industry at the tail end of the time when people used Sun and DEC gear. I got to use just a little bit of it and it seemed so much more "put together" then PC stuff is even now.
Oxide feels like it'll be that "integrated" experience, but with the added benefit of software freedom.
I wish they’d sell a tabletop version for hobbyists, but realize this is probably a distraction. But… the problem with a lot of these systems (including the old Sun boxes and things like ibm mainframes and the AS/400) is that they sound cool but there’s no real way for the typical new developer to “get into them” for fun and, as a result, you lose the chance for some developer selling it to their company based on his experience with the things.
For example, this form factor looks really nice for a “hobbyist edition” or “evaluation edition”: https://zimacube.zimaboard.com/. I would probably buy an Oxide rack like this as soon as pre-orders were announced.
I own a Turing Pi 2 but the hardware it is running on is proprietary. The switch isn't managed. The manament software is very archaic. Yes, it is modular and stackable and probably thousands of times more hobbyist friendly than Oxide but so is edge computing in general.
They won't even tell you how much a rack will cost. Infuriating typically B2B "talk to Sales so we can decide exactly how much we can get out of you and segment the market on the fly" approach persists even here, it seems.
I wouldn’t expect anything else for a full rack in this segment: it’s going to be tens or hundreds of thousands of dollars, and big enough that there will be some inevitable negotiation about prices.
That doesn't mean you can't have a thin spec builder and a pricing page, even if what that mostly gets used for is devs putting together a comparison of that to a cloud deployment or similar and taking that to the procurement department to argue it's worth opening the conversation.
Apparently (I don't remember it, although I probably did read the Byte magazine at the time) there was a rumor in the early 1980s that IBM's PC was going to be a shrunken 370, called the 380. [1][2]
I wish IBM would shrink their LinuxONE Rockhopper 4 Rack Mount down to at least an "under the desktop" model. To my knowledge, IBM still makes quality products and has excellent customer service. They have fun names too (Rockhopper and Emperor are types of penguins!) and they even have 3D models of their rack mount cloud computers with shadows. [3] In fact, when I first read about Oxide a year or two ago, I searched for "IBM cloud server", and left it at that. So IBM, could you please send someone from the LinuxONE down to Boca Raton to create our new PC? :) Thanks!
While working in telecom data centers circa 2016 I've seen many single rack computers from Dell, IBM, HP, Huawei... Not sure that's a new ideia, ex. the open source bits.
Existing vendors will provide rack integration services and deliver a turn key solution like this. Also vendors of virtualization management software have partnerships with hardware suppliers and be happy to deliver fully integrated solutions if you're buying by the rack. The difference is in those cases you have flexibility in the design which seems to be missing here.
Proxmox and a full rack of Supermicro gear would not be as sophisticated, but end result is pretty much the same, with I imagine far far better bang for buck.
I like it, but it doesn't seem like a big deal or revolutionary in any way.
Those of us who've bought large "turn-key" solutions from Dell etc. have often discovered that it's actually just a cobbled-together bunch of things which may or may not work well together on a good day, depending on what you're trying to do. Just because it's all got the word "Dell" written on it, doesn't mean that the components were all engineered by people who were working together to build a single working system.
Total agreement. Another point: Having the "Dell" name on the front doesn't give you a "throat to choke" as so many people seem to think is important. Unless you're very large scale then, at best, you can threaten them that they don't get your next business. You're certainly not going to get help.
You're no worse-off with Oxide from that perspective. Their open source firmware means that thr opportunity to pay somebody else to support you at least exists.
Even small shops can use bad experience as leverage for credits and discounts, especially if the vendor has account managers. This is one of the (few) benefits of having a human involved in invoicing vs. self-serve.
Same is true of Oxide, it'll be up to actual experience to see how well it works. Oxide seems to have written their own distributed block storage system (https://github.com/oxidecomputer/crucible), have their own firmware, kernel and hypervisor forks, etc -- when any of that breaks, good luck!
The premise is that you don't need luck, you can call Oxide. As you said, they wrote all of it, so they own all the interaction so they can diagnose all of it.
When I call Dell with a problem between my OS filesystem and the bus and the hardware RAID, there's at least three vendors involved there so Dell doesn't actually employ anyone that knows all of it so they can't fix it.
Sure, Oxide now needs to deliver on that support promise but at least they are uniquely positioned to be able to do it.
> That's the same premise as with all "turn-key" solutions. If it didn't come with software support, it wasn't really turn-key.
Just about any company will sell your company a support contract.
The more interesting question is, can they back it up with action when push comes to shove? I suspect most people have plenty of stories of opening support tickets with big name vendors that never get resolved. And through the grapevine you find out that they won't fix it because they can't fix it. They might not even have access to the source code or anyone on staff who has a clue about it because it came from who knows where. Sales is happy to sell you the support contract but it doesn't mean your problems can be fixed. BTDT.
From listening to the Oxide podcasts, my impression is that Oxide actually can technically fix anything in the stack they sell, which would make them vastly different from Dell et.al.
Skill-wise, yes for sure (except perhaps for storage -- I haven't heard them talk about that much). Bandwidth wise, though?
I used to work for a company targeting Fortune 500s. At that level of spend, when a client had a problem, somebody got on a plane. Only a fraction of those problems escalated all the way to R&D, which is where Oxide skills are. That's where VMWare etc are hard to beat.
The premise is that the bandwidth needed will be orders of magnitude less, because the engineering will be orders of magnitude better. The opportunity makes sense as we've long been climbing up the local maximum peak of enterprise sales driven tech behemoths built on a cobbled together mix of open source and proprietary pieces held together with bubblegum.
Can an engineering first approach break into the cloud market? Hard to say as enterprise sales is very powerful, and the numerous "worse is better" forces always loom large in these endeavours. That said, enterprise sales driven companies are fat, slow and complacent. Oxide is lean and driven, and a handful of killer use cases and success stories is probably enough to sustain them and could be the thin end of the wedge on long-term success. We can hope anyway.
> Proxmox and a full rack of Supermicro gear would not be as sophisticated, but end result is pretty much the same, with I imagine far far better bang for buck.
I think the question is how well they can do the management plane. Dealing with the "quirks" of a bunch of grey box supermicro stuff is always painful in one way or another. The drop shipped, pre-cabled cab setups are definitely nice but that's only a part of what Oxide is doing here. No cables and their own integrated switching sounds nice too (stuff from the big vendors like UCS is closer to this ballpark but also probably closer to the cost too).
I suspect cooling and rack density could be better in the Oxide solution too, not having to conform to the standards might afford them some possibilities (although that's just a guess, and even if they do improve there these may not be the bottlenecks for many).
Also future datacenter builds are going to be focusing on specific applications which means specific builds. I think Nvidia has a much better chance here with their superpod than Oxide. The target use case is pretty unclear.
On-prem buyers are doing cost reduction and cost reduction targets things like, as one example, the crazy cost of GPU servers on the CSPs. Your run of the mill stuff is very hard to cost reduce.
You can see their sort of lack of getting it by using Tofino2 as their switch. That’s just a very bad choice that was almost certainly chosen for bad reasons.
You don't build a new greenfield compute pod because you want to, you do it because it makes sense. Making sense is about cost and non-cost needs like data gravity and regulatory issues.
The cost case only works for GPU heavy workloads which this isn’t - wrong chassis, wrong network, etc.
Tofino2 is the wrong choice because even when they made that choice it would have been clear that it’s doa. Intel networking has not been a success center in, well, ever. That’s a selection that could only have been made for nerd reasons and not sensible business goals alignment or risk mitigation.
When you make an integrated solution you’d better be the best or close to the best at everything. This does not seem to be the best at anything. I will grant that it is elegant and largely nicer than the hyper converged story from other vendors but in practical terms this is the 2000s era rack scale VxBlock from Cisco or whatever Dell or HPE package today. Marginally better blade server is not a business.
They also make a big deal and have focused on things no one who actually builds data center pods cares about.
I actually hope they get bought by Dell or HPE or SuperMicro. Those companies could fix what’s wrong here and benefit a lot from the attention to detail and elegance on display.
AWS outposts have been there in the market for a long time .. though I am sure there are differences but to say extisting cloud vendors were blind to on prem requirements is a stretch.
> Existing vendors will provide rack integration services and deliver a turn key solution like this.
My experience with the likes of Dell is that they'll deliver it but they won't support it.
Sure, there's a support contract. And they try. But while they sell a box that says Dell, the innards are a hodgepodge of stuff from other places. So when certain firmware doesn't work with something else, they actually can't help because they don't own it, they're just a reseller.
You own this. AWS Outpost is leased and you still also pay for the resource usage on top of the outpost unit itself. And this would not be integrated with your AWS account.
It's mostly not true that you still pay for resource usage on top of the Outpost unit. That's only true, AFAIK, for EBS local snapshots and Route 53 Resolver endpoints. The big boys--EC2 instances, S3 storage, and EBS volumes--are all "free" on Outposts. That is, included in the cost of the unit and not double-charged.
Charging for EBS local snapshots on your own Outpost S3 storage and Route 53 Resolvers on your own compute is a weird one. I don't know how they defend that. To me, it seems indefensible.
> You can purchase Outposts servers capacity for a three-year term and choose between three payment options: All Upfront, Partial Upfront, and No Upfront. … At the end of your Outposts servers term, you can either renew your subscription and keep your Outposts server(s), or return your Outposts server(s). If you do not notify AWS of your selection before the end of your term, your Outposts server(s) will be renewed on a monthly basis, at the rate of the No Upfront payment option corresponding to your Outposts server configuration.
> You can purchase Outposts rack capacity for a 3-year term … either renew your subscription and keep your existing Outposts rack(s), or return your Outposts rack(s)
Route 53 on outposts doesn’t charge for your own compute. The outpost resolver is free. The oupost resolver endpoint must be backed by an in-region resolver endpoint, which doesn’t become free just because you own an outpost. What you’re paying for is the ability to sustain high query volume from instances in your VPC to an on-prem DNS server.
This is a turn-key solution, ready to use without eventually dealing with multiple devices with its own firmware and caveats revealing after where put to work together.
The closest to that is that AWS managed rack that works with the web APIS you know already
I would assume so. They've said before you can make modifications to the firmware and deploy it yourself if you so wish. That's one of the major reasons that making the firmware open source is so useful.
It's meant for orgs running at a certain scale, but you'd be surprised how early that starts making sense. AWS isn't exactly paying the economy of scale savings on to you.
True but at for companies operating at scale they not only operate on AWS but on other cloud providers as well as legacy data centers … but business wise it’s a hard sell , it may sell for couple of cycles to build a new dc or use an existing ones but then it will be back to the cloud again for many more cycles.
I've found that the scale these start making sense is only a handful of racks. We're not talking full DCs, but a room in a building or some colo space.
And beyond that, there's all sorts of weird environments that need a lot of local compute. You'd be shocked how many servers are on a cruise ship for one example among many.
I know you are half joking, but it would really be helpful to have ball-park pricing available. Are we talking Sun-level markups here, or how should we think about it? Given the enterprise sales contact form, I'm thinking yes, but I'd love to know for sure.
I am not in sales and so I hesitate to speak on it in case I am incorrect, but the way that I personally think about it is that it is true that it is not an inexpensive product: there's a LOT of computer here. But the goal is to be competitively priced.
The last time (one of them) Oxide hit HN there were some ballpark estimates based on the CPUs in use, switches etc. Someone else said 500K and up.
I wish there was a 4U version (10-25K but I don't think that they could come close to that price point - regardless even that is out of reach for me to ever get to play on one :-/ )
Coupling vs Decoupling is not some one-sided thing. It's a major trade-off.
One of the most obvious examples of the problem with this approach is that they're shipping previous generation servers on Day 1. One can easily buy current generation AMD servers from a number of vendors.
They will also likely charge a significant premium over decoupled vendors that are forced to compete head-to-head for a specific role (server vendor, switch vendor, etc).
Their coupling approach will most likely leave them perpetually behind and more expensive.
But there are advantages too. Their stuff should be simpler to use and require less in-house expertise to operate well.
This is probably a reasonable trade-off for government agencies and the like, but will probably never be ideal for more savvy customers.
And I don't know how truly open source their work is but if it's truly open source, they'll most likely find themselves turned into a software company with an open core model. Other vendors that are already at scale can almost certainly assemble hardware better than they can.
If they standardize and open the server shape and plug interface then it gets really cool. Then I could go design a GPU server myself and add it to their rack. The rack is no longer a hyperconverged single-user proprietary setup and becomes something that can be extended and repurposed.
> They will also likely charge a significant premium over decoupled vendors
It seems like they're trying to hit a middle ground between cloud vendors and fully decoupled server equipment companies.
Using Oxide is likely cheaper over the life of the hardware than using a cloud vendor. A company who already has in-house expertise on running racks of systems may be less the target market here than people who want to do cloud computing but under their own control.
> A company who already has in-house expertise on running racks of systems may be less the target market here than people who want to do cloud computing but under their own control.
True, but Oxide may find themselves competing against Dell or HP if they adopt Oxides software for their respective servers. Additionally, Oxide may find itself competing against consultants and vendors in specialized verticals (e.g. core Banking software + Oracle DB + COTS servers + Oxide software). Oxide, and their competitors are going for people who used to buy racks of Sun hardware.
HP and Dell would have to fundamentally change the way they design hardware and software to be that kind of threat, and if that ever happens I think I would be pretty okay with that outcome.
> One of the most obvious examples of the problem with this approach is that they're shipping previous generation servers on Day 1. One can easily buy current generation AMD servers from a number of vendors.
> Their coupling approach will most likely leave them perpetually behind
This is a startup that took years to get their initial hardware developed. The time between this version and the version using the next version of AMD chips will be shorter than the time it took to develop this product. This is not an inherent issue with coupling vs decoupling.
Also, most servers are rarely running on the most recent cpus anyway. At least in companies I've worked at with on-site hardware they're usually years (sometimes even a decade) out of date getting the last life sucked out of them before too many internal users start complaining and they get replaced.
Coupling requires more integration work, including writing and testing custom firmware. Oxide will be a tiny market player for a long time, even if things go very well. Are AMD and Broadcom really going to spend as much time helping Oxide as they do helping Dell? Of course not, Oxide's order volume will be a rounding error.
I'm sure they'll improve their processes over time but the lag will probably always be a non-zero value. Hopefully they'll be able to keep it low enough that it's not an important factor but as a customer it's certainly something one should consider.
It would be surprising if they don't run into some nasty issue that leaves their customers 6+ months behind on servers or switches at some point.
From listening to their talks they've actually gotten pretty good direct responses from AMD and AMD likes them quite a bit. They've done what no other system integrator has done and brought up the CPU without using AMD's AGESA firmware bootloader. By simplifying the system they've reduced the workload on what they need to handle.
As to your second point, unless AMD somehow becomes supply constrained and only wants to ship to their most important customers first I don't see a future where there would be any lag. Again, the delay this time is from how long it took from company start until product release. Future delays will be based on the time it takes from them getting early development parts to released products, which they could even possibly beat Dell to market on given the smaller company size and IMO more skilled employees.
> It would be surprising if they don't run into some nasty issue that leaves their customers 6+ months behind on servers or switches at some point.
I mean they've already hit tons of nasty issues, for example finding two zero-day vulnerabilities in their chosen security processor. They've shown they can work around issues pretty well.
> it would be surprising if they don't run into some nasty issue that leaves their customers 6+ months behind on servers or switches at some point.
I just think your premise is wrong - most customers don't care about not having the absolute latest and greatest. Indeed they will often avoid them because
1. They are new so more likely to have as yet undiscovered issues ( hardware or drivers ).
2. If you buy top end, they sell at a premium well above their performance premium.
ie the customers who are perennially chasing the latest hardware are in the minority.
Most customers care about having the best of the available options. Rarely would any company deliberately choose to be behind where their competitors can be.
1. The way to run into undiscovered issues is to choose a completely custom firmware/hardware/software stack that almost no one else in the world is running.
2. Not sure where you're getting this from. There is almost always a price:performance calculation that results in current generation smashing the previous generation with server and switch hardware. Often this means not buying the flagship chips but still the current generation.
And a major reason to get off old generations of hardware is that they become unavailable relatively quickly. It's always easier to buy current generation hardware than previous generation hardware, especially a couple years into the current generation. This has nothing to do with chasing the latest hardware.
> And a major reason to get off old generations of hardware is that they become unavailable relatively quickly.
That's not in the customers interests per se- in fact it's a pain. Having control of their own stuff could mean they could offer a much longer effective operational life.
> The way to run into undiscovered issues is to choose a completely custom firmware/hardware/software stack that almost no one else in the world is running.
What breaks stuff is change - sure when they are starting up it's higher risk - but again if they can manage the lifecycle better, not have change for changes sake, then they could be much more reliable.
> Not sure where you're getting this from.
I was talking about not taking the flagship stuff - which is typically a few months ahead of the best price/performance stuff.
> will probably never be ideal for more savvy customers
IDK about every use case, but slightly older generations of CPUs would affect me roughly zero. I'm sure there are things so compute-intensive that one would care very much, but a lot of people probably wouldn't bat an eye about that, and not because they're unsavvy.
To the extent that these things are supported as a whole by the vendor rather than a bunch of finger pointing though, that could be massive, specifically in terms of how many staff members you could "not hire" compared to if you had to employ someone to both build and continually maintain it.
I'm posting this not to invalidate what you're saying, just to say that a little predictable upfront amount of money (the premium) will be spent very happily by lots of people who value predictability and TCO over initial price.
If you're not rapidly scaling it probably doesn't matter. But if you're still buying (and maybe even using) Haswell CPUs in 2023, you may be missing out in a big way.
A moderately large Haswell cluster is equivalent in power to a moderately powerful modern server.
No not buying new, just using what was bought years ago. It still works, it does the job. Is it the best performance per watt, clearly no but the budget for electricity and the budget for new capital expenses are two different things.
If you go on Google cloud and select an E2 instance type (atleast in `us-central1` where my company runs most of it's infra) you'll usually get Broadwell chips.
I don't see it as a big deal - rather, I see it as a huge amount of venture cap spent on some very bright people to build something no one really wants, or, at best, is niche.
Also, it has little to do with the cloud; it is yet another hyperconverged infra.
Weirdly, it is attached to something very few people want: Solaris. This relates to the people behind it who still can't figure out why Linux won and Solaris didn't.
Right, who wants or benefits from open source firmware anyway.
Also there are many situations where renting, for example a flat makes a lot of sense. And there are many situations where the financials and or enabled options of owning something make a lot of sense. Right now, the kind of experience you get with AWS and co. can only be rented, not bought. Some people want to buy houses instead of renting them.
So OpenBMC is fine (happy for them!), but having open firmware is much deeper and broader than that: yes, it's the service processor (in contrast to the BMC which is a closed part on Dell machines) -- but it's also the root-of-trust and (especially) the host CPU itself. We at Oxide have open source software from first instruction out of the AMD PSP; I elaborated more on our approach in my OSFC 2022 talk.[0]
Dell uses trusted platform modules (TPM). It's a separate chipset than the BMC chipset.
For a mostly open source solution, not only would you need open source BMC firmware, you must have an open source UEFI/BIOS/boot firmware like CoreBoot, LinuxBoot, Oreboot, Uboot, etc.
Well, you can buy your own hardware and set it up with OpenStack and use it as a private cloud. Companies like Canonical or Redhat make a lot of money by providing software (mostly open source) to support exactly that use case.
> Well, you can buy your own hardware and set it up with OpenStack and use it as a private cloud. Companies like Canonical or Redhat make a lot of money by providing software (mostly open source) to support exactly that use case.
Sure you can, but then who will diagnose and fix your hardware/OS interaction problems when you have parts from five vendors in the mix?
If you haven't lived through this, the answer is: nobody. Everyone points fingers at the other 4 and ignore your calls.
Back in the day you could buy a fully integrated system (from CPU to hardware to OS) from Sun or SGI or HP and you had a single company to answer all the calls, so it was much better. Today you can't really get this level of integration and support anymore.
(Actually, you probably can from IBM, which is why they're still around. But I have no experience in the IBM universe.)
This is why Oxide is so exciting to me. I hope I can be in a company that becomes a customer at some point.
>Sure you can, but then who will diagnose and fix your hardware/OS interaction problems when you have parts from five vendors in the mix?
Dell is a single vendor that will diagnose and fix all of your hardware issues.
With Oxide you're locked into what looks like a Solaris derivative OS running on the metal and you're only allowed to provision VMs which is a huge disadvantage.
I run a fleet of over 30,000 nodes in three continents and the majority is Flatcar Linux running on bare metal. Also have a decent amount of RHEL running for specific apps. We can pick and choose our bare metal OS which is something you cannot do with Oxide. That's a tough pill to swallow.
> Off by a few orders of magnitude. Dell on-site SLA with pre-purchased spares was about 6 hours.
You're talking about replacement parts. Yes Dell is good about that.
The discussion above is asking them to diagnose and fix a problem with the interaction of various hardware components (all of which come from third parties).
But they _are_ writing the firmware that runs most of them and need to understand those devices at a deep level in order to do that, unlike Dell. Dell slaps together hardware and firmware from other vendors with some high level software of their own on top. They don't do the low level firmware and thus don't understand the low level intricacies of their own systems.
No they're not unless I'm mistaken. They're not writing the firmware that runs on the NVMe drives, nor the NICs (they're not even writing the drivers for some of the NICs), etc.
There's a line in the sand that you must cross when it comes to understanding the true nature of the componentry that you're using. At the end of the day, your AMD CPUs may be lying to you, to all of us, but we just don't know it yet.
I'm not speaking hypothetically. If you hit a "zero-day" bug that Dell has never seen it's going to take time. And somehow every large customer finds bugs that Dell certification didn't.
> And somehow every large customer finds bugs that Dell certification didn't.
It's a law of computer engineering.
In the Apollo 11 decent sequence the Rendezvous Radar experienced a hardware bug[0] not uncovered during simulation. They found it later, but until then, the solution was adding a "turn off Rendezvous Radar" checklist item.
[0] The Rendezvous Radar would stop the CPU, shuttle some data into areas it could be read, and woke the CPU back up to process it. The bug caused it to supuriously do this dance just to tell it "no new data", which then caused other systems to overload.
> Dell is a single vendor that will diagnose and fix all of your hardware issues.
I've been a Dell customer at a previous company. I know for a fact that's not true.
I had a support ticket for a weird firmware bug open for two years, they could never figure it out. I left that job but for all I know the case is still open many years later.
Dell doesn't know how to fix things like that because they don't design and engineer the systems they sell. Dell is a reseller who puts components together from a bunch of vendors and it mostly works but when it doesn't, there's nobody on staff who can fix it.
I've been a Dell customer for decades at this rate and I know for a fact it's true.
I've had support tickets open for all kinda of weird firmware, hardware, etc. bugs and they've been well resolved, even if it meant Dell just replaced the part with something comparable (NIC swap).
>Dell doesn't know how to fix things like that because they don't design and engineer the systems they sell.
Of course they do. That's like saying Oxide doesn't know how to fix stuff because they don't design the CPU, NVMe, DIMMs, etc. Oxide is still going to vendors for these things.
Ironically, it was Dell's total inability to resolve a pathological rash of uncorrectable memory errors very much is part of the origin story of Oxide: this issue was very important to my employer (who was a galactic Dell customer) and as the issue endured and Dell escalated internally, it became increasingly clear that there was in fact no one at Dell who could help us -- Dell did not understand how their own systems work.
At Oxide, we have been deliberate at every step, designing from first principles whenever possible. (We -- unlike essentially everyone else -- did not simply iterate from a reference design.)
To make this concrete with respect to the CPU in particular, we have done our own lowest-level platform enablement software[0] -- we have no BIOS. No one -- not the hyperscalers, not the ODMs and certainly not Dell -- has done this, and even AMD didn't think we could pull it off. Why did we do it this way? Because all along our lodestar was that problem that Dell was useless to us on -- that we wanted to understand these systems from first principles, because we have felt that that is essential to deliver the product that we ourselves wanted to by.
There are plenty of valid criticisms of Oxide -- but that we don't understand our system simply isn't one of them.
As a side question, what's the name of your custom firmware that is the replacement of the AGESA bootloader? I tried searching on the oxide github page but couldn't find anything that seemed to fit that description.
(The AGESA bootloader -- or ABL -- is in the AMD PSP.) In terms of our replacement for AGESA: the PSP boots to our first instruction, which is the pico host bootloader, phbl[0]. phbl then loads the actual operating system[1], which performs platform enablement as part of booting. (This is pretty involved, but to give you a flavor, see, e.g. initialization of the DXIO engine.[2])
Thanks, are the important oxide branches of illumos-gate repo (and any other cloned repos) defined anywhere? I definitely wouldn't have found that branch without you mentioning it here.
Interesting enough I also ran into something somewhat related with Dell that they were not able to resolve so they ended up working in a replacement from another vendor.
Nonetheless, it is quite interesting what you've built, but as the end user I'm not quote convinced that it matters. Sure you can claim it reduces attack vectors and such but we'll still see Dells and IBMs in the most restricted and highest security postured sites in the world. Think DoD and such. Core/libreboot with RoT will get me through compliance the same.
The software management plane y'all built is the headlining feature IMHO, not so much what happens behind the scenes that the vast majority of the time will not have a fatal catastrophic upstream effect.
>There are plenty of valid criticisms of Oxide -- but that we don't understand our system simply isn't one of them.
That's not what I said. There's a line in the sand that you must cross when it comes to understanding the true nature of the componentry that you're using. At the end of the day, your AMD CPUs may be lying to you, to all of us, but we just don't know it yet.
It's ironic coming from a company who's CTO has harped about containers on bare metal for years. Maybe a large swath only need to deploy VMs, but the future will most definitely involve bare metal for many use cases, and oddly Oxide doesn't support that currently.
See the pattern? Dell only care about the big guys.
Set aside the childish tone ...
> Dell is a single vendor that will diagnose and fix all of your hardware issues.
There are two anecdotes here disagreeing with you, and frankly that's enough to say what you said above isn't true, not universally so. I doubt Odixe is targeting big deployment like yours, but more like theirs. Whether they will succeed is another matter, but they do have a valid sales pitch and the expertise to pull it off.
The fact that it's not on linux is one of the great things about it. There is too much linux on critical infrastructure already and the monoculture just keeps on growing.
At least with Oxide there is a glimmer of hope for a better future in this regard.
When you're deploying VMs, which is the use case here, the substrate OS becomes significantly less important. Those VMs will mostly just be linux.
Yes they are using illumos/Solaris to host this but they don't sell on that, they sell on the functionality of this layer — allowing people to deploy to owned infra in a way that is similar to how they'd deploy to AWS or Azure. How much do you ever think about the system hosting your VM on those clouds? You think about your VMs, the API or web interface to deploy and configure, but not the host OS. With Oxide racks the customers are not maintaining the illumos substrate (as long as Oxide is around).
You could be right about demand, there is risk in a venture like this. But presumably the team thought about this - I think folks who worked at Sun, Oracle, Joyent, and Samsung and made SmartOS probably developed a decent sense of market demand, enough to make a convincing case to their funders.
> When you're deploying VMs, which is the use case here, the substrate OS becomes significantly less important. Those VMs will mostly just be linux.
Now you need to know both the OS they chose and the OS you chose...
(No, I don't believe it'll be 100% hands-off for the host. This is an early stage product, with a lot of custom parts, their own distributed block storage, hypervisor, and so on.)
This true for other hypervisors too. Enterprises are still paying hundreds of millions to VMware, who knows what's going on in there?
I wouldn't have picked Opensolaris, but it's a lot better than other vendors that are either fully closed source, or thin proprietary wrappers over Linux with spotty coverage and you're not allowed to touch the underlying OS for risk of disrupting the managed product.
What's more important is that the team actually knows Illumos/Solaris inside out. You can work wonders with a less than ideal system. That said, Illumos is of high quality in my opinion.
Seems risky considering how small of a developer pool actively works on illumos/Solaris. The code is most definitely well engineered and correct, but there are huge teams all around the world deploying on huge pools of Linux compute that have contributed back to Linux.
They had a bug in the database they are using that was due to a Go system library not behaving correctly specifically on illumos. They've got enough engineering power to deal with such a thing but damn..
Linux grew up in the bedrooms of teenagers. It was risky in the era of 486 and Pentiums. The environment and business criticality of a $1-2M rack-size computer is quite different.
I had similar thoughts about VMware (large installations) back in the day. Weird proprietary OS to run other operating systems? Yet they turned out fine.
This appears to be a much better system than VMware, is free as in software, and it builds upon a free software operating system with lineage that predates Linux.
I say this in the most critical way possible, as someone who has built multiple Linux-based "cloud systems", and as a GNU/Linux distribution developer: I love it!
Completely tangential, but this reminds me of an interview I had for my first job out of college in 1995. I mentioned to the interviewer that I had some Linux experience. "Ah, Linux" he said. "A cool little toy that's gonna take over the world".
In hindsight of course it was remarkably prescient. This from a guy at a company that was built entirely around SGI at the time.
It was totally a risky choice for companies in the 1990s and early 2000s to put all their web stuff onto Linux on commodity hardware instead of proprietary Unix or Windows servers. Many did it when their website being up was totally mission critical. Lots did it on huge server farms. It paid off very quickly but it's erasing history to suggest that it didn't require huge amounts of guts, savvy and agility to even attempt it.
Indeed, for me GNU/Linux was always a cheap way to have UNIX at home, given that Windows NT POSIX support never was that great.
The first time I actually saw GNU/Linux powering something in production was in 2003, when I joined CERN and they were replacing their use of Solaris, and eventually alongside Fermilabs came up with Scientific Linux in 2004.
Later at Nokia, it took them until 2006 to consider Red-Hat Linux a serious alternative to their HP-UX infrastructure.
This is a skewed view - the critical piece that made Linux "enterprise-ish" was the memory management system that was contributed by IBM, part of the SCO lawsuit
I have a feeling they knew exactly from the start who their customers would be: People who have the budget to care about things like trust and observability in a complex system. But these would also be the kind of customers who require absolute secrecy and so this why you don't hear about them even though they might have bankrolled a sizable portion of the operation. Just like the first Cray to officially be shipped was actually serial number 2...
Back in the day... Sun Micro was a GOAT and pushed the envelope on Unix computing 20-30 years ago. Solaris was stable and high performing.
I don't run on-prem clusters or clouds but know a couple people who do and, at large enough scale, it is a constant "fuck-shit-stack on top of itself" (to quote Reggie Watts). There is almost always something wrong and some people upset about it.
The promise of a fully integrated system (compute HW, network HW, all firmware/drivers written by experts using Rust wherever possible) that pays attention to optimizing all your OpEx metrics is a big deal.
It may take Oxide a couple more years to really break into the market in a big way, but if they can stick it out, they will do very well.
It won't. In the same way that AWS customers aren't debugging hypervisor, or Dell customers aren't debugging the BIOS, or Samsung SSD customers aren't debugging the firmware. Products choose where to draw the line between customer-serviceable parts and those that require a support call. In this case, expect Oxide to fix it when something doesn't work right.
When Apple supports OSX for consumers, they don't exactly surface the fact that there's BSD semi-hidden in there somewhere.
That's because they own the whole stack, from CPU to GUI and support it as a unit. That's the benefit of having a product where a single owner builds and supports it as a whole.
My impression of Oxide is that that's the level of single source of truth they are bringing to enterprise in-house cloud. So, I strongly doubt the innards would ever become customer-facing (unless the customer specifically wants that, being open source after all).
Apple is a horrible example, with Apple when you have a problem, you often end up with an unfixable issue that Apple won't even acknowledge. You definitely don't want to taint Oxide's reputation with that association.
As for why I think Helios will become customer facing: Oxide is a small startup. They have limited resources. Their computers expensive enough to be very much business critical. You'll get some support by Oxide logging in remotely to customer systems and digging around, but pretty soon the customer will want to do that themselves to monitor/troubleshoot the problems as they happen.
Imagine you're observing a recurring but rare I/O slowdown that seems to trigger under some certain conditions, and tell me a competent sysadmin wouldn't want to log in on all the related boxes (client Helios, >=3 server Helioses for the block store) and look at the logs & stats.
Apple is a great example of the benefits of an integrated system where the hardware and software are designed together. There are tons of benefits to that.
What makes Apple evil (IMO, many people disagree) is how everything is secret and proprietary and welded shut. But that doesn't take away from the benefits of an integrated hardware/software ecosystem.
Oxide is open source so it doesn't suffer from the evil aspect but benefits from the goodness of engineered integration. Or so I hope.
Android is potentially a better example. Compare Android to trying to get Linux working on <some random laptop>. You might get lucky and it works out of the box or you might find yourself in a 15 page "how to fix <finger print reader, ambient light sensor, etc>" wiki where you end up compiling a bunch of stuff with random patches.
Afaik Android phones tend to have a lot more hardware than your average laptop, too (cell modem, gps, multiple cameras, gyro, accelerometer, light sensors, finger print readers)
In practice I don't think it's as good as in theory. I had Apple Macbook Pro with Apple Monitor, and 50% of the time when unplugging the monitor the laptop screen would stay off. Plugging back in to the monitor wouldn't work at that point so all I could do was hold the power button to force it off and reboot. That's with Apple controlling the entire stack - software, hardware, etc.
I think the real benefit is being able to move/deprecate/expand at will. For example, want an app that would require special hardware? You can just add it. Want to drop support for old drivers? Just stop selling them and then drop (deprecate) the software support in the next release.
I fully agree about the evilness, and it baffles me how few people do!
> As for why I think Helios will become customer facing: Oxide is a small startup. They have limited resources.
Have you looked at the pedigree of many of the people behind the project? I don't say this because "these guys smart", but because these guys bent over backwards for their customers when they were Sun engineers. Bryan didn't write dtrace for nothing.
> Imagine you're observing a recurring but rare I/O slowdown that seems to trigger under some certain conditions, and tell me a competent sysadmin wouldn't want to log in on all the related boxes (client Helios, >=3 server Helioses for the block store) and look at the logs & stats.
I think you're simultaneously over-estimating and under-estimating the people who will deploy this. There's a lot of companies who would want a "cloud in a box" that would happily plug hardware in and submit a support ticket if they ever find an issue, because their system engineers either don't have the time, desire, or competence (unfortunately common) to do anything more. The ones who are happy to start debugging stuff on their own would have absolutely wonderful tooling at their fingertips (dtrace) and wouldn't have any issue figuring out how to adapt to something other than Linux (hell, I've been running TrueNAS for the better part of a decade and being on a *BSD has never bothered me).
Apple is the survivor of 16 bit home micros integration, PC clones only happened as IBM failed to prevent Compaq's reverse engineering to take over their creation, they even tried to get hold of it afterwards via PS/2 and MCA.
As we see nowadays on tablets and laptops, most OEMs are quite keen in returning back to those days, as otherwise there is hardly any money left on PC components.
Funny how you mentioning BSD got me to thinking of Sony Playstation and Nintendo Switch. Which are proprietary and not user serviceable. A Steam Deck, Fairphone, or Framework laptop is each less proprietary and more FOSS stack, and user serviceable. Which a user may or may not want to do themselves; at the very least they can pay someone and have them manage it.
Also, Apple is just the one who survived. Previously I'd have thought of SGI, DEC, Sun, HP, IBM, Dell some of whom survived some not.
Those three consumer products I mentioned each provide a platform for a user and business space to floroush and thrive. I expect a company doing something similar for cloud computing to want the same. But it will require some magick: momentum, money, trust. That kind of stuff, and loads of it. (With some big names behind it and a lot of FOSS they got me excited, but I don't matter.)
If you have a bug in how a lambda function is run on AWS, do you find yourself looking for the bug in firecracker? It is open source, so you technically could, but I just don't see many customers doing that. Same can be said about KNative on GCP.
Their choice in foundation OS (for lack of a better term) really should not matter to any customer.
Ok but then that is purely additive then, right? Like, "have to find someone with Illumos expertise to fix something that was never intended to be customer-facing" may not be easy, but is still easier than the impossibility of doing the same thing on AWS / Azure / Google Cloud.
Its a huge deal. I'm biased though because my own takes on how things should evolve were very similar. I was however completely unsuccessful in getting those ideas into production! And that, that is a huge deal. Through out my career it has been interesting to meet people with great ideas and then they are unable to get them into production, and when the idea does come into production everyone feels like "Wow, this is so obvious why didn't we do it sooner?" and some folks are banging their head against the wall :-).
One of the more interesting discussions I had during my tenure at Google was about the "size" of the unit of clusters. If you toured Google you got the whole "millions of cheap replaceable computers" mantra. Sitting in Building 42 was a "rack" which had cheap PC motherboards on "pizza dishes" without all that superfluous sheet metal. Bunches of these in a rack and a festoon of network cables. What are the "first class" elements of these machines? Compute? Networking? Storage? Did you replace components? Or a whole "pizza slice" (which Google called an 'index' at the time). Really a great systems analysis problem.
FWIW I'm more of a "chunk" guy (which is the direction 0xide went) and less of a "cluster" guy (which is the way Google organized their infrastructure). A lot of people associated with 0xide are folks I worked with at Sun in the early days and during that period the first hints of "beowulf" clusters vs "super computers", was memory one thing (UMA) or did it vary from place to place (NUMA). I have a paper I wrote from that time about "compute viscosity" where the effective compute rate (which at the time largely focused on transactional databases) scaled up with resource (more memory more transactions/sec for example) and scaled down with viscosity (higher latencies to get to state meant fewer transactions/sec) Sun was invested heavily in the TPC-C benchmarks at the time but they were just one load pattern one could optimize for.
These guys have capitalized on all that history and it is fucking amazing! I just hope they don't get killed by acquisition[1].
[1] KbA is a technique where people who are invested in the status quo and have resources available use those resources to force the investors in a disruptive technology to sell to them and then they quietly bury the disruptive technology.
Can you clarify a bit on what you mean by "chunk" guy? Are you alluding to the ability to distribute work by an isolation mechanism such as cgroups vs machine a-la borg/google?
More on infrastructure composition software is an abstraction above that.
Is the unit of composition a rack (chunk), a server (smaller chunk), or a blade (smallest chunk)? In what I think of as classic systems architecture you've got a 'store' (storage), 'state' (memory), 'threads' (computation), and 'interconnect' (fabrics). In the 90's a lot of folks focused on fabrics (Cray, Connection Machine, Sun, etc) somewhat on threads (compute blades), and state came along for the ride. How these systems were composed was always a big thing, then along came the first Beowulf clusters that used off the shelf motherboards (a "chunk" of threads/state/store) with a generic fabric (Ethernet). Originally NASA showed that you could do some highly parallel processing on these sorts of systems and Larry and Sergei at Stanford applied it to the process of internet search.
Collectively you have a 'system resource' and with software you can make it look like anything you want. When you do compute with it, its performance becomes a function of its systems balance and the demands of the workload. Its all computer sciencey and yes there is a calculus to it. This isn't something that most people dive into (or are even interested in[1]) but it was one of the things that captured my imagination early on as an engineer. I was consumed with questions like what was the difference between a microprocessor, a mini-computer, a workstation, and a mainframe? Why do they each exist? What does one do that they other can't? Things like that.
[1] At Google I worked in what they called 'Platforms' early on and clearly most of the company didn't really care about the ins and outs of the systems bigtable/gfs/spanner/etc ran on, they just wanted APIs to call. But they also didn't care about utilization or costs. By the time I left some folks had just figured out (and one guy was building his career on) the fact that utilization directly affected operational costs. They still hadn't started thinking about non-uniform rack configurations for different workloads.
> I just hope they don't get killed by acquisition.
Private equity in the US has collectively determined that no company shall exist outside of investment ownership. I don't know what the ownership structure looks like, but generally speaking, it seems that nearly everyone has a "fuck you" number. Now that Oxide is venturing into Dell and HP's turf, I worry someone will get a fix on what Brian's number is.
>They sell servers, but as a finished product. Not as a cobbled together mess of third party stuff where the vendor keeps shrugging if there is an integration >problem. They integrated it.
> Ironically this looks like the realization of Richard Stallman's dream where users can help each other if something doesn't work.
that's only true if you think that "users" means "people who operate cloud computers", which is about as far from understanding what Stallman is talking about as is possible. Someone who makes SaaS and runs it on an Oxide computer is no less of a rentier capitalist than someone who makes SaaS and runs it on AWS.
I always thought it's just about the hardware but it seems like it also includes it's own kind of virtualization / provisioning interface for VMs, firewalls and also an overview over running software.
Does this "lock" you into the Oxide platform and you just buy into the whole thing instead of buying some server from Dell, then running some Proxmox like software and a Docker host?
I saw a demo from them and it looked nice but then they told me they are stopping at the VM and storage blob level of abstraction.
Virtually all the value we get from cloud is in their managed offerings of complex and difficult to admin systems like Kubernetes, Postgres, and managed storage layers.
That’s where all the value is but it’s also where all the cost is. If you just want compute, storage, and bandwidth all those can be had at commodity prices. Look at Hetzner, Hivelocity, FDCServers, DataPacket, OVH, or VPS providers like Vultr. You can get dozens of cores, terabytes of SSD, and gigabits of unmetered bandwidth for a few thousand dollars or less. It’s very cheap.
Without the high level managed stuff we would be off cloud five minutes from now.
I’m sure there is a market for this among people who run on premise data centers, but I think they would have a much larger market if they went further up the software stack. Right now they just look like they are competing with Dell and Supermicro, not Amazon or Google Cloud.
Exactly, man! That's the way it should be done. It's not a company's business to decide how much your life should cost. It's its business to reward you for the value you provide, and that value is not tethered to your location, so neither should be your salary!
This really is one of these things where every employee in a cheap cost of living area will say "Exactly, man!" and people in an expensive area will see it differently because it might cap them lower than what they could get.
I always find that a very naive point of view, of course I would want to earn a US tech salary while living somewhere in the country side, who wouldn't. But I'm also aware that this is not how the world works in reality, there's different tax systems, different expense costs and we don't live in a global one-market world.
I find the strategy of defining different "zones", like most of the remote first / salary transparency companies much more realistic.
> This really is one of these things where every employee in a cheap cost of living area will say "Exactly, man!" and people in an expensive area will see it differently because it might cap them lower than what they could get.
Well the co-founders live in the Silicon Valley area, with their physical HQ being in Emeryville:
Early stage VC-backed startup compensation is very different from later stage companies or bootstrapped companies. It's very common for founders or early employees to receive a lower salary and receive equity instead.
I don't know how you're coming to that conclusion. 200k's not going to buy you a large house, but it'll comfortably pay rent/food/savings for a 2-3 bedroom house or apartment for a family of 2-4 in all but the very most expensive parts of the Bay.
For mortgages, you'd need to be looking in the cheaper parts of the Bay, but that still means "dense, boring suburb" as opposed to "crime-ridden slum".
If you owned a home in the Bay Area prior to 2020 and refinanced down to a sub 3% mortgage 200k is plenty. If you’re trying to buy now on that salary it will be challenging.
Hmmm...I actually had to look up what crab mentality meant but I don't see how it applies here. From my quick lookup, crab mentality apparently refers to a reactionary state of mind where a person wants to sabotage another's success even when it doesn't directly impact their own.
In the case of everyone making the same salary, you could certainly still sabotage someone's success but I don't see how having the same salary makes this type of mentality _more_ likely than at a company with a more typical salary distribution.
My concern with everyone having the same salaries is that, potentially, employees have less motivation to excel in their individual work and are more likely to do the minimum to just stay in good standing with their employer and not get fired. Maybe a company can offset the lack of direct financial motivation with more of a team motivation that the financial success of the company as a whole results in financial reward for the individual or some other way of recognizing individual success inside the company.
I am doubtful this flat salary structure will result in a more successful company overall but I do think it's good to try new things. And yes this has probably been tried a number of times before but maybe not exactly like this. Or maybe some other external variables have changed w.r.t. other attempts in the past and this time it works. The typical salary structures we see in US companies today are the result of a large number of trials and errors and learning.
Ah I think you misunderstand. I think employees in the west being pissed that someone in a third world country makes the same salary as them is crab mentality.
I totally understand why companies want to pay less though. It's massive cost savings and it makes sense for them to hire for less money.
> of course I would want to earn a US tech salary while living somewhere in the country side
It's not about what you want, it's about knowing your value. If your work is worth a SF salary then that's what you should be getting.
Moving from Idaho to SF doesn't magically make you more productive. The company knows it's still getting more value from you than what you're being paid. They just want to keep more of that value for themselves whenever possible.
Have some respect for yourself and know your worth
I am so confused by this. I demand more in the Bay as Bay area landlords and their Nimby pals are exploitative jerks and California taxes and fees add up quick. Why is that cleanly separated from the discussion? I see it as more “I will demand more here than other places” vs. “I know my worth and it’s exactly X”
Being in SF makes the market for your labor more competitive. If I'm living on a ranch in rural Idaho, I would interact with very few people on a given day, and most of them would be the same people I interacted with yesterday. In SF, I'd be interacting with far more people, and far more new people, with a much higher probability of those people working in tech, some subset of whom will be willing to offer me a job.
> Moving from Idaho to SF doesn't magically make you more productive.
If your entire world consists of staying home and interacting remotely with a company, then you are correct: Location doesn't change anything.
However, moving to a high-energy city with a high density of experienced engineers and tech companies can increase your rate of learning, career advancement, and experience much more rapidly than living in a smaller city. You have to actually branch out and interact with local companies and people, but it does happen.
But this is all beside the point. Hiring is a labor market. Developers who live in SF have more high-paid job options to choose from than someone living in Idaho. As a result, you need to bid more to get them into your company. Hence, the higher salary.
The discussion about cost of living misses this point. The real reason developers from places like SF get paid more is because if you don't pay them wages that are competitive with their local companies, they're just going to walk away and take any number of higher paying jobs they have access to.
Just like Wikipedia and many other information sources. Don't trust one source blindly, do your research and you'll be fine. It gives a good enough general indication. A comparison like this will always depend on too many factors to make it applicable to everyone in any case.
My employer takes cost of living into account by multiplying your salary by the numbeo factor for where you live. Base salaries are all calculated for Cologne (HO) and then adjusted by the relative cost of living factor for your locale. (we're entirely remote)
I am personally not a fan of this: early in my career, a colleague moved from the Bay Area to Boston to try to save his marriage (which sadly didn't work) -- and our employer adjusted his salary down accordingly. As you might imagine, the difference between the Bay Area and Boston is minimal -- they adjusted his salary down something like 4% -- but I just recall how dispiriting it was for him to have this massive, stressful (expensive!) life transition exacerbated needlessly. Employers don't factor in other elements of cost-of-living (can you imagine giving someone a pay bump when a kid goes to college -- or a reduction when they graduate?!), and I don't believe geography should be factored in either: pay people for their work, not their ZIP code.
Whilst I completely understand your viewpoint, I have a more nuanced take on it (understandably, as I work there). Firstly, we don't pay peanuts. We have never taken outside funding; it has always been a company goal to grow organically. As such, we don't have as much cash to go throwing around like VC-funded startups. However, as I said we still pay well; yes I could make more by taking a job in London (I'm in the UK), but then I sacrifice so many intangible things. Working remotely has allowed me to be present whilst my 18 month-old grows up, and a 4 day work-week has helped even more too. Unlimited holidays, very flexible working hours and being treated like an adult about how you organise them, a flat company structure, a company which genuinely cares about employees and their wellbeing - all of these things add up. Salary isn't everything. Whilst no job will ever be perfect, this company is by far the best place I have ever (and will likely ever) work. Employee satisfaction is always very high and salary is rarely even a factor (let alone the main reason) if people decide to move on.
Assuming interchangeable human resources there are basically 2 options, either the company is extracting more value from everyone than the salaries they pay, in that case paying fairly would eat into the company's bottom line. Poor CXOs would not turn extra profits by keeping less fortunate employees on low salaries but just the regular one. Or the highest paid employees are not pulling their weight and their salaries are already subsidized by the rest of the company, which is also not quite fair.
Nevertheless, I agree everyone looks at this problem from their own POV, however it should not be the norm to provide equal compensation for equal work.
I understand! That's what I suspected, but wasn't sure. I tried understanding it for a minute then thought I'll just ask you! :) haha
Also, thank you for the links. I'll read this because I'm interested in knowing more about thinking behind this. My instinct is in this direction regarding salaries, but I want to develop more clarity. :)
If that's your blog, well done! I am totally on board with everything you say there. I've been thinking this way for years, thanks for articulating some more reasons why this is really important! Love your work! :)
Replace "somewhere in the country side" with "some other place where the cost of living is lower than the highest one in the country" if it makes you happier.
Your perspective on egalitarian salaries is intriguing, but it's important to remember you can't speak for others regarding their intentions or their whys, or what it is for them. You can't pretend to define that for someone who is not you.
While you argue it's naive and cite different tax systems and costs, companies are successfully adopting this approach even in high-cost areas like the Valley. It might be hard to argue that the world doesn't work like that in reality, given that it’s already happening.
While it may not be widespread, it doesn’t mean it’s without merit or unrealistic. After all, remember that today's 'unrealistic' could be tomorrow's norm! It's tempting to think that people in less expensive areas would be the main proponents of a uniform salary, but the reality is nuanced. Preferences are likely influenced by a variety of factors, not just cost of living.
Your later points about the differences between compensation at different company stages are well taken, however it could be difficult to assert how much this dynamic affects preferences given the practice is not limited to early stage companies and equity vests often fail to yield returns.
In addition, your suggestion that only people in poor places want egalitarian salaries, could be seen as disrespectful of other people, because it seems to ignores the totality of an individual while preferring to try to reduce them to simplistic motivations. In that way, it’s also considered abusive. And can also be seen as disrespectful of others experience, and maybe arrogant: "Only people in poor areas want such naive, unrealistic salaries."
Looking deeper, this aspect of your comment, combined with its narrow focus on a single explanation, might be interpreted as your attempt to express your personal frustrations at your own salary performance, or justify and rationalize why you may not be making more. This might occur because you may find it easier to view something you don't have as unrealistic and naive, rather than the result of choices you could change.
In short, while you mention that egalitarian salaries and enthusiastic support of them is naive and unrealistic, it could be argued that the view espoused in your comment is naive and unrealistic because: it lack awareness of complex dynamics; ignores the totality of an individual while preferring to try to reduce them to simplistic motivations, and dismisses real practices as unrealistic, which might also be seen as out of touch. Overall, your views unfortunately could be interpreted as narrow and an expression of personal frustration, instead of a reflection of underlying real dynamics.
To conclude, while it's likely there's some truth to the correlation you propose, it's also likely true that even if some correlation exists, location is not the only factor at play. People may have various salary preferences, independent of their location, just as the value they provide is also independent. Finally, indeed, your view could be expressed more respectfully of others.
Anywho, it's understandable you may have that perspective, given what your background might be. Yet it's always good to remember that you can adapt your view over time, can grow and can include more data to expand that awareness of reality which you value! :)
Except if everyone gets paid the same, you're not being rewarded for the value you provide. I don't think salary should be tied to location, but it should be tied to experience, ability, and effort.
You raise an important point. There's certainly multiple ways to look at it.
The egalitarian way where a business can divide the share of revenue that is allocated for salary equally, no matter the role. This makes sense from a philosophy of everybody being a team and contributing equally to the results of the company. It can foster an esprit de corps and and a sense of fairness. On the negative side it could encourage companies to have more burdensome measures of fairness and contribution, and lead to resentment towards colleagues who don't pull their weight.
Then there's the other method, where a value-based salary is allocated to each employee taking into account their experience, ability and effort. Crucially, however, this salary is not adjusted for location. That's the case to which I was speaking, specifically, even tho the type used by 0xide is clearly the egalitarian one.
> I don't think salary should be tied to location, but it should be tied to experience, ability, and effort.
That's the labor theory of value (see: Smith, Marx), which in theory sounds meritocratic but it can't really be measured or assessed.
In reality compensation either becomes a function of power, social currency and negotiation skills, which is the general norm in professions, or you have an institutionalized, perhaps even democratic process to determine salaries. Both of these variants generate overhead and are only approximations to what anyone would see as fair.
The variant here where everyone gets the same, generous piece of a pie seems refreshingly simple and honest. I would also assume that it attracts the right kind of people, who are intrinsically motivated (at least after the threshold of a very high level of comfort is reached.)
Saying that people should be paid according to experience, ability, and effort is absolutely not the labor theory of value.
The idea that contribution "can't really be measured" is a cop-out. Contribution can't be measured perfectly but it can be estimated with some accuracy by people who are involved in day-to-day work. "Some accuracy" is really all that's required: as long as contribution is correlated with compensation to some extent, you have a functioning meritocracy.
> The variant here where everyone gets the same, generous piece of a pie seems refreshingly simple and honest.
I bet it works great if you have a small team, are extremely picky about hiring, and quickly fire bad hires. Otherwise it will be awful.
> Contribution can't be measured perfectly but it can be estimated with some accuracy by people who are involved in day-to-day work
This is handwaving in the extreme. Anyone involved in software development knows that a single line of code can be critical to success or failure, as can a blob of 100k LOC, so product-quantity metrics are of almost no use. The "estimate" you're talking about generally comes down to general "feelings" about who works hard, which have repeatedly been shown to be poor metrics for actual contributions.
Life is full of handwaving. In almost any workplace, it's very simple to know who's doing the work (and it's usually a shockingly small number of people). It's the idea that we can reduce this to a mathematical formula (the opposite of handwaving) that's odd.
> The "estimate" you're talking about generally comes down to general "feelings" about who works hard, which have repeatedly been shown to be poor metrics for actual contributions.
How has this been "shown"? Anyway, you're begging the question that there's some way to determine "actual contributions" that we can compare to "feelings".
If you actually work with a group of people on a daily basis and can't rank order them in terms of usefulness, I find that astonishing. And remember, rankings don't have to be perfect, they just have to more accurate that random.
> And remember, rankings don't have to be perfect, they just have to more accurate that random.
No, they have to better than both random and "everyone is, on average, and over an extended period of time contributing roughly the same". That's quite a challenge.
Do you go to customers and ask them which features provide the most value to them, and then follow the code back to the people who implemented them? Do you go to the customers who paid the most, and repeat the question to them only?
We're not talking about some award-prize ceremony speech in which we acknowledge that Dmitri and Aneka led the work to get version 8.0 which has been a huge success. We're talking about actual salaries, which are presumably linked in some way to actual sales, and I'm insisting that connecting individual developer efforts to the sales numbers is extraordinarily hard. "More" and "less" are not enough to come up with actual numbers.
> No, they have to better than both random and "everyone is, on average, and over an extended period of time contributing roughly the same". That's quite a challenge.
This is something virtually all functional companies do when they decide raises. The fact that they don't do it perfectly isn't a huge problem, they just need to be more right than wrong.
> Do you go to customers and ask them which features provide the most value to them, and then follow the code back to the people who implemented them? Do you go to the customers who paid the most, and repeat the question to them only?
Does productivity equal sales? I don't think so. If someone does a good job implementing a feature that doesn't drive sales, that should count toward their productivity. Equally, imagine a task that could be assigned to anyone that drive sales: it doesn't make sense to reward the person who happened to be assigned this task when anyone could have done it.
You're demanding way too much here because you're unwilling to get "handwavy" and instead want some rigorous way to quantify productivity. Instead, embrace subjectivity! Imagine you're in charge and ask yourself questions like:
1. If I need to organize a meeting to address some problem that needs to be solved as soon as possible, who would I invite to the meeting?
2. If someone tells me he plans to quit, how much would I be willing to offer to convince him to stay?
3. If someone quits, how hard are they to replace? In terms of hiring a replacement and/or transferring their responsbilities to someone else.
I suppose there are some workplaces where it's genuinely hard to rank people. But my sense is that they're rare, small, and careful about hiring. Everywhere I've worked, this is not the case and I'm fairly sure this is the norm.
The 2nd and 3rd questions you pose are all about sales.
If your company brings in $10M a year, and somebody plans to quit (or has quit), how much will your sales drop (immediately, and over a period of time). That's the answer to how much you can afford to offer them to stay, and that's how much you should offer their replacement. Suppose that says drop by $1M and you can be satisfied that the drop is 100% a consequence of the departed (or soon to depart) developer - that's how much value they bring to the company, and by the logic of capitalism (which I don't play by, btw), you should pay them some amount less than that.
The problem is: you can't determine the value before they quit, and you can't be sure that their replacement will provide that value after they are hired.
Look, I understand that in an organization of any size, there are likely to be slackers that feel like a deadweight, and others who feel like the contribute far more than the average.
The question is: does tying salary to this perception actually bring the benefits you think it does (which are intimately connected to the notion of incentive) ? There's some good evidence that for developers and other "head-based" employees, it does not, and that flat pay scales create an environment in which you get different kinds of benefits.
I've worked exclusively in a distributed FLOSS project for the last quarter century, so in many respects, I'm not well positioned to talk about what happens inside traditional corporations.
The problem is that in the end, power, negotiation and social currency often dominate over merit (such as effort, experience etc.) when it comes to compensation.
Even if you actually do measure and agree on metrics, then the measurement can easily become the goal for those who are not intrinsically motivated. Work ethic can't be taught by dangling carrots in front of people, because acquiring the carrot becomes the goal instead of moving the cart. This can be detrimental in a highly collaborative workplace.
Having a flat, generous salary might solve this problem, because you filter out carrot hunters and get cart movers.
> I bet it works great if you have a small team, are extremely picky about hiring, and quickly fire bad hires. Otherwise it will be awful.
Finding the right people to work with is difficult regardless. The same worker can be miserable in one place and flourish in another.
That is not the labor theory of value. The whole point of LTV, at least in Marxian economics, is that workers can't be payed according to their socially necessary labor, because a surplus labor is extracted.
Besides, LTV as a theory is meant to be a description of the world as Smith, Marx, etc. see it, not a prescription for how things should be done.
You're right that it is descriptive and not normative.
But the underlying belief of paying someone according to their effort, is very much based on the same premise.
What I'm saying is that nobody is _really_ paid according to their effort, experience etc. because those things cannot be reasonably measured.
The typical process of determining compensation is based on negotiation and power. In some places the process is more democratized and rules based. Both of these are only to some degree related to actual effort, experience and so on. This discrepancy becomes larger the more people are involved as well.
Location independence is not the interesting part of Oxide’s compensation strategy. It’s that everyone makes exactly the same. There are no negotiations, no levels, no promotions, and consequently no promo packets or promo projects. This is extremely enticing to FAANG types who are tired of a certain kind of bullshit, at the cost of a certain level of ambition.
Yeah this I’ve seen everywhere, but more related to the company size. In a 50 employee company, it’s hard to pull that off. As organizations get larger, you get this as inevitable part of human nature. If the salary table is flat, then highly ambitious people will ask for more skin-in-the-game.
The skin-in-the-game bit can be promotions, equity, stock compensation, etc.
I wonder if they offer equity. Hierarchies always form, they tried this with Gore Associates in Arizona (totally flat company structure) and has major problems.
We've got designers, me included, and a few others on the team who aren't engineers. I also hail from what people like to call a "third-world country". So far it's working great, and we're hiring more folks from all over the world, not just the USA.
Salary is that high in the US because we have no social net or price caps here. Your on your own for everything. Healthcare, retirement, overpriced homes, out of control rent, etc
With this in mind, it's just a good salary. Until they have competition squeezing them on margins it looks like an OK approach. At some point their board most likely won't agree with paying some people below market rates for important roles and other more fungible roles paying 2-3x market rates but while they can keep doing it, it's great free marketing.
Overpriced homes, out of control rent, retirement are extremely problematic in a lot of European countries. And in the US, healthcare is usually covered by the company (talking about big tech corps).
So the extremely high salary is still a net positive compared to EU
Wouldn't an employee making that $201k salary in another country be responsible for paying all the taxes to support the socialized healthcare, etc? In other words the take home pay may be significantly less.
Other taxes (GST~VAT 15%, city rates, high petrol excise, etcetera) will easily take another $10k. Interest on your home is not deductible (NZ has very few deductables). People earning six figures will often pay for private health insurance and medical fees on top of the socialised healthcare - maybe another few thousand.
retirement is covered by social security
healthcare by insurance (offered by the state if you can't afford it yourself)
homes, etc is true though and we just need to build more inventory IMO. but there are tons of areas with affordable homes, they just aren't near the big cities like NYC or SF or LA
You can not live on Social Security, no way. If you don't have a job your not getting healthcare and even if your job provides healthcare. Its too expensive to actually use. Also you need to be living below poverty wages before a state will give you healthcare.
That's probably [EDIT: decidedly] on the low side for the bay area for rock star devs, and Oxide has lots of rock star devs. I haven't looked but I assume they pay bonuses, probably differential bonuses.
Hi Steve, bit of a tangent, I stumbled upon your blog from intro section and went down the rabbit hole. Wanted to ask if you have any recommendations on documentaries. TIA!
Like Steve, I also took a pay cut (from around 400-600k a year TC at a FAANG) to work at Oxide. I'm very passionate about what we do and was attracted to the values-oriented culture.
While that may be true, it's also unfortunately true that stock in startups has a high probability of ending up worthless. It's important to compare real money now, not just theoretical money in future.
I doubt my Aspen vacation home developer will accept payment in startup stock.
That is only a problem if you insist on hiring people from such expensive areas. Given Oxide hires remotely, I doubt this is an actual problem for them.
Hence the word "concentrated". Your snark seems misdirected.
I live in flyover country, and there are no shortage of talented people here, but I wouldn't dispute that the per-capita software developer talent in SF is going to be higher.
I think it's mostly societal perception. For whatever reasons, a lot of people seem to think of professionals in SF and NYC as smarter or more capable than professionals from Boise or Salt Lake City. Just like a lot of people assume those who got into Harvard or Stanford as 18yr olds are smarter or more capable than those with degrees from the large public universities.
I don't understand why you would compare total compensation at Oxide with base salary at other employers? Total compensation is the relevant figure across the board, and senior (as in, experienced, not some specific levels.fyi tier) engineers often make significantly more than $201,000.
Oxide equity, like for all private companies, isn't worth anything until and if they go public; it has no cash value. For total compensation purposes, it's $0.
> Oxide equity, like for all private companies, isn't worth anything until and if they go public
Open AI just had a tender offer for employees this month. It's not public. Clearly the equity is worth something even though the company is not public.
Some companies facilitate secondary transactions for their employees.
There are ways many employee friendly companies provide early liquidity without IPO.
user: sunshowers has been replying in this thread and mentioned going from FAANG-> Oxide and taking what seemed like about a 60% paycut.
Yes it is somewhat of a stumbling block, but I'm betting they are getting extremely high quality employees who are doing it for the passion and not just for the money.
I really wish oxide had a Homelab/consumer centric offering!
Spec wise, some low power systems like an Intel NUC, LattePanda Sigma, or Zimaboard. You could fit 3/4 of them in a single 1u with a shared power supply. They could even offer a full 1u with desktop grade chips on the same sleds.
I have thought about building one myself, but it's a large investment of time that I can't seem to find lately.
Even just a medium business offering would be great. I'd love to not have to use Dell or HP gear-- anything to get away from the cobbled-together stack of legacy IBM PC compatibility and third-party ODM/OEM stuff glue-and-taped together by the vendor.
On prem. Reliable and inexpensive network connectivity they has any resemblance to a 10G LAN doesn't exist where I am.
I work with some businesses who need very, very reliable, high-bandwidth, and low latency connectivity to their data. The amortized cost of on-prem beats the cost of any off-prem offering as soon as the cost of the necessary connectivity is factoted-in.
I'm talking shops who spend $200-$500K on servers and storage, not north of $1M (which is where this Oxide gear lives). Something like a 1/4 scale Oxide rack, perhaps.
I work at SoftIron, another startup in this space. Our HyperCloud product might be interesting for you. I'm not in sales, so I can't comment on the prices, but I'd guess we're much more competitive since you don't actually need to buy an entire rack of our gear at a time.
That said, where this product-space gets tough is actually scaling it down. It's pretty challenging to create something that is remotely stable/functional in a homelab (space/power/money) budget. Three servers and a switch would probably be the bare minimum. We (and I'm sure Oxide :) scale up like a dream.
This all has me wondering, if I just want to play with stuff in this space as an individual homelabber who earns a tech salary and wants a nicely designed rack-mounted alternative to a mess of unorganized NUCs and cables and whatnot, what are my best options?
If you're willing to spend money on rack-mounted gear you definitely have options, and what you get sort of depends on what you're interested in playing with.
A lot of homelabbers (and even some small businesses) go for Proxmox as a virtualization distribution. I don't use it myself, but IIUC it's effectively a Debian distro packaged to run KVM/LXC, with support for things like ZFS, Ceph, etc. It has some form of HA, an API used by standard open source devops tools, handles live migration, etc.
So buy some used rack-servers on Ebay (or new, if you're ballin'). A lot of businesses sell their old stuff, so you can pick up a generation or two out of date for a good price. If you want to do fancy stuff like K8s, Ceph, etc you'll probably want at least three nodes, ideally more, and a bunch of disks in them. Networking gear is a sort of pick your poison thing. A lot of people love Ubiquiti gear; a lot of people hate it. TP-Link is another that's good and budget friendly. StarTech sells smallish racks (including on Amazon), if you want to start there.
It won't look exactly like SoftIron's HyperCloud or Oxide's Cloud Computer, but you can certainly get pretty sophisticated.
Not sure if this answers your question, but other great spaces to explore are the 2.5 Admins and Self-Hosted podcasts.
I'm really thinking mostly about the hardware part here, and maybe just enough layers of the stack to feel like an integrated hardware setup. Let the nerds play with whatever software they want above that.
To go ahead and dream a bit:
I'd hope for an online configurator like the one SoftIron's HyperCloud has [1] but instead of "talk to a sales rep", show a price for what you just configured, like you're configuring a macbook.
Relatedly, there should be a standard rack form factor in the size category of NUCs and Mac Minis, rather than having to go all the way to the 19 inch monster racks that medium to large businesses use. If it were nailed down to the point of being able to blind mate (just learned that term from Oxide's article here!) gear into it, that would be kind of perfect.
AWS Outposts is the solution. I like Oxide but people seem to be blind to the actual competition when they focus on Dell as the competitor. AWS has been shipping Outposts racks for years. All prices are public on their website and you can order it today. Nearly every configuration is sub-$500k. Fully managed and AWS supports the entire stack; no buck-passing among vendors, same as Oxide.
I’m not sure where his customers are, but Outpost up/downlinks are supposed to to be at least 1gbit, and they don’t behave well in situations where the latency to the paired region is high. EBS lazy loading blocks is great in region but awful when your ping is 300ms.
I'd imagine they'll get to that eventually, these types of companies generally start at enterprise level because that's the most profitable and requires closing smaller numbers of deals. Once the product is proven and their support infrastructure is in place they can go for other market segments to try and maximize revenue
It's not just about maximizing revenue, it's also about getting it into developer hands early (homelabs, side projects, college students, etc) so they can become familiar with it, and become an advocate for it within their company. Cloudflare is a good example of this.
It would be great if Oxide had something like Canonical's "Orange Box"/cloud-in-a-box for homelabs, evaluation, training (in the management bits) - and hobby work loads!
Not 1U but perhaps a box design that isn’t noisy like a pizza box server.
Don’t know if oxide would want or be able to compete in the low cost market but a bigger a more expensive desktop/workstation as a mini homelab cloud could be a great option to get people trained on the oxide platform.
Unfortunately they are not planning home lab things anytime soon, per a recent podcast episode [0].
If you want to play around with their Hubris OS: "You wanna buy an STM32H753 eval board. You can download Hubris, and then you’ve got – you’ve got an Oxide computer. You have it for 20 bucks.”
I'm a bit confused about them calling it a cloud computer. The primary benefit of cloud is you spend opex instead of capex. Isn't this moving back to capex again?
Ya, in the past the messaging was more like "get the ergonomics of the cloud with the economics of owning your own hardware", I think it's confusing to just call it a "cloud computer"
Yes - that makes loads more sense to me. It's cool that you can spin stuff up and it's all integrated with the hardware - having seen fairly large VMWare installations and all the faff it takes to make them work, this seems really useful. I just don't think it's the same as the cloud, and the article seems to discount the main reason for the cloud: opex, letting you get started (and stopped) quickly and cheaply.
That was the line we were sold by cloud vendors. There is a reason that significant portions of both native cloud and migrated workloads are shifting back to self-hosted; the cost savings never materialized.
Of course - every pay as you go service that goes above a certain amount of utilization will be better replaced with up front investment. But that doesn't mean that the opex model is bad in general; you've just only picked cases where it's bad.
So, what you're saying is, that whether opex or capex models are better or not...depends? That there is no silver bullet, golden gun, one-size-fits-all platitude anyone can post in an HN comment?
If the opex model doesn't work for an organization, that they perhaps just went above that utilization threshold where it doesn't make sense anymore, and would be a potential 0xide customer?
There's two independent issues: capex vs. opex, and flexibility.
When opex is a primary benefit of the cloud, that's specifically for start-ups and other businesses that have little working capital. The actual primary benefit of the cloud for most cloud customers is flexibility - prior to AWS adoption, getting a server provisioned could be a multi-week if not multi-month affair negotiating between devs or operators and on-prem infrastructure workers to get the servers through capacity planning and provisioning. AWS made provisioning so simple, that you could start to set up auto-scaling, because you had extraordinarily high confidence that the capacity would be immediately available when your autoscaler tried to scale up.
But opex is not a benefit of the cloud for heavy/established businesses. Cloud opex is a financial expenditure every month that you can't get rid of and counts directly against your profits. Indeed, the desire for capex in the cloud is so high that businesses routinely purchase Reserved Instances and other forms of committed usage, which allows accountants to treat the cost of the RI as capex and then discount the expenditure through depreciation (to zero, since there will be nothing to sell when the RI expires) over the lifetime of the RI. It is normal and frequent for businesses to make capital expenditures to reduce their operational expenditures over time, thus increasing their monthly/quarterly profits.
Oxide's unique value proposition is to give customers, particularly those with high monthly cloud bills that they have difficulty reducing, the operational flexibility of cloud computing with the profit-improving benefits of capital expenditures.
The way you put it, this capex thing though is sounding like just sacrificing cashflow for some accounting sophistry.
Surely the main benefit of reserved instances is lower TCO and if you can show you can afford AWS for 3 years a bank would surely loan you the money to pay upfront if you can save say 50% it is simply cheaper even with interest.
Again this goes back to flexibility. RIs necessarily take away from your flexibility. AWS and others try to grant you the flexibility anyway, by allowing you to shift RI credits between physical instances, but lower TCO is definitely not guaranteed. If you buy an RI for a server type you're not actually using, you're spending money on servers that's getting wasted compared to not actually buying the RI.
Private Cloud allows infrastructure team to run big server parks efficiently, while product teams can "buy" resources easily. It's essentially why Amazon made aws, why Google made Borg.
For a regional hospital for example, there might be a desire for in-house network and hardware - but perhaps the system for digital patient journals run on Kubernetes and managed databases.
Reminds of what they worked on before at Sun ZFS storage appliance.
Can it run Linux or Windows Data Center as ex-Sun folk they seem to have loved the last open source release before Oracle but now is still community developed variant of SunOS ???.
The hypervisor OS is based on Illumos, which was forked from OpenSolaris, and it uses Bhyve from FreeBSD for virtualization.
I would imagine the system architecture is different enough that running Linux as the base OS would take some work, for example it doesn't have an AMIBIOS.
Sort of. The big deal with Oxide is that all the legacy compatibility with the IBM PC platform is gone, and the whole stack, top-to-bottom, is built by then (including firmware).
It's not like commodity x86 gear with black-box (often buggy) firmware and layer-upon-layer of hacks and compatibility kludges to present the hardware interface of a late model IBM PC AT.
The CPU is commodity, nothing else is. Costume Mainboard and firmware without BIOS and their own BMCish thing and their own Root of Trust. Same for their router. Standard chip, everything else is costume.
Similar in some ways different in others. But in terms of not being a PC architecture. Yes it is. But in many other ways its not at all like a Mainframe.
It's similar to hyperscale infrastructure — it doesn't matter as long as it looks like a PC architecture from the OS running inside a VM. The layers and layers of legacy abstraction firmware, BMC, drivers BIOS, hypervisors you get with a typical on premise Dell/HP/SuperMicro/... server motherboard are responsible for a cold start lasting 20 minutes, random failures, weird packet loss, SMART telemetry malfunctions, etc.
This is the type of "PC architecture" cruft many customers have been yearning to ditch for years.
I’m not in the bare metal/data center business anymore at the moment, but I was for more or less the last 25 years. I never had such issues. Maybe I was just lucky?
Maybe you were. :-) And maybe this is not for you or me (I haven't contacted their sales, yet) it's not for everyone.
Personally, I have always been annoyed that the BIOS is clunky and every change requires a reboot, taking several minutes. As computers got faster over the years, this has gotten worse, not better. At the core of cloud economics is elasticity: don't pay for a service that you don't use. Wouldn't it be great to power down an idle server, knowing that it can be switched on seconds before you actually need it?
> Wouldn't it be great to power down an idle server, knowing that it can be switched on seconds before you actually need it?
considering you would still need to boot the VMs then, once the Oxide system is up, I’m not sure if this is such a big win.
And at a certain scale you’d probably have something like multiple systems and VMware vMotion or alternatives anyway. So if the ESXi host (for example) takes a while to boot, I wouldn’t care too much.
And, economics of elasticity - you’d still have to buy the Oxide server, even if it’s idle.
> considering you would still need to boot the VMs then, once the Oxide system is up, I’m not sure if this is such a big win.
To be honoust, I'm using containers most of the time these days but even the full blown windows VMs I'm orchestrating boot in less than 20s, assuming the hypervisor is operational. I think that's about on par with public cloud, no?
> [...] vMotion [...] ESXi.
Is VMware still a thing? Started with virsh, kvm/qemu a decade ago and never looked back.
> And, economics of elasticity - you’d still have to buy the Oxide server, even if it’s idle.
That's a big part of the equation indeed. This is where hyperscalers have an advantage that Oxide at some point in the future might enjoy as well. Interesting to see how much of that they will be willing to share with their customers...
Re VMware, it’s certainly still a thing in enterprise environments. Can kvm do things like live migration in the meantime? For me it’s the other way round, haven’t looked into that for a while ;)
How do you mean Oxide might have that advantage as well in the future? As I understand, you have to buy hardware from them?
Ah yes live migration, off course. We design "ephemeral" applications that scale horizontally and use load balancer to migrate. With 99% of traffic serviced from CDN cache updates and migrations have a very different set of challenges.
As to your question, I meant to say that as volumes and scales economies increase they can source materials far cheaper than regular shops. Possibly similar to AWS, gcs, Azure, akamai etc. It would be nice if they were able and willing to translate some of those scales economies into prices commensurate with comparable public cloud instances.
If you want more insight into all of the things that normally run on "PC architecture" - the 2.5 other kernels/operating systems running underneath the one you think you're running - https://www.youtube.com/watch?v=mUTx61t443A
I imagine they are aware that this isn't a solution for many customers. A John Deere tractor makes a poor minivan. This isn't for you. That's fine. It's not for me either. That's ok. I don't need to poo-poo their efforts and sit and moan about how it's not for me.
Every PC has millions of lines of firmware code that often fails and causes problems. Case and point, pretty much all hyperscalers rip out all the traditional vender firmware and replace it with their own, often partially open source.
The BMC is often a huge problem, its a shifty architecture and extremely unsafe. Meta is paying for u-bmc development to have something a bit better.
Doing things like rack attestation of a whole racks worth of firmware if stacked PC is incredibly hard and so many companies simply don't do it. And doing it with the switch as well is even harder.
Sometimes the firmware runs during operations and takes over your computer causing strange bugs, see SMM. If there is a bug anywhere in that stack, there are 10 layers of closed source vendors that don't care about you.
Costumers don't care if its PC or not, but they do care if the machine is stable, the firmware is bug free and the computer is safe and not infected by viruses. Not being a PC enables that.
The difference is that when you're buying a x86 system, the entire CPU bringup (incl. AMEGAS/openSIL on AMD) runs proprietary and poorly documented firmware. You're entirely at the mercy of the vendor.
Oxide has put immense effort into writing open-source platform initialization code, and built their own open-source BMC/RoT solution.
So effectively I’m at the mercy of Oxide, at least as long as their system does not become some kind of standard.
Not in theory maybe, but in practice. Because as a customer, I would probably also need to put in immense effort to understand and maintain that software myself.
Their firmware is open source. You can pay whoever you want to maintain it. You can't do that with Dell, HP, Supermicro, and the unknowable rabbit hole of ODMs and sub-suppliers and contractors who actually make the hardware and firmware for these companies.
Until you've dealt with a malfunctioning Dell or HP server and have to live with being told "we don't know why it acts that way, we'll try to get the ODM to repro" I don't think you can appreciate how cool Oxide's offering seems.
If I have server under maintenance with Dell or HP, they would replace the server or component for me in such a case.
Which would probably be a lot faster than trying to find someone who could maintain some non-standard firmware (as good as it might be).
Even if I had to replace the server on my own cost it would probably still be cheaper. And it would be easy to replace because it's commodity hardware, that was kind of my point.
I have had experiences with tens of Dell servers with the same model NIC having the same fault. The servers were absolutely under maintenance. I fought with tech support for weeks before I was finally told it was a driver/firmware issue and that I had to work around it (and lose performance for the sake of reliability).
Maybe if I had hundreds of servers Dell would have helped me out. At the scale of tens they told me to take what I got. The Customer got a lower performance solution and nobody anywhere could help them for any amount of money, short of replacing the gear.
That's just a performance issue. I've heard horror stories about reliability-- All the way down to disk firmware and RAID controllers. I consider myself lucky.
But how much effort (or money) do you think it would have taken to fix this issue if the NIC firmware was open source?
And with standard hardware, depending on the model, you might have had the option to add dedicated PCIe NICs for example. Not great, but at least something. Now try that with something proprietary (as in non-standard) like this Oxide system.
Replacing hardware? Sure, they'll help. What about debugging firmware though? I'm curious how much help you would get from Dell fixing and patching complicated firmware errors. A side benefit of the openness is that firmware issues can be discussed publicly, and the patches can be upstreamed into the main repo and made available to every customer (and even competitor). This gives you the kind of network effects that you'd never see in a locked-down ecosystem.
The CPUs are x64 but the architecture is not that of a PC, there is no BIOS, etc. You couldn't boot Windows or Linux on the bare metal. The hardware, firmware and hypervisor are custom built for control, safety and observability. On top of that, the application OS all run on VMs which _do_ have a (virtual) PC architecture.
To me as a casual outside observer, the fact that they're using hardware virtualization at the top of the stack, after bcantrill gave so many talks about running containers on bare metal, is the most disappointing part. They could have had unbroken control and observability from the bottom of the stack all the way to the top. They got so close!
It's possible that the hypervisor can reserve you a full CPU or full cores for the guest OS to work with, so you still get most of that bare metal goodness.
But there's no pricing! How will I know if this could potentially save me money compared to AWS if I cannot make a sensible comparison until I contact sales (or probably after I contact sales).
Why though? Quantity based pricing is not hard to express in a table.
What is so different about CapEx and a bag of peanuts that defies even providing an approximate range? Number of zeroes, no?
I don’t like it, but it is the norm sadly. It’s just a signal for “it’s expensive and if you even care about the number we don’t want to talk with you”.
In fairness, the blogpost argues that owning is more economical than renting. Everybody knows what cloud compute costs and if this more economical then maybe there should be some price indicator?
If you are genuinely in the market for multiple racks of servers you (a) know how much a rack of hp/dell gear costs, which gets you within an order of magnitude of what this is going to cost, and (b) would not buy one of these without a sales call even if you could.
I wouldn’t buy these without a sales call, but costing it out to less than an order of magnitude would make it a lot easier to determine economic viability.
I just hate calling people to determine if we’re going to match, when they could have given me that information up front.
Let's start with their beliefs:
1. Cloud computing is the future of all computing infrastructure.
Yeah, no. There's still gonna be things like mobile phones which will do some stuff locally. And probably also desktops. And probably also on-prem servers.
Which brings us to their belief no. 2:
The computer that runs the cloud should be able to be purchased and not merely rented.
If you buy a computer and run virtualized loads on it, then you're not doing cloud computing. As simple as that.
Basically, as others have commented already, this is just something like a blade server system or a hyperconverged system like for example Nutanix is offering. The only real difference seems to be the open source approach.
The name is "Oxide Computer Company." But since "0x" is often used for hexidecimal numbers, and "0x1de" happens to be one, we play around with it from time to time.
> The computer that runs the cloud should be able to be purchased and not merely rented.
This 100%!
Not everything needs to be a subscription. Sure there are running costs for operating that cloud computer as well as ongoing software development costs. But having photo albums in the cloud shouldn't cost me $30/month just because of storage - let me buy that hard drive (and maybe compute why not) and pay $10/y for operational costs
$10 per year, minus credit card fees and taxes. If you send a single email to customer support in 5 years the company already loses money. The company can't just sell you the HDD and put a warning in the FAQ that you are responsible for data loss. Because that will result in angry emails and bad reviews.
How many engineers and how many years does it take to build the infrastructure, write all the software, and deal with all the hardware to provide a photo service that "just works"? How many customers do they need before they break even? And obviously they can't raise VC because a business model that is predicated on making basically no money per customer can never have an exit. How would you bootstrap such a thing?
It feels strange that physical storage costs almost nothing on amazon but the storage attached to a cloud machine is expensive. But run the numbers and you'll see that running a cloud service isn't about hardware costs at all.
If you don’t listen to their “On the Metal” podcast, you’re missing out. So many great stories from legends in the industry. Just start with the first episode.
And they currently run a podcast weekly called "Oxide and Friends" where they talk about miscellaneous things in the software space.
The most recent few episodes have been about corporate open source and they've had excellent guests, like Kelsey Hightower. Definitely the best computer related podcasts out there. Bryan and Adam are great hosts and their humor is always a delight.
Other podcasts I'd recommend: ADSP [1] (if you're into programming), 2.5 admins [2] (if you're into computers). But I have no recommendations about hardware design because AFAIK the podcasts you mention and what Oxide is doing are pretty unique.
The Amp Hour is really the only popular hardware/electronics design podcast as far as I can tell, and it's pretty decent, especially when they have interesting guests.
There’s plenty of questions about ZFS in the show that get answered, that’s always a given.
I’d say their opinion on “ransomware being a backup problem not a security problem” was enlightening and their backup opinions are also interesting (Push vs. Pull, ZFSes snapshotting, etc.)
Allan’s experience in FreeBSD is welcome too, not every day you get to hear about FreeBSD in prod.
They both skew FOSS too but not in an idealistic way, it’s based on decades of experience.
Jim Salter is an insufferable has been. He's a SJW-type that threatened to delete his reddit account and closed off the ZFS subreddit but came crying back like a childish addict after the reddit admins threatened to take his fiefdom away.
In a similar vein, Embedded.fm is a great podcast for embedded SW. Though I bet Oxide's take on embedded rust is a lot different than the hosts of that show!
I know price will vary wildly based on how many you’re buying, but does anyone have the roughest ballpark for how much it would cost to buy one (1), or like two?
I mean, the raw hardware surely costs around 100k, and no way that it costs more than ten million, so you're always going to be right with that "ballpark" qualification
They mentioned it really quickly in their Oxide an Friends podcast but, IIRC, prices start at $500k. Some of the audience asked if they were going to do a smaller configuration like half or quarter rack. And they said they were looking into it but weren't sure the of the business case.
the performance may be about the same, and CapEx as well (or maybe a little higher), but OpEx could be where you make it up in large(r)-scale operations.
And space efficiency is also not to be sneezed at: for some operations DCs/compute can be place anywhere because latency is that big of a deal, but in other places you need to be close to certain things (trading), and real estate can get expensive.
It depends on how many servers you put in a rack. It's been years now I did this kind of work but I would say that an average rack with 20-25U in computing, 5-10 in storage and 5 in networking will cost you 300k$ easily. I'm pretty sure that Oxide will be more an Apple-esque experience, also on the price side, so a "normal" rack giving the same performances will be cheaper but if you want Oxide you are looking for other features beside the pure HW.
It's very expensive hardware. I think they are trying to bring TCO down on the operation side with better control plane. I work in hyperconverged systems and it's a bunch of tradeoffs. Nothing I've configured has approached $500k so their control plane and OS has to make a really good show of why this 2x expensive cabinet is better than rack and stack Dell.
Let's just say I hope I am interpreting something incorrectly, because if 500k is the minimum price and you match it to the minimum configuration found here:
That said compared to competitors it's in the right ballpark, but I have no idea how companies manage to spend so much money for this stuff. I am the founder of my own tech startup and I remember when I was looking at storage solutions and building out computing clusters there were companies charging absolutely insane prices.
I literally just spent about a week of my own time studying and learning as much as I could about it on my own and ended up building out my own custom solution for about 20-25% of the price these other companies were charging. I remember hearing people trying to scare me out of it saying if I did my own solution I'd need to hire full time operations people, and I'd always have to worry about things breaking and maintenance or headaches and nightmares etc...
It's been over 10 years now and absolutely no headaches, no nightmares, and very very minimal maintenance is needed.
Yeah sure, my process to learn was literally I went down to a local computer store, I bought a cheap desktop computer and 6 cheap hard drives, an HBA, a RAID controller, a bunch of cables, went back to the office and installed Ubuntu Server onto it and practiced several "skills", like how to setup automated backups using rsync, how to physically install these components, how to install mdadm for software RAID etc... comparing software RAID to using a RAID controller. I setup several drills for each task involving various types of failures and how to manage them to the point that it was part of muscle memory.
Some drills I practiced were setting up RAID 10 and on hardware failure having an email sent to me, so I went through the process of getting RAID 10 working using 4 of the drives, and then I would physically pull a hard drive out of the system as it was running to simulate a failure, and then I swapped in a new hard drive and ensured that the RAID rebuild process took place.
Once I was confident enough, I went to thinkmate.com and bought two JBOD expansion chassis each of which supported 78 drives and filled each of them up with 5 TB Seagate drives to get a total of 390 TB. The JBODs are managed by a 1U server running Ubuntu Server and software RAID using mdadm. I also bought 8 compute servers that were considered high density and could fit in 4U. I got a Cisco router for Internet and I networked everything within the data center together using a used Infiniband switch that I bought off of ebay. I also got a KVM so I could remotely access all of the systems. If there's one thing that I would change today, it would be to use ZFS.
I remember comparing the cost of doing this DIY setup to some other premade solutions like EMC and the cost was astronomical, like 3-4x what I ended up having to pay. I also even remember watching Linus Tech Tips and Level1Techs on Youtube and they both had good content about how they managed their storage that was fairly reasonable but still slightly on the pricey side, nevertheless I learned quite a bit from it.
At any rate, the bottom line is that it's not trivial to learn all this stuff by any means and I remember having some serious frustrations due to just how bad and demoralizing some error messages can be, but it's not thaaaat difficult either and in my situation my company is self funded, no venture capital or outside investors of any kind so every dollar my company spends is a dollar out of my own pocket. You better believe when I'm starting my business I'm not going to just blow 100s of thousands of dollars extra unless I absolutely have to.
About 5 years ago I managed to use my own storage system to ditch Dropbox in favor of Nextcloud, which is just leaps and bounds superior. I remember I got so frustrated with Dropbox because I wanted to just do something as simple as create a sub-directory and grant only read only permissions to some accounts. I also remember wanting to do some simple things like create a fake account with very limited permissions that could be used in our conference room for presentations or demos but the only way to do that with Dropbox would be to create an actual account and have to pay the full price for it.
Nextcloud works amazingly well, has all kinds of cool plugins, and gives our organization a lot of flexibility that Dropbox doesn't and I can make as many accounts as I feel like for whatever reason I feel like.
That's a lot more involved. We're a quant firm that have servers colocated globally at various exchanges, so our data is distributed across a lot of systems. The main and original purpose of this storage system is to provide one centralized repository of all of this global data so that we can run backtests and perform analytics locally as opposed to having to hit our Australian system when we need ASX data, and then hit our German system when we need Xetra data, etc etc...
So different backups happen in varying directions, with the colocated servers using the main storage system to back themselves up, and the main storage system in turn backing portions of itself onto the colocated servers.
Not everything gets backed up, however, and most of what does get backed up can be compressed quite significantly. We mostly just back things up that can't be recovered from third parties or that are proprietary to our organization.
> They mentioned it really quickly in their Oxide an Friends podcast but, IIRC, prices start at $500k. Some of the audience asked if they were going to do a smaller configuration like half or quarter rack. And they said they were looking into it but weren't sure the of the business case.
That strikes me as being in the right ballpark, but it's going to be tough to swallow since that's the lowest level of granularity.
For most orgs you'd be left paying for a lot of excess capacity you couldn't immediately put to use as you migrate workloads in. I guess in ~4 years once you reach steady state and you're retiring / replacing these things it all works out, but if you're migrating from vmware or something else in a traditional blade/chassis world it's not like you can just wave a magic wand and move $500k worth of compute over to this thing at once.
If you're green fielding something, that's a lot of cash to sink in on compute you may not need for some time in the future. Never mind your DR site(s) also needing that much...
I'd like to add that their open source code is also EXTREMELY high quality. If you're an embedded developer go take a look at Hubris and Humility. I ended up using those to GREAT effect for this custom one-off aerospace device and it was a fantastic experience to integrate with. Definitely a change from what I was used to that took a bit of getting used to.
Their ring logger was really enlightening to me about the value of Rust enums. Heterogeneous log events are dropped into the ring with some holding only the fact that something happened, and others holding additional data about what happened. Then Humility is able to print out the contents of the ring either online or in crash dumps. This is how you get logging in nostdlib Rust without ending up without half of a badly implemented printf. Instead, Humility, which has the full stdlib available, formats the enums for the firmware.
I've been interested in Cogmind for a while, unfortunately it only works on Windows (which seems an odd choice for a text-based game) so I have no ability to play it.
I design them in Monodraw, pass it through a janky converter I wrote that converts text into a json grid of characters. I then render a number of layers that get combined, which is a mix of the static art layer, and others generated from functions that spit out a similar cell based frame.
1. The state or quality of being humble; freedom from pride and
arrogance; lowliness of mind; a modest estimate of one's own worth; a
sense of one's own unworthiness through imperfection and sinfulness;
self-abasement; humbleness.
-- Webster's 1918 Dictionary
Our avionics software was originally spec-ed to not NEED a debugger it was to be of such high quality, but the designer added break point op-codes in the VM anyway. I guess he was aware of the concept of humility, too. ;) Like humor, humility seems to be in short supply these days.
I think 'throw-DO-178C' was being less literal than you are assuming here. Note that they quoted the comment about extremely high quality. What gives Hubris/Humility that level of quality?
When I see the concept of "extremely high quality" applied to software in the aerospace domain, I tend to think of methods like formal proofs and "semi-formal" methodologies like DO-178C (including MC/DC, not as a tool, but as a critical sub-process). It does appear that Humility adheres to some traditional ideas used in even "safety conscious", real-time embedded work:
> However, Hubris may be more interesting for what it doesn't have. There are no operations for creating or destroying tasks at runtime, no dynamic resource allocation, no driver code running in privileged mode, and no C code in the system. This removes, by construction, a lot of the attack surface normally present in similar systems. [1]
Oxide surely didn't invent all these concepts, we were applying some of this in the early '90s on human-safety-critical embedded projects, and they were most likely used before that. The attack surface eliminated is more than just security, e.g. even self-induced "attacks" from task priority inversion. Maybe the concept of applying them to rack mount servers is novel, but I have no experience there. Techniques like DO-178C go farther and do things like trace requirements to object code and strict coverage criteria.
When you are considering where Hubris/Humility might be applied in aerospace, also note that it would most likely be denied at the proposal stage of talking to a regulatory body like the FAA if you said, "The methodology for our brake controller is cloning a Github repo written by 100X S/W engineers in the Rust language, then modding that to fit." It's a process you have to follow in a documented fashion, from the start. Who knows--I certainly don't know everything--if you do have a identify reproducible process that improves S/W quality, the FAA might be interested.
And as far as eliminating debuggers goes, Boeing did actually do a joint academic/industry project on a zero-bug reduced (subset) Ada compiler (Zbra), which itself would be qualified as a tool instead of the more laborious process of mapping requirements to object code directly. [2]
In the spirit of humor shown on this topic, I would suggest considering a reduced subset of Rust called Iron. :) Regards, and best of luck!
Tangent. In my admittedly limited experience, embedded code seems to have a tendency to be some of the worst code you can come across. The stuff is already low level and not the most easy to follow, but then embedded developers seem to despise names with more than two letters and a number. Without the datasheet at hand it is impossible to figure out what any of the code does because everything is just an abbreviation or acronym from some block or pin-out diagram.
Yeah, I just skimmed through the reference docs for Hubris and was very impressed with what I saw. No "rocket science", just (apparently) solid technical decisions that are extremely well justified and documented.
Some interesting rare choices there (some of which would be applicable to normal userlands or even to language VMs):
* Memory access is protected, but addresses are not virtually mapped. The lack of paging of course ties strongly to the inability to dynamically add tasks.
* They wish they could have read-only shared libraries (viable since there's only one address space) so they could truly be shared, but the current ecosystem assumes mutability is possible (even though in Rust mutable globals are rare).
With respect to Hubris, the build badge was, in turns out, pointing to a stale workflow. (That is, the build was succeeding, but the build badge was busted.) This comment has been immortalized in the fix.[0]
With respect to Humility, I am going to resist the temptation of pointing out why one of those directories has a different nomenclature with respect to its delimiter -- and just leave it at this: if you really want to find some filthy code in Humility, you can do much, much better than that!
I met a couple of their folks on site last week in Raleigh at the All Things Open conference. Saw a couple quick demos, and... it's not for me. But I can see the benefits, and many other folks stopping by the booth seemed to get the benefits as well. The folks at the booth were really nice too. Granted, that's sort of your job when being in a vendor booth at a conference, but it's surprising how often that's not the case (booths staffed by people who don't know the product, or are simply indifferent to the company, etc).
EDIT: "it's not for me" - I'm not working with organizations that have that sort of need directly. Re-reading that phrase, it came across as a bit dismissive of what they've built (which is undoubtedly impressive).
> The computer that runs the cloud should be able to be purchased and not merely rented.
How is this different than colo space that has been predominant for years? Perhaps simply that it can be purchased more easily and standardized like a typical cloud VM?
That it's a turnkey, vertically integrated and open platform with everything included. Seems big enough differentiator for me. It's like going to going to a restaurant with table service vs a BYOB pizza shop.
I believe-as a few other commenters have pointed out and you allude to, the advantage is that it can be configured and operated like a massive set of cloud VM’s. There’s monitoring, provisioning, network, etc etc all setup and fully integrated.
I imagine colo’s have something like this, but I _suspect_ operationally it’s a lot more powerful, easy to use and functionally closer to the API’s and behaviour users are used to on current cloud providers.
Now, if someone who actually knows for sure feels like chiming in, I’m happy to be corrected.
It is the combination of things. Full FOSS stack, only cables being power and networking, and it being AIO package instead of conventional server rack.
993 comments
[ 3.6 ms ] story [ 201 ms ] threadAlso a former employer who was a bit below the hyperscaler level also had what they called "roll-in racks", though I believe they were just taking standard servers and networking gear and integrating them somewhere (presumably with cheap labor) and then bringing them into the DC as needed.
So I don't think it's a particularly new concept, but I agree it looks like it has some potential as a product.
Also, I remember for example HP BladeCenter BL20p systems. There you would have up to 8 or 16 servers per 6 rack units IIRC. So about 56 to 112 servers in 42U.
https://en.wikipedia.org/wiki/HPE_BladeSystem
Just my perception given the front page. I agree it's a bit vague.
None of that applies here.
Unless you rent it out and are a cloud provider, but the website at least does not seem to target those.
Also you could don’t really get elasticity with a system like this. If anything, that would be the operative bit for me.
Hybrid clouds even means devs might not know whether it sits in the corporate data centre or a public cloud, because it could be either/or depending on current capacity.
> Also you could don’t really get elasticity with a system like this. If anything, that would be the operative bit for me.
"You" as in "the organisation as a whole" don't get elasticity. "You" as in "your department" or "you as an individual" do get elasticity.
Right, but to the degree that you get elasticity, it starts to look more and more like "someone else's computer", no? If multiple people/departments/etc are provisioning virtual instances on one shared cloud infra, with nobody who's using the provisioning API caring about the underlying capacity (and capacity is planned indirectly by forecasting, etc), then it really starts to sound like "someone else's computer" to me. That "someone else" just happens to be another org within the same company.
I've spent the last decade building on-premises systems very like what Oxide is doing, but I've had to build them out of stacks of servers, switches, storage appliances and VMWare licenses. And the network cabling, and fan noise, and the number of power cables, and.. oh man, I can't wait to install one of these things myself. Having a single point of responsibility for the whole thing shouldn't be underestimated either - I've spent far too long trying to resolve problems with vendors on both sides blaming each other.
It's worth mentioning too that building something equivalent to this would be across more than one rack, and easily cost in excess of $1M.
That might be true historically, because the only way you could get resources provisioned on-demand via an API from someone else who'd built it. You had to run in someone else's datacenter to get the capability which you actually wanted.
Times have changed. Now, businesses think about "Cloud compute" as being synonymous with "on-demand", "elastic" etc. Where the actual silicon lives is merely an after-thought.
> Also you could don’t really get elasticity with a system like this. If anything, that would be the operative bit for me.
Buy enough of them and you will :)
If you have to buy the silicon and plan capacity for it (as in the case with Oxide for example), then it cannot be an afterthought. Which is exactly why I would not consider it cloud computing.
Someone's always got to buy the tin and manage that. Some people are big enough that they might get a benefit from doing that themselves, rather than having Jeff Bezos do it for them.
From the application team's perspective, call API then container go whirr.
Of course you do, right up until the point where you’ve used all available capacity. Just like with public clouds (ask anyone using meaningful amounts of {G,T}PUs). Elasticity doesn’t imply infinitely elastic, that would be ridiculous.
But that is literally not possible with hardware you purchase yourself.
Sure, you can buy X amount of hardware, and provision up to X amount of virtual hardware via an API, but then what? You can't provision any more until you go and buy more hardware. This is why "cloud, but local" is a contradiction IMO. You can only be "cloud-like" if you're under-provisioning. The moment you want to actually use all of the capacity you already paid for, you're not a cloud any more, because you've provisioned all of it.
No elasticity, no cloud.
Sure it is. I think TFA is talking about a company selling you exactly that capability.
> You can't provision any more until you go and buy more hardware.
But this is also true of AWS etc. When their estate gets full, they need to go buy more hardware. Regardless of who owns the tin, someone's doing a capacity plan and buying hardware to meet demand.
The point of 'cloud' is that you move that function out of the domain who are actually using resources to solve business problems, which is where it traditionally sat. Historically, if you wanted to run a service, you had to go buy some hardware and hire someone to manage it for you.
A cloud-like model means that the application engineers no longer care about servers, disks and switches. Instead, they just use some APIs to request some resources and then deploy a workload onto them. The details of what hardware, where and how is fuzzy and abstracted. Or cloud-like.
> You can only be "cloud-like" if you're under-provisioning
Everyone under-provisions. Nobody runs at 100% utilisation.
The definition of cloud is “out there in the sky - not here”
https://www.merriam-webster.com/dictionary/cloud
and that isn't it...
You may not like it, but it's been a long time since "cloud" has exclusively referred to "someone in another organisation entirely runs the computers" as opposed to virtualised allocation of resources.
I suspect that while I do appreciate how some posters upthread find the website a tad on the vague side, the target customers-in-potentia will understand it fine.
Define "I"?
I-as-developer can call an VMware/OpenStack API with an on-prem/private cloud and get a new instance just as easily as calling an AWS API. I-as-developer does not have to worry about elasticity if the IT hardware folks have the capacity.
From a developer's perspective an API call is an API call, and to them it's just another instance.
As an engineer, an Oxide system works like any other cloud provider. You’re just interacting with its API and tooling like you would with Google Cloud or AWS.
To someone on the IT/Operations side, obviously there are differences but theres SIGNIFICANTLY less labor required to build-out and operate an Oxide system vs a rack full of servers. The biggest difference for these people is that there’s actual hardware vs a Cloud Provider, but also costs are fixed so there’s likely no monthly or quarterly meetings with finance arguing over the cloud bill, tying people up to try and shave a few thousand off the bill every month.
In finance/accounting, Oxide is probably the most different: now compute is CapEx rather than OpEx. Depending on your company’s stage that can be a wonderful thing for the bean counters.
But it’s also gonna be much more restricted. So I guess one could see as kind of “Apple for data centers”? Have a nice appliance and be happy as long as it runs as it should (but hope it never stops working as it should).
What they're selling is building blocks for private/hybrid/public clusters that may or may not fit your definition of cloud depending on where they happen to be located but where what the term signifies is that it is built to be a building block of a cloud setup that includes features above and beyond a "regular" server to provide what most people tend to associate with a cloud. That is, you're getting APIs to spawn virtualized compute and storage, rather than having to install a hypervisor and management APIs etc. and combine the resources into a cohesive whole yourself.
My guess is that most of them will end up being sold to either hosting providers to provide cloud services to their customers or companies large enough to operate their own data centres where the IT department will use them to offer cloud services to other parts of the business, where the term really is not misleading anyone.
It's too big even for most "private clouds" in smaller companies, because they tend to be too small to be able to order by the rack even when there are APIs etc. offered to developers to provision compute and storage (e.g. years ago I used to operate a hybrid cloud setup with a ~1000 or so VM instances across several countries, and while we had several racks worth of gear we physically owned in aggregate, we didn't have any full racks in any individual location).
The Oxide machines seem to be aimed at 2020.
That being said I'd still expect a monthly service fee for networking, electricity and service in general.
> “Oxide is a strong believer in the need for open-source software at the lowest layers of the stack -- including silicon initialization and platform enablement. With the availability of AMD openSIL, AMD is showing that they share this vision. We believe that the ultimate beneficiaries of open-source silicon initialization -- as it has been for open-source revolutions elsewhere in the stack -- will be customers and end-users, and we applaud AMD for taking this important and inspiring step!”
* https://community.amd.com/t5/business/empowering-the-industr...
See Bryan Cantrill's (Oxide CTO) presentation on their adventures in this space:
* https://www.osfc.io/2022/talks/i-have-come-to-bury-the-bios-...
* https://news.ycombinator.com/item?id=33145411 (discussion at the time)
They are likely looking at future racks with GPU as well, but as a first product getting the basics right makes more sense.
Nvidia GPU drivers are very proprietary, which means that admins and developers have limited visibility into them if they misbehave in any way. This goes against Oxide's philosophy of full visibility into a system that you purchase.
Nvidia's CUDA software has a significant lead ahead of AMD and Intel GPUs, and they're not going to open source it any time soon. But this is a rapidly changing landscape, and AMD and Intel and others are pouring an enormous amount of research into getting their hardware and software to match what Nvidia has going. Nvidia is in pole position, but they're not guaranteed to stay there.
There's still a large market for the CPU workloads that Oxide is offering. For now, Oxide will be concentrating on meeting this traditional compute demand. But you're right to point out that in 2023, the absence of a top tier GPU in these racks is noticeable. I suspect Oxide will want to include some form of GPU or TPU into the next version of their system, but they won't just grab whatever hardware happens to be in fashion. It needs to work with their system as a whole.
This is where they will enter the real “hard” part of hardware, with an exec team from software. Can they respond to the market while making hardware?
They seem to have presented as generally thoughtful about their approach. If they can release major variants about annually or even sub-annually that is what I think will enable them to win.
If you take some time and read around their site you'll see that they're offering a ready to run (turnkey) server. They have everything packed together, they've integrated everything that is needed (CPU, disk, networking...) into a nice looking cabinet, with not too many wires and they're selling it as a complete package.
If you're in need of a server (cloud computer) and you don't want to but separate components and unpack them and connect them yourself, then this looks like a possible solution.
Oxide Cloud Computer
No Cables. No Assembly. Just Cloud.
Contact Sales
How much easier can they make it? They clearly want to sell computers.
I mean, the former is just the latter with some of the setup done for you no? Anyways, it’s a full server rack, with tightly vertically integrated hardware and software. Not sure if you’ve poked around the rest of their site, but it seems like their whole software stack is designed with some really nice usability and integration in mind: there’s a little half-snippet there suggesting that provisioning bare-metal VM’s out of the underlying hardware could as trivial as provisioning an EC2 with Terraform, and if that’s the case, that’s _massive_.
> And if it's a server rack, how come you don't need any cables?
Because they’ve gone to great lengths and care to design it to not need anything extraneous IIUC. I think the compute sleds all automatically mount into some automatic backplane that presumably gives you power, cooling and networking, and then, as above, you presumably configure all that via software, as you would your AWS setup. Not an expert here though, happy to be corrected by anyone who actually knows better.
> Do they host it in their data center or deliver it to you?
Presumably the latter, given they’re a hardware company, but if their software is even a 10th as good as it seems, I fully believe there’ll be a massive market for renting bare-metal capacity from them.
You use words like: it seems, I think, presumably, I believe... This is what we're arguing. A company that has raised $44 million Series A for sure can afford to clearly write what they offer.
I understand, you can't have all the people happy and no matter what you do there will always be "weirdos" that don't like your page/design/wording, but hey at least recognize it :-)
We have a terraform provider, yes https://github.com/oxidecomputer/terraform-provider-oxide
In a sense. Yes, it is a rack of servers. You're buying a computer. But we've designed the rack of servers as a full rack of servers, rather than an individual 1U. Comes with software to manage the rack like you would a cloud; you don't think of an oxide rack as individual compute sleds, you think of it as a pool of capacity.
> And if it's a server rack, how come you don't need any cables?
Because you are buying an entire rack. The sleds are blind mated. You plug in power, you plug in networking, you're good to go. You're not cabling up a bunch of individual servers when you're installing.
> Do they host it in their data center or deliver it to you?
Customers get them delivered to their data center.
Happy to answer any other questions.
It wasn't obvious to me you can own the stuff where it runs (I hope I understood it correctly)
So it is big, it looks good, but no specs. What architecture ? Some "cinebench" numbers ...
BTW, they are following some no pictures policy. I found only a few pictures of boards but no picture of the product as a whole.
The key difference is the price structure -- IBM leases the hardware wherever they can get away with it, and uses a license manager to control how many of the machine's resources you can access (based on how much you can bear to pay). This, however, is like a mainframe you own.
This is what Oxide has to offer. They have built essentially an entire rack that solves the physical space, power, cooling, networking, and storage issues by simply giving you a turn-key box you plant in your data center and hook power and interconnects to. In addition it is a fully integrated solution so they can capture a lot of efficiencies that would be hard or impossible in traditional design.
As someone with a lot of data center experience I am very excited to see this. It is built by people with the correct attitude toward compute, imo.
The main USP: you own it, you're not renting it. You're not beholden to big cloud's pricing strategies and gotchas.
Secondary USP, why you buy this rather than DIY / rack computer vendor: it's a vertically integrated experience, it's preassembled, it's plug and play, compatibility is sorted out, there's no weird closed third party dependencies. Basically, the Apple experience rather than the PC experience.
It's the iPhone of hyperconverged infrastructure.
(Sorry; three sentences.)
The industry moved to commodity x86 for compute for a reason.
https://www.asrockind.com/en-gb/4X4-7840U-1U
https://youtu.be/WCRK-Uwb0EA?si=zMYn2Gf3vTe-qMl8
https://www.amd.com/en/products/apu/amd-ryzen-7-7840u
They sell servers, but as a finished product. Not as a cobbled together mess of third party stuff where the vendor keeps shrugging if there is an integration problem. They integrated it. It comes with all the features they expect you to want if you wanted to build your own cloud.
Also, they wrote the software. And it's all open source. So no "sorry but the third party vendor dropped support for the bios". You get the source code. Even if Oxide goes bust, you can still salvage things in a pinch.
Ironically this looks like the realization of Richard Stallman's dream where users can help each other if something doesn't work.
They sell rack-as-compute.[0] Their minimum order is one rack: You plug in power and network, connect to the built-in management software (API), and start spinning up VMs.
[0] With built-in networking and storage.
This achievement is clearly worth a lot to people.
* https://docs.oxide.computer/api/guides/responses
It's nice, it's just nothing new.
I believe the episode name was "Virtualizing Time"
https://pca.st/episode/c10ce39c-1348-407f-b9c2-a36ced4e6be8
"no cables no assembly just cloud" wtf is that
Size of doorways, weight bearing capacity of floors, electrical service parameters, environmental conditions, etc.
e.g. Does it actually handle electrical voltage fluctuations of +/- 1V, or whatever is advertised?
The guaranteed parameters of a fully set up machine:
Minimum performance metrics, software compatibility with whatever the sales department promised, maximum power draw, etc.
e.g. Can it reliably hit X metric (FLOPS, IOPS, Integer calculations, etc.)?
And so on.
Or does the customer need to take on some risk and hazard associated with installation, configuration, initial boot up, etc.?
e.g. If someone buys with the intention of using it up to X FLOPS, and the machine only delivers Y FLOPS once it's all said and done, what happens?
Otherwise I would imagine it would be a major selling point and be advertised publicly.
But if it is a complicated enough internal issue for that not to be the case, then my apologies for pestering.
The comment you replied to was not questioning the value of integrated cabling. It was pointing out that the product description on the site does not make sense.
"Cloud computer" sounds like a server you rent from AWS. It's kind of like calling Rust "cloud compiler."
If you choose to use words that your audience doesn't understand, or even worse understands to mean the opposite of what you want them to mean, it's a good idea to explain these words immediately using conventional words with conventional meaning. The comments by throw0101a did that.
The product seems really cool, but there is no way I would've understood what it was from the website.
And for what it's worth, I don't think you need to explain what's happening to Steve, it seems to me that he understands perfectly well. To me you come across as being rather condescending and in my opinion Steve is being commendably polite in response.
Cloud as a term is pretty undefined/overloaded. It makes sense to me if I think about 'cloud' computing as API-based provisioning on pooled resources.
No cables, except for a few cables.
To be super clear about it, this is referring to not needing to cable up all of the individual sleds to the rack upon installation. It doesn't mean that we recommend connecting a rack of compute to your data center via wifi.
I think their description conveys what it does just fine for the target audience.
When people hear cloud, it means that aspects such as electricity costs, electricity stability, Internet, bandwidth, fire protection, safety, etc etc are abstracted away.
Oxide IS on-premise, right? The website is very vague and wishy-washy.
If you have a rack like that you'd probably want it to run a real database or server backend.
Ehrm. What is that exactly?
Are you alluding to cute design, different user interface? Or ditching then common PC component modularity? "Thinking differently"?
One difference is that Oxide development was done in the open and they don't seem hell bent on creating a closed ecosystem. (Yet, at least)
You are also correct that we diverge from Apple in other ways, such as our commitment to openness, rather than secrecy.
The first iMac famously made it easy to connect to the Internet; The 'i' in iMac was for "Internet". Its setup manual was a couple of pages long, mostly pictures and IIRC, just 37 words.
https://www.youtube.com/watch?v=rX4PzvWgu6c
Sun did that experiment:
https://en.wikipedia.org/wiki/Sun_Modular_Datacenter
* https://en.wikipedia.org/wiki/Modular_data_center
Also:
* https://www.deltapowersolutions.com/en/mcis/data-center-solu...
* https://atos.net/en/solutions/high-performance-computing-hpc...
* https://www.zelladc.com/zella-max
Search for "shipping container data centre" or "containerized datacenter".
I’m actually extremely impressed. I want one. I haven’t worked in a data center in years, but I’d be tempted to do it again just to get my hands on one.
Oxide feels like it'll be that "integrated" experience, but with the added benefit of software freedom.
Also https://artemis.sh/2022/03/14/propolis-oxide-at-home-pt1.htm...
Does illumos run on ARM?
I wish IBM would shrink their LinuxONE Rockhopper 4 Rack Mount down to at least an "under the desktop" model. To my knowledge, IBM still makes quality products and has excellent customer service. They have fun names too (Rockhopper and Emperor are types of penguins!) and they even have 3D models of their rack mount cloud computers with shadows. [3] In fact, when I first read about Oxide a year or two ago, I searched for "IBM cloud server", and left it at that. So IBM, could you please send someone from the LinuxONE down to Boca Raton to create our new PC? :) Thanks!
P.S. I would even accept a shrunken 370. :)
[1] https://en.wikipedia.org/wiki/IBM_Personal_Computer
[2] https://archive.org/details/byte-magazine-1981-01/page/n313/...
[3] https://www.ibm.com/demos/it-infrastructure/ux/product.html#...
Proxmox and a full rack of Supermicro gear would not be as sophisticated, but end result is pretty much the same, with I imagine far far better bang for buck.
I like it, but it doesn't seem like a big deal or revolutionary in any way.
When it breaks, good luck!
You're no worse-off with Oxide from that perspective. Their open source firmware means that thr opportunity to pay somebody else to support you at least exists.
Even small shops can use bad experience as leverage for credits and discounts, especially if the vendor has account managers. This is one of the (few) benefits of having a human involved in invoicing vs. self-serve.
> when any of that breaks, good luck!
The premise is that you don't need luck, you can call Oxide. As you said, they wrote all of it, so they own all the interaction so they can diagnose all of it.
When I call Dell with a problem between my OS filesystem and the bus and the hardware RAID, there's at least three vendors involved there so Dell doesn't actually employ anyone that knows all of it so they can't fix it.
Sure, Oxide now needs to deliver on that support promise but at least they are uniquely positioned to be able to do it.
The rest comes down to execution. Sure, we all have high hopes for Oxide. Sure, we all hate established players like Dell.
Just about any company will sell your company a support contract.
The more interesting question is, can they back it up with action when push comes to shove? I suspect most people have plenty of stories of opening support tickets with big name vendors that never get resolved. And through the grapevine you find out that they won't fix it because they can't fix it. They might not even have access to the source code or anyone on staff who has a clue about it because it came from who knows where. Sales is happy to sell you the support contract but it doesn't mean your problems can be fixed. BTDT.
From listening to the Oxide podcasts, my impression is that Oxide actually can technically fix anything in the stack they sell, which would make them vastly different from Dell et.al.
I used to work for a company targeting Fortune 500s. At that level of spend, when a client had a problem, somebody got on a plane. Only a fraction of those problems escalated all the way to R&D, which is where Oxide skills are. That's where VMWare etc are hard to beat.
Can an engineering first approach break into the cloud market? Hard to say as enterprise sales is very powerful, and the numerous "worse is better" forces always loom large in these endeavours. That said, enterprise sales driven companies are fat, slow and complacent. Oxide is lean and driven, and a handful of killer use cases and success stories is probably enough to sustain them and could be the thin end of the wedge on long-term success. We can hope anyway.
I think the question is how well they can do the management plane. Dealing with the "quirks" of a bunch of grey box supermicro stuff is always painful in one way or another. The drop shipped, pre-cabled cab setups are definitely nice but that's only a part of what Oxide is doing here. No cables and their own integrated switching sounds nice too (stuff from the big vendors like UCS is closer to this ballpark but also probably closer to the cost too).
I suspect cooling and rack density could be better in the Oxide solution too, not having to conform to the standards might afford them some possibilities (although that's just a guess, and even if they do improve there these may not be the bottlenecks for many).
Docs:
* https://docs.oxide.computer/api/guides/responses
See perhaps "This repo houses the work-in-progress Oxide Rack control plane."
* https://github.com/oxidecomputer/omicron
On-prem buyers are doing cost reduction and cost reduction targets things like, as one example, the crazy cost of GPU servers on the CSPs. Your run of the mill stuff is very hard to cost reduce.
You can see their sort of lack of getting it by using Tofino2 as their switch. That’s just a very bad choice that was almost certainly chosen for bad reasons.
The cost case only works for GPU heavy workloads which this isn’t - wrong chassis, wrong network, etc.
Tofino2 is the wrong choice because even when they made that choice it would have been clear that it’s doa. Intel networking has not been a success center in, well, ever. That’s a selection that could only have been made for nerd reasons and not sensible business goals alignment or risk mitigation.
When you make an integrated solution you’d better be the best or close to the best at everything. This does not seem to be the best at anything. I will grant that it is elegant and largely nicer than the hyper converged story from other vendors but in practical terms this is the 2000s era rack scale VxBlock from Cisco or whatever Dell or HPE package today. Marginally better blade server is not a business.
They also make a big deal and have focused on things no one who actually builds data center pods cares about.
I actually hope they get bought by Dell or HPE or SuperMicro. Those companies could fix what’s wrong here and benefit a lot from the attention to detail and elegance on display.
My experience with the likes of Dell is that they'll deliver it but they won't support it.
Sure, there's a support contract. And they try. But while they sell a box that says Dell, the innards are a hodgepodge of stuff from other places. So when certain firmware doesn't work with something else, they actually can't help because they don't own it, they're just a reseller.
Charging for EBS local snapshots on your own Outpost S3 storage and Route 53 Resolvers on your own compute is a weird one. I don't know how they defend that. To me, it seems indefensible.
> You can purchase Outposts servers capacity for a three-year term and choose between three payment options: All Upfront, Partial Upfront, and No Upfront. … At the end of your Outposts servers term, you can either renew your subscription and keep your Outposts server(s), or return your Outposts server(s). If you do not notify AWS of your selection before the end of your term, your Outposts server(s) will be renewed on a monthly basis, at the rate of the No Upfront payment option corresponding to your Outposts server configuration.
https://aws.amazon.com/outposts/servers/pricing/
> You can purchase Outposts rack capacity for a 3-year term … either renew your subscription and keep your existing Outposts rack(s), or return your Outposts rack(s)
https://aws.amazon.com/outposts/rack/pricing/
There is no permanent purchase option.
I do wonder if AWS will eventually go down market if oxide gets any scale.
It seems like AWS has all the pieces to compete with Oxide if they care to.
Is this true? Can you set your own root of trust for the firmware signing key and build and deploy it yourself?
DC build out cost and effort
Power cooling requirements
Dark fiber bandwidth requirement
New headcount to support all this
No thanks , I have widgets to make and sell and a business to run.
And beyond that, there's all sorts of weird environments that need a lot of local compute. You'd be shocked how many servers are on a cruise ship for one example among many.
Prices are here: https://oxide.computer/sales ;)
I am not in sales and so I hesitate to speak on it in case I am incorrect, but the way that I personally think about it is that it is true that it is not an inexpensive product: there's a LOT of computer here. But the goal is to be competitively priced.
I wish there was a 4U version (10-25K but I don't think that they could come close to that price point - regardless even that is out of reach for me to ever get to play on one :-/ )
One of the most obvious examples of the problem with this approach is that they're shipping previous generation servers on Day 1. One can easily buy current generation AMD servers from a number of vendors.
They will also likely charge a significant premium over decoupled vendors that are forced to compete head-to-head for a specific role (server vendor, switch vendor, etc).
Their coupling approach will most likely leave them perpetually behind and more expensive.
But there are advantages too. Their stuff should be simpler to use and require less in-house expertise to operate well.
This is probably a reasonable trade-off for government agencies and the like, but will probably never be ideal for more savvy customers.
And I don't know how truly open source their work is but if it's truly open source, they'll most likely find themselves turned into a software company with an open core model. Other vendors that are already at scale can almost certainly assemble hardware better than they can.
It seems like they're trying to hit a middle ground between cloud vendors and fully decoupled server equipment companies.
Using Oxide is likely cheaper over the life of the hardware than using a cloud vendor. A company who already has in-house expertise on running racks of systems may be less the target market here than people who want to do cloud computing but under their own control.
True, but Oxide may find themselves competing against Dell or HP if they adopt Oxides software for their respective servers. Additionally, Oxide may find itself competing against consultants and vendors in specialized verticals (e.g. core Banking software + Oracle DB + COTS servers + Oxide software). Oxide, and their competitors are going for people who used to buy racks of Sun hardware.
> Their coupling approach will most likely leave them perpetually behind
This is a startup that took years to get their initial hardware developed. The time between this version and the version using the next version of AMD chips will be shorter than the time it took to develop this product. This is not an inherent issue with coupling vs decoupling.
Also, most servers are rarely running on the most recent cpus anyway. At least in companies I've worked at with on-site hardware they're usually years (sometimes even a decade) out of date getting the last life sucked out of them before too many internal users start complaining and they get replaced.
I'm sure they'll improve their processes over time but the lag will probably always be a non-zero value. Hopefully they'll be able to keep it low enough that it's not an important factor but as a customer it's certainly something one should consider.
It would be surprising if they don't run into some nasty issue that leaves their customers 6+ months behind on servers or switches at some point.
The talk here talks about that from about 32:15 : https://www.osfc.io/2022/talks/i-have-come-to-bury-the-bios-...
As to your second point, unless AMD somehow becomes supply constrained and only wants to ship to their most important customers first I don't see a future where there would be any lag. Again, the delay this time is from how long it took from company start until product release. Future delays will be based on the time it takes from them getting early development parts to released products, which they could even possibly beat Dell to market on given the smaller company size and IMO more skilled employees.
> It would be surprising if they don't run into some nasty issue that leaves their customers 6+ months behind on servers or switches at some point.
I mean they've already hit tons of nasty issues, for example finding two zero-day vulnerabilities in their chosen security processor. They've shown they can work around issues pretty well.
I just think your premise is wrong - most customers don't care about not having the absolute latest and greatest. Indeed they will often avoid them because
1. They are new so more likely to have as yet undiscovered issues ( hardware or drivers ).
2. If you buy top end, they sell at a premium well above their performance premium.
ie the customers who are perennially chasing the latest hardware are in the minority.
1. The way to run into undiscovered issues is to choose a completely custom firmware/hardware/software stack that almost no one else in the world is running.
2. Not sure where you're getting this from. There is almost always a price:performance calculation that results in current generation smashing the previous generation with server and switch hardware. Often this means not buying the flagship chips but still the current generation.
And a major reason to get off old generations of hardware is that they become unavailable relatively quickly. It's always easier to buy current generation hardware than previous generation hardware, especially a couple years into the current generation. This has nothing to do with chasing the latest hardware.
That's not in the customers interests per se- in fact it's a pain. Having control of their own stuff could mean they could offer a much longer effective operational life.
> The way to run into undiscovered issues is to choose a completely custom firmware/hardware/software stack that almost no one else in the world is running.
What breaks stuff is change - sure when they are starting up it's higher risk - but again if they can manage the lifecycle better, not have change for changes sake, then they could be much more reliable.
> Not sure where you're getting this from.
I was talking about not taking the flagship stuff - which is typically a few months ahead of the best price/performance stuff.
IDK about every use case, but slightly older generations of CPUs would affect me roughly zero. I'm sure there are things so compute-intensive that one would care very much, but a lot of people probably wouldn't bat an eye about that, and not because they're unsavvy.
To the extent that these things are supported as a whole by the vendor rather than a bunch of finger pointing though, that could be massive, specifically in terms of how many staff members you could "not hire" compared to if you had to employ someone to both build and continually maintain it.
I'm posting this not to invalidate what you're saying, just to say that a little predictable upfront amount of money (the premium) will be spent very happily by lots of people who value predictability and TCO over initial price.
A moderately large Haswell cluster is equivalent in power to a moderately powerful modern server.
How is it ironic?
Also, it has little to do with the cloud; it is yet another hyperconverged infra.
Weirdly, it is attached to something very few people want: Solaris. This relates to the people behind it who still can't figure out why Linux won and Solaris didn't.
Also there are many situations where renting, for example a flat makes a lot of sense. And there are many situations where the financials and or enabled options of owning something make a lot of sense. Right now, the kind of experience you get with AWS and co. can only be rented, not bought. Some people want to buy houses instead of renting them.
Their competition has open source firmware as well:
https://www.dell.com/en-us/blog/enabling-open-embedded-syste...
[0] https://www.osfc.io/2022/talks/i-have-come-to-bury-the-bios-...
https://www.dell.com/en-us/blog/hardware-root-trust/
For a mostly open source solution, not only would you need open source BMC firmware, you must have an open source UEFI/BIOS/boot firmware like CoreBoot, LinuxBoot, Oreboot, Uboot, etc.
And Canonical played with a cluster-in-a-box all the way back in 2013-2014: https://www.zdnet.com/article/canonicals-cloud-in-a-box-unde...
You could turn it into an OpenStack cloud in ~20 mins with an automated Juju OpenStack install.
Sure you can, but then who will diagnose and fix your hardware/OS interaction problems when you have parts from five vendors in the mix?
If you haven't lived through this, the answer is: nobody. Everyone points fingers at the other 4 and ignore your calls.
Back in the day you could buy a fully integrated system (from CPU to hardware to OS) from Sun or SGI or HP and you had a single company to answer all the calls, so it was much better. Today you can't really get this level of integration and support anymore.
(Actually, you probably can from IBM, which is why they're still around. But I have no experience in the IBM universe.)
This is why Oxide is so exciting to me. I hope I can be in a company that becomes a customer at some point.
Dell is a single vendor that will diagnose and fix all of your hardware issues.
With Oxide you're locked into what looks like a Solaris derivative OS running on the metal and you're only allowed to provision VMs which is a huge disadvantage.
I run a fleet of over 30,000 nodes in three continents and the majority is Flatcar Linux running on bare metal. Also have a decent amount of RHEL running for specific apps. We can pick and choose our bare metal OS which is something you cannot do with Oxide. That's a tough pill to swallow.
And you'll be down for weeks or months while they do it.
Off by a few orders of magnitude. Dell on-site SLA with pre-purchased spares was about 6 hours.
With Oxide, you'd be lucky to get same day service.
You're talking about replacement parts. Yes Dell is good about that.
The discussion above is asking them to diagnose and fix a problem with the interaction of various hardware components (all of which come from third parties).
There's a line in the sand that you must cross when it comes to understanding the true nature of the componentry that you're using. At the end of the day, your AMD CPUs may be lying to you, to all of us, but we just don't know it yet.
Neither am I.
>If you hit a "zero-day" bug that Dell has never seen it's going to take time.
If you hit a "zero-day" bug that Oxide has never seen it's going to take time.
>And somehow every large customer finds bugs that Dell certification didn't.
Yes, happens. And I'm sure the exact same will happen with Oxide, so it's not a differentiator.
It's a law of computer engineering.
In the Apollo 11 decent sequence the Rendezvous Radar experienced a hardware bug[0] not uncovered during simulation. They found it later, but until then, the solution was adding a "turn off Rendezvous Radar" checklist item.
[0] The Rendezvous Radar would stop the CPU, shuttle some data into areas it could be read, and woke the CPU back up to process it. The bug caused it to supuriously do this dance just to tell it "no new data", which then caused other systems to overload.
I've been a Dell customer at a previous company. I know for a fact that's not true.
I had a support ticket for a weird firmware bug open for two years, they could never figure it out. I left that job but for all I know the case is still open many years later.
Dell doesn't know how to fix things like that because they don't design and engineer the systems they sell. Dell is a reseller who puts components together from a bunch of vendors and it mostly works but when it doesn't, there's nobody on staff who can fix it.
I've had support tickets open for all kinda of weird firmware, hardware, etc. bugs and they've been well resolved, even if it meant Dell just replaced the part with something comparable (NIC swap).
>Dell doesn't know how to fix things like that because they don't design and engineer the systems they sell.
Of course they do. That's like saying Oxide doesn't know how to fix stuff because they don't design the CPU, NVMe, DIMMs, etc. Oxide is still going to vendors for these things.
At Oxide, we have been deliberate at every step, designing from first principles whenever possible. (We -- unlike essentially everyone else -- did not simply iterate from a reference design.)
To make this concrete with respect to the CPU in particular, we have done our own lowest-level platform enablement software[0] -- we have no BIOS. No one -- not the hyperscalers, not the ODMs and certainly not Dell -- has done this, and even AMD didn't think we could pull it off. Why did we do it this way? Because all along our lodestar was that problem that Dell was useless to us on -- that we wanted to understand these systems from first principles, because we have felt that that is essential to deliver the product that we ourselves wanted to by.
There are plenty of valid criticisms of Oxide -- but that we don't understand our system simply isn't one of them.
[0] https://www.osfc.io/2022/talks/i-have-come-to-bury-the-bios-...
[0] https://github.com/oxidecomputer/phbl
[1] https://github.com/oxidecomputer/illumos-gate/tree/stlouis
[2] https://github.com/oxidecomputer/illumos-gate/blob/stlouis/u...
Nonetheless, it is quite interesting what you've built, but as the end user I'm not quote convinced that it matters. Sure you can claim it reduces attack vectors and such but we'll still see Dells and IBMs in the most restricted and highest security postured sites in the world. Think DoD and such. Core/libreboot with RoT will get me through compliance the same.
The software management plane y'all built is the headlining feature IMHO, not so much what happens behind the scenes that the vast majority of the time will not have a fatal catastrophic upstream effect.
>There are plenty of valid criticisms of Oxide -- but that we don't understand our system simply isn't one of them.
That's not what I said. There's a line in the sand that you must cross when it comes to understanding the true nature of the componentry that you're using. At the end of the day, your AMD CPUs may be lying to you, to all of us, but we just don't know it yet.
Set aside the childish tone ...
> Dell is a single vendor that will diagnose and fix all of your hardware issues.
There are two anecdotes here disagreeing with you, and frankly that's enough to say what you said above isn't true, not universally so. I doubt Odixe is targeting big deployment like yours, but more like theirs. Whether they will succeed is another matter, but they do have a valid sales pitch and the expertise to pull it off.
Huh, then why did Sun, Oracle, and Veritas have to set up a shared tech support center in San Jose?
"Accelerated finger pointing", said a friend who had to do business with them.
At least with Oxide there is a glimmer of hope for a better future in this regard.
Yes they are using illumos/Solaris to host this but they don't sell on that, they sell on the functionality of this layer — allowing people to deploy to owned infra in a way that is similar to how they'd deploy to AWS or Azure. How much do you ever think about the system hosting your VM on those clouds? You think about your VMs, the API or web interface to deploy and configure, but not the host OS. With Oxide racks the customers are not maintaining the illumos substrate (as long as Oxide is around).
You could be right about demand, there is risk in a venture like this. But presumably the team thought about this - I think folks who worked at Sun, Oracle, Joyent, and Samsung and made SmartOS probably developed a decent sense of market demand, enough to make a convincing case to their funders.
Now you need to know both the OS they chose and the OS you chose...
(No, I don't believe it'll be 100% hands-off for the host. This is an early stage product, with a lot of custom parts, their own distributed block storage, hypervisor, and so on.)
I wouldn't have picked Opensolaris, but it's a lot better than other vendors that are either fully closed source, or thin proprietary wrappers over Linux with spotty coverage and you're not allowed to touch the underlying OS for risk of disrupting the managed product.
GNU/Linux was also "risky" at some point.
This appears to be a much better system than VMware, is free as in software, and it builds upon a free software operating system with lineage that predates Linux.
I say this in the most critical way possible, as someone who has built multiple Linux-based "cloud systems", and as a GNU/Linux distribution developer: I love it!
In hindsight of course it was remarkably prescient. This from a guy at a company that was built entirely around SGI at the time.
The first time I actually saw GNU/Linux powering something in production was in 2003, when I joined CERN and they were replacing their use of Solaris, and eventually alongside Fermilabs came up with Scientific Linux in 2004.
Later at Nokia, it took them until 2006 to consider Red-Hat Linux a serious alternative to their HP-UX infrastructure.
"Sir, if you have nothing to hide, why do you talk so fast, and pronounce words like a foreigner who learned English from a book?"
:)
We love you Bryan, never change.
I don't run on-prem clusters or clouds but know a couple people who do and, at large enough scale, it is a constant "fuck-shit-stack on top of itself" (to quote Reggie Watts). There is almost always something wrong and some people upset about it.
The promise of a fully integrated system (compute HW, network HW, all firmware/drivers written by experts using Rust wherever possible) that pays attention to optimizing all your OpEx metrics is a big deal.
It may take Oxide a couple more years to really break into the market in a big way, but if they can stick it out, they will do very well.
That's because they own the whole stack, from CPU to GUI and support it as a unit. That's the benefit of having a product where a single owner builds and supports it as a whole.
My impression of Oxide is that that's the level of single source of truth they are bringing to enterprise in-house cloud. So, I strongly doubt the innards would ever become customer-facing (unless the customer specifically wants that, being open source after all).
As for why I think Helios will become customer facing: Oxide is a small startup. They have limited resources. Their computers expensive enough to be very much business critical. You'll get some support by Oxide logging in remotely to customer systems and digging around, but pretty soon the customer will want to do that themselves to monitor/troubleshoot the problems as they happen.
Imagine you're observing a recurring but rare I/O slowdown that seems to trigger under some certain conditions, and tell me a competent sysadmin wouldn't want to log in on all the related boxes (client Helios, >=3 server Helioses for the block store) and look at the logs & stats.
Apple is a great example of the benefits of an integrated system where the hardware and software are designed together. There are tons of benefits to that.
What makes Apple evil (IMO, many people disagree) is how everything is secret and proprietary and welded shut. But that doesn't take away from the benefits of an integrated hardware/software ecosystem.
Oxide is open source so it doesn't suffer from the evil aspect but benefits from the goodness of engineered integration. Or so I hope.
Afaik Android phones tend to have a lot more hardware than your average laptop, too (cell modem, gps, multiple cameras, gyro, accelerometer, light sensors, finger print readers)
I think the real benefit is being able to move/deprecate/expand at will. For example, want an app that would require special hardware? You can just add it. Want to drop support for old drivers? Just stop selling them and then drop (deprecate) the software support in the next release.
I fully agree about the evilness, and it baffles me how few people do!
Have you looked at the pedigree of many of the people behind the project? I don't say this because "these guys smart", but because these guys bent over backwards for their customers when they were Sun engineers. Bryan didn't write dtrace for nothing.
> Imagine you're observing a recurring but rare I/O slowdown that seems to trigger under some certain conditions, and tell me a competent sysadmin wouldn't want to log in on all the related boxes (client Helios, >=3 server Helioses for the block store) and look at the logs & stats.
I think you're simultaneously over-estimating and under-estimating the people who will deploy this. There's a lot of companies who would want a "cloud in a box" that would happily plug hardware in and submit a support ticket if they ever find an issue, because their system engineers either don't have the time, desire, or competence (unfortunately common) to do anything more. The ones who are happy to start debugging stuff on their own would have absolutely wonderful tooling at their fingertips (dtrace) and wouldn't have any issue figuring out how to adapt to something other than Linux (hell, I've been running TrueNAS for the better part of a decade and being on a *BSD has never bothered me).
As we see nowadays on tablets and laptops, most OEMs are quite keen in returning back to those days, as otherwise there is hardly any money left on PC components.
Or Linux running underneath all the Java-y Android stuff.
Also, Apple is just the one who survived. Previously I'd have thought of SGI, DEC, Sun, HP, IBM, Dell some of whom survived some not.
Those three consumer products I mentioned each provide a platform for a user and business space to floroush and thrive. I expect a company doing something similar for cloud computing to want the same. But it will require some magick: momentum, money, trust. That kind of stuff, and loads of it. (With some big names behind it and a lot of FOSS they got me excited, but I don't matter.)
Their choice in foundation OS (for lack of a better term) really should not matter to any customer.
Now imagine a multi-million dollar mission critical pile of computers running on premises, and your sysadmin being able to do so.
Oxide is closer to a rack of Supermicros than AWS.
Solaris is still Solaris, as of the latest release last month. OpenSolaris hasn't been OpenSolaris in a while and is Illumos, yes.
Could be! Seems too early to tell though, and remains to be seen whether it pencils. Which is the whole idea of starting a new venture, no?
One of the more interesting discussions I had during my tenure at Google was about the "size" of the unit of clusters. If you toured Google you got the whole "millions of cheap replaceable computers" mantra. Sitting in Building 42 was a "rack" which had cheap PC motherboards on "pizza dishes" without all that superfluous sheet metal. Bunches of these in a rack and a festoon of network cables. What are the "first class" elements of these machines? Compute? Networking? Storage? Did you replace components? Or a whole "pizza slice" (which Google called an 'index' at the time). Really a great systems analysis problem.
FWIW I'm more of a "chunk" guy (which is the direction 0xide went) and less of a "cluster" guy (which is the way Google organized their infrastructure). A lot of people associated with 0xide are folks I worked with at Sun in the early days and during that period the first hints of "beowulf" clusters vs "super computers", was memory one thing (UMA) or did it vary from place to place (NUMA). I have a paper I wrote from that time about "compute viscosity" where the effective compute rate (which at the time largely focused on transactional databases) scaled up with resource (more memory more transactions/sec for example) and scaled down with viscosity (higher latencies to get to state meant fewer transactions/sec) Sun was invested heavily in the TPC-C benchmarks at the time but they were just one load pattern one could optimize for.
These guys have capitalized on all that history and it is fucking amazing! I just hope they don't get killed by acquisition[1].
[1] KbA is a technique where people who are invested in the status quo and have resources available use those resources to force the investors in a disruptive technology to sell to them and then they quietly bury the disruptive technology.
Is the unit of composition a rack (chunk), a server (smaller chunk), or a blade (smallest chunk)? In what I think of as classic systems architecture you've got a 'store' (storage), 'state' (memory), 'threads' (computation), and 'interconnect' (fabrics). In the 90's a lot of folks focused on fabrics (Cray, Connection Machine, Sun, etc) somewhat on threads (compute blades), and state came along for the ride. How these systems were composed was always a big thing, then along came the first Beowulf clusters that used off the shelf motherboards (a "chunk" of threads/state/store) with a generic fabric (Ethernet). Originally NASA showed that you could do some highly parallel processing on these sorts of systems and Larry and Sergei at Stanford applied it to the process of internet search.
Collectively you have a 'system resource' and with software you can make it look like anything you want. When you do compute with it, its performance becomes a function of its systems balance and the demands of the workload. Its all computer sciencey and yes there is a calculus to it. This isn't something that most people dive into (or are even interested in[1]) but it was one of the things that captured my imagination early on as an engineer. I was consumed with questions like what was the difference between a microprocessor, a mini-computer, a workstation, and a mainframe? Why do they each exist? What does one do that they other can't? Things like that.
[1] At Google I worked in what they called 'Platforms' early on and clearly most of the company didn't really care about the ins and outs of the systems bigtable/gfs/spanner/etc ran on, they just wanted APIs to call. But they also didn't care about utilization or costs. By the time I left some folks had just figured out (and one guy was building his career on) the fact that utilization directly affected operational costs. They still hadn't started thinking about non-uniform rack configurations for different workloads.
Private equity in the US has collectively determined that no company shall exist outside of investment ownership. I don't know what the ownership structure looks like, but generally speaking, it seems that nearly everyone has a "fuck you" number. Now that Oxide is venturing into Dell and HP's turf, I worry someone will get a fix on what Brian's number is.
>They sell servers, but as a finished product. Not as a cobbled together mess of third party stuff where the vendor keeps shrugging if there is an integration >problem. They integrated it.
that's only true if you think that "users" means "people who operate cloud computers", which is about as far from understanding what Stallman is talking about as is possible. Someone who makes SaaS and runs it on an Oxide computer is no less of a rentier capitalist than someone who makes SaaS and runs it on AWS.
Does this "lock" you into the Oxide platform and you just buy into the whole thing instead of buying some server from Dell, then running some Proxmox like software and a Docker host?
Virtually all the value we get from cloud is in their managed offerings of complex and difficult to admin systems like Kubernetes, Postgres, and managed storage layers.
That’s where all the value is but it’s also where all the cost is. If you just want compute, storage, and bandwidth all those can be had at commodity prices. Look at Hetzner, Hivelocity, FDCServers, DataPacket, OVH, or VPS providers like Vultr. You can get dozens of cores, terabytes of SSD, and gigabits of unmetered bandwidth for a few thousand dollars or less. It’s very cheap.
Without the high level managed stuff we would be off cloud five minutes from now.
I’m sure there is a market for this among people who run on premise data centers, but I think they would have a much larger market if they went further up the software stack. Right now they just look like they are competing with Dell and Supermicro, not Amazon or Google Cloud.
Man I would love to work for a company like that. I don't know why more people don't set their startups like that.
I always find that a very naive point of view, of course I would want to earn a US tech salary while living somewhere in the country side, who wouldn't. But I'm also aware that this is not how the world works in reality, there's different tax systems, different expense costs and we don't live in a global one-market world.
I find the strategy of defining different "zones", like most of the remote first / salary transparency companies much more realistic.
Well the co-founders live in the Silicon Valley area, with their physical HQ being in Emeryville:
* https://en.wikipedia.org/wiki/Emeryville,_California
For mortgages, you'd need to be looking in the cheaper parts of the Bay, but that still means "dense, boring suburb" as opposed to "crime-ridden slum".
In the case of everyone making the same salary, you could certainly still sabotage someone's success but I don't see how having the same salary makes this type of mentality _more_ likely than at a company with a more typical salary distribution.
My concern with everyone having the same salaries is that, potentially, employees have less motivation to excel in their individual work and are more likely to do the minimum to just stay in good standing with their employer and not get fired. Maybe a company can offset the lack of direct financial motivation with more of a team motivation that the financial success of the company as a whole results in financial reward for the individual or some other way of recognizing individual success inside the company.
I am doubtful this flat salary structure will result in a more successful company overall but I do think it's good to try new things. And yes this has probably been tried a number of times before but maybe not exactly like this. Or maybe some other external variables have changed w.r.t. other attempts in the past and this time it works. The typical salary structures we see in US companies today are the result of a large number of trials and errors and learning.
I totally understand why companies want to pay less though. It's massive cost savings and it makes sense for them to hire for less money.
It's not about what you want, it's about knowing your value. If your work is worth a SF salary then that's what you should be getting.
Moving from Idaho to SF doesn't magically make you more productive. The company knows it's still getting more value from you than what you're being paid. They just want to keep more of that value for themselves whenever possible.
Have some respect for yourself and know your worth
If your entire world consists of staying home and interacting remotely with a company, then you are correct: Location doesn't change anything.
However, moving to a high-energy city with a high density of experienced engineers and tech companies can increase your rate of learning, career advancement, and experience much more rapidly than living in a smaller city. You have to actually branch out and interact with local companies and people, but it does happen.
But this is all beside the point. Hiring is a labor market. Developers who live in SF have more high-paid job options to choose from than someone living in Idaho. As a result, you need to bid more to get them into your company. Hence, the higher salary.
The discussion about cost of living misses this point. The real reason developers from places like SF get paid more is because if you don't pay them wages that are competitive with their local companies, they're just going to walk away and take any number of higher paying jobs they have access to.
How much salary do you think should allocate to base and how much toward any location adjustments, in general?
- https://handbook.gitlab.com/handbook/total-rewards/compensat...
- https://www.checklyhq.com/blog/open-sourcing-our-pay-calcula...
- https://buffer.com/salary-calculator
Nevertheless, I agree everyone looks at this problem from their own POV, however it should not be the norm to provide equal compensation for equal work.
In slightly more detail
https://elsajohansson.wordpress.com/2017/09/13/what-does-a-w...
https://elsajohansson.wordpress.com/2022/09/16/the-wage-gap-...
Also, thank you for the links. I'll read this because I'm interested in knowing more about thinking behind this. My instinct is in this direction regarding salaries, but I want to develop more clarity. :)
I wouldn’t. I like cities and center of culture and human activity. And the stats generally show cities growing globally, so I’m not the only one.
While you argue it's naive and cite different tax systems and costs, companies are successfully adopting this approach even in high-cost areas like the Valley. It might be hard to argue that the world doesn't work like that in reality, given that it’s already happening.
While it may not be widespread, it doesn’t mean it’s without merit or unrealistic. After all, remember that today's 'unrealistic' could be tomorrow's norm! It's tempting to think that people in less expensive areas would be the main proponents of a uniform salary, but the reality is nuanced. Preferences are likely influenced by a variety of factors, not just cost of living.
Your later points about the differences between compensation at different company stages are well taken, however it could be difficult to assert how much this dynamic affects preferences given the practice is not limited to early stage companies and equity vests often fail to yield returns.
In addition, your suggestion that only people in poor places want egalitarian salaries, could be seen as disrespectful of other people, because it seems to ignores the totality of an individual while preferring to try to reduce them to simplistic motivations. In that way, it’s also considered abusive. And can also be seen as disrespectful of others experience, and maybe arrogant: "Only people in poor areas want such naive, unrealistic salaries."
Looking deeper, this aspect of your comment, combined with its narrow focus on a single explanation, might be interpreted as your attempt to express your personal frustrations at your own salary performance, or justify and rationalize why you may not be making more. This might occur because you may find it easier to view something you don't have as unrealistic and naive, rather than the result of choices you could change.
In short, while you mention that egalitarian salaries and enthusiastic support of them is naive and unrealistic, it could be argued that the view espoused in your comment is naive and unrealistic because: it lack awareness of complex dynamics; ignores the totality of an individual while preferring to try to reduce them to simplistic motivations, and dismisses real practices as unrealistic, which might also be seen as out of touch. Overall, your views unfortunately could be interpreted as narrow and an expression of personal frustration, instead of a reflection of underlying real dynamics.
To conclude, while it's likely there's some truth to the correlation you propose, it's also likely true that even if some correlation exists, location is not the only factor at play. People may have various salary preferences, independent of their location, just as the value they provide is also independent. Finally, indeed, your view could be expressed more respectfully of others.
Anywho, it's understandable you may have that perspective, given what your background might be. Yet it's always good to remember that you can adapt your view over time, can grow and can include more data to expand that awareness of reality which you value! :)
The egalitarian way where a business can divide the share of revenue that is allocated for salary equally, no matter the role. This makes sense from a philosophy of everybody being a team and contributing equally to the results of the company. It can foster an esprit de corps and and a sense of fairness. On the negative side it could encourage companies to have more burdensome measures of fairness and contribution, and lead to resentment towards colleagues who don't pull their weight.
Then there's the other method, where a value-based salary is allocated to each employee taking into account their experience, ability and effort. Crucially, however, this salary is not adjusted for location. That's the case to which I was speaking, specifically, even tho the type used by 0xide is clearly the egalitarian one.
That's the labor theory of value (see: Smith, Marx), which in theory sounds meritocratic but it can't really be measured or assessed.
In reality compensation either becomes a function of power, social currency and negotiation skills, which is the general norm in professions, or you have an institutionalized, perhaps even democratic process to determine salaries. Both of these variants generate overhead and are only approximations to what anyone would see as fair.
The variant here where everyone gets the same, generous piece of a pie seems refreshingly simple and honest. I would also assume that it attracts the right kind of people, who are intrinsically motivated (at least after the threshold of a very high level of comfort is reached.)
The idea that contribution "can't really be measured" is a cop-out. Contribution can't be measured perfectly but it can be estimated with some accuracy by people who are involved in day-to-day work. "Some accuracy" is really all that's required: as long as contribution is correlated with compensation to some extent, you have a functioning meritocracy.
> The variant here where everyone gets the same, generous piece of a pie seems refreshingly simple and honest.
I bet it works great if you have a small team, are extremely picky about hiring, and quickly fire bad hires. Otherwise it will be awful.
This is handwaving in the extreme. Anyone involved in software development knows that a single line of code can be critical to success or failure, as can a blob of 100k LOC, so product-quantity metrics are of almost no use. The "estimate" you're talking about generally comes down to general "feelings" about who works hard, which have repeatedly been shown to be poor metrics for actual contributions.
> The "estimate" you're talking about generally comes down to general "feelings" about who works hard, which have repeatedly been shown to be poor metrics for actual contributions.
How has this been "shown"? Anyway, you're begging the question that there's some way to determine "actual contributions" that we can compare to "feelings".
If you actually work with a group of people on a daily basis and can't rank order them in terms of usefulness, I find that astonishing. And remember, rankings don't have to be perfect, they just have to more accurate that random.
No, they have to better than both random and "everyone is, on average, and over an extended period of time contributing roughly the same". That's quite a challenge.
Do you go to customers and ask them which features provide the most value to them, and then follow the code back to the people who implemented them? Do you go to the customers who paid the most, and repeat the question to them only?
We're not talking about some award-prize ceremony speech in which we acknowledge that Dmitri and Aneka led the work to get version 8.0 which has been a huge success. We're talking about actual salaries, which are presumably linked in some way to actual sales, and I'm insisting that connecting individual developer efforts to the sales numbers is extraordinarily hard. "More" and "less" are not enough to come up with actual numbers.
This is something virtually all functional companies do when they decide raises. The fact that they don't do it perfectly isn't a huge problem, they just need to be more right than wrong.
> Do you go to customers and ask them which features provide the most value to them, and then follow the code back to the people who implemented them? Do you go to the customers who paid the most, and repeat the question to them only?
Does productivity equal sales? I don't think so. If someone does a good job implementing a feature that doesn't drive sales, that should count toward their productivity. Equally, imagine a task that could be assigned to anyone that drive sales: it doesn't make sense to reward the person who happened to be assigned this task when anyone could have done it.
You're demanding way too much here because you're unwilling to get "handwavy" and instead want some rigorous way to quantify productivity. Instead, embrace subjectivity! Imagine you're in charge and ask yourself questions like:
1. If I need to organize a meeting to address some problem that needs to be solved as soon as possible, who would I invite to the meeting?
2. If someone tells me he plans to quit, how much would I be willing to offer to convince him to stay?
3. If someone quits, how hard are they to replace? In terms of hiring a replacement and/or transferring their responsbilities to someone else.
I suppose there are some workplaces where it's genuinely hard to rank people. But my sense is that they're rare, small, and careful about hiring. Everywhere I've worked, this is not the case and I'm fairly sure this is the norm.
If your company brings in $10M a year, and somebody plans to quit (or has quit), how much will your sales drop (immediately, and over a period of time). That's the answer to how much you can afford to offer them to stay, and that's how much you should offer their replacement. Suppose that says drop by $1M and you can be satisfied that the drop is 100% a consequence of the departed (or soon to depart) developer - that's how much value they bring to the company, and by the logic of capitalism (which I don't play by, btw), you should pay them some amount less than that.
The problem is: you can't determine the value before they quit, and you can't be sure that their replacement will provide that value after they are hired.
Look, I understand that in an organization of any size, there are likely to be slackers that feel like a deadweight, and others who feel like the contribute far more than the average.
The question is: does tying salary to this perception actually bring the benefits you think it does (which are intimately connected to the notion of incentive) ? There's some good evidence that for developers and other "head-based" employees, it does not, and that flat pay scales create an environment in which you get different kinds of benefits.
I've worked exclusively in a distributed FLOSS project for the last quarter century, so in many respects, I'm not well positioned to talk about what happens inside traditional corporations.
Even if you actually do measure and agree on metrics, then the measurement can easily become the goal for those who are not intrinsically motivated. Work ethic can't be taught by dangling carrots in front of people, because acquiring the carrot becomes the goal instead of moving the cart. This can be detrimental in a highly collaborative workplace.
Having a flat, generous salary might solve this problem, because you filter out carrot hunters and get cart movers.
> I bet it works great if you have a small team, are extremely picky about hiring, and quickly fire bad hires. Otherwise it will be awful.
Finding the right people to work with is difficult regardless. The same worker can be miserable in one place and flourish in another.
Besides, LTV as a theory is meant to be a description of the world as Smith, Marx, etc. see it, not a prescription for how things should be done.
But the underlying belief of paying someone according to their effort, is very much based on the same premise.
What I'm saying is that nobody is _really_ paid according to their effort, experience etc. because those things cannot be reasonably measured.
The typical process of determining compensation is based on negotiation and power. In some places the process is more democratized and rules based. Both of these are only to some degree related to actual effort, experience and so on. This discrepancy becomes larger the more people are involved as well.
The skin-in-the-game bit can be promotions, equity, stock compensation, etc.
I've worked at two companies where everyone got the same base salary, but the variation came in equity.
If ya'll are looking for a remote security engineer from Europe hit me up. :)
For those numbers I'll walk the servers into the customers myself.
With this in mind, it's just a good salary. Until they have competition squeezing them on margins it looks like an OK approach. At some point their board most likely won't agree with paying some people below market rates for important roles and other more fungible roles paying 2-3x market rates but while they can keep doing it, it's great free marketing.
So the extremely high salary is still a net positive compared to EU
Plenty of that in London unfortunately and our salaries are half.
>Healthcare
And not so much of that... despite paying tax for it anyway.
Income tax: 34% total (top marginal rate of 39%).
Leaving $133k.
Other taxes (GST~VAT 15%, city rates, high petrol excise, etcetera) will easily take another $10k. Interest on your home is not deductible (NZ has very few deductables). People earning six figures will often pay for private health insurance and medical fees on top of the socialised healthcare - maybe another few thousand.
homes, etc is true though and we just need to build more inventory IMO. but there are tons of areas with affordable homes, they just aren't near the big cities like NYC or SF or LA
I took a base-pay cut to work at Oxide. Zero regrets.
I doubt my Aspen vacation home developer will accept payment in startup stock.
[0] https://www.linkedin.com/company/oxidecomputer/people/
I live in flyover country, and there are no shortage of talented people here, but I wouldn't dispute that the per-capita software developer talent in SF is going to be higher.
But for senior engineers at these companies, 200k is pretty competitive with the base salaries offered at those places
I meant 200k is just base salary at Oxide. It does not include equity
Open AI just had a tender offer for employees this month. It's not public. Clearly the equity is worth something even though the company is not public.
Some companies facilitate secondary transactions for their employees.
There are ways many employee friendly companies provide early liquidity without IPO.
Yes it is somewhat of a stumbling block, but I'm betting they are getting extremely high quality employees who are doing it for the passion and not just for the money.
Spec wise, some low power systems like an Intel NUC, LattePanda Sigma, or Zimaboard. You could fit 3/4 of them in a single 1u with a shared power supply. They could even offer a full 1u with desktop grade chips on the same sleds.
I have thought about building one myself, but it's a large investment of time that I can't seem to find lately.
Price point?
I work with some businesses who need very, very reliable, high-bandwidth, and low latency connectivity to their data. The amortized cost of on-prem beats the cost of any off-prem offering as soon as the cost of the necessary connectivity is factoted-in.
Your not going to find any serious hardware product with reliability guarantees, in writing, for much less than half a million anyways.
That said, where this product-space gets tough is actually scaling it down. It's pretty challenging to create something that is remotely stable/functional in a homelab (space/power/money) budget. Three servers and a switch would probably be the bare minimum. We (and I'm sure Oxide :) scale up like a dream.
A lot of homelabbers (and even some small businesses) go for Proxmox as a virtualization distribution. I don't use it myself, but IIUC it's effectively a Debian distro packaged to run KVM/LXC, with support for things like ZFS, Ceph, etc. It has some form of HA, an API used by standard open source devops tools, handles live migration, etc.
So buy some used rack-servers on Ebay (or new, if you're ballin'). A lot of businesses sell their old stuff, so you can pick up a generation or two out of date for a good price. If you want to do fancy stuff like K8s, Ceph, etc you'll probably want at least three nodes, ideally more, and a bunch of disks in them. Networking gear is a sort of pick your poison thing. A lot of people love Ubiquiti gear; a lot of people hate it. TP-Link is another that's good and budget friendly. StarTech sells smallish racks (including on Amazon), if you want to start there.
It won't look exactly like SoftIron's HyperCloud or Oxide's Cloud Computer, but you can certainly get pretty sophisticated.
Not sure if this answers your question, but other great spaces to explore are the 2.5 Admins and Self-Hosted podcasts.
To go ahead and dream a bit:
I'd hope for an online configurator like the one SoftIron's HyperCloud has [1] but instead of "talk to a sales rep", show a price for what you just configured, like you're configuring a macbook.
Relatedly, there should be a standard rack form factor in the size category of NUCs and Mac Minis, rather than having to go all the way to the 19 inch monster racks that medium to large businesses use. If it were nailed down to the point of being able to blind mate (just learned that term from Oxide's article here!) gear into it, that would be kind of perfect.
I’ve not personally used it, but their stack of software is open source, and according to some commenters in the thread, super high quality.
https://canonical.com/blog/jumpstart-training-with-the-orang...
https://arstechnica.com/information-technology/2014/06/hands...
Don’t know if oxide would want or be able to compete in the low cost market but a bigger a more expensive desktop/workstation as a mini homelab cloud could be a great option to get people trained on the oxide platform.
If you want to play around with their Hubris OS: "You wanna buy an STM32H753 eval board. You can download Hubris, and then you’ve got – you’ve got an Oxide computer. You have it for 20 bucks.”
[0] https://changelog.com/friends/8
https://estore.st.com/en/stm32h753i-eval2-cpn.html
Dude, why are you even in this thread? Trying to be a Debbie Downer is just going to ruin the rest of the day.
* https://xkcd.com/386/
Trying to get the last word in on the Internet is a quick and easy way to have a bad time. Let it go and move onto something else.
My google-fu failed to find any articles for or against my statement that weren’t paid advertising or lightweight tech summaries. StackOverflow will have to do. https://stackoverflow.blog/2023/02/20/are-companies-shifting...
If the opex model doesn't work for an organization, that they perhaps just went above that utilization threshold where it doesn't make sense anymore, and would be a potential 0xide customer?
When opex is a primary benefit of the cloud, that's specifically for start-ups and other businesses that have little working capital. The actual primary benefit of the cloud for most cloud customers is flexibility - prior to AWS adoption, getting a server provisioned could be a multi-week if not multi-month affair negotiating between devs or operators and on-prem infrastructure workers to get the servers through capacity planning and provisioning. AWS made provisioning so simple, that you could start to set up auto-scaling, because you had extraordinarily high confidence that the capacity would be immediately available when your autoscaler tried to scale up.
But opex is not a benefit of the cloud for heavy/established businesses. Cloud opex is a financial expenditure every month that you can't get rid of and counts directly against your profits. Indeed, the desire for capex in the cloud is so high that businesses routinely purchase Reserved Instances and other forms of committed usage, which allows accountants to treat the cost of the RI as capex and then discount the expenditure through depreciation (to zero, since there will be nothing to sell when the RI expires) over the lifetime of the RI. It is normal and frequent for businesses to make capital expenditures to reduce their operational expenditures over time, thus increasing their monthly/quarterly profits.
Oxide's unique value proposition is to give customers, particularly those with high monthly cloud bills that they have difficulty reducing, the operational flexibility of cloud computing with the profit-improving benefits of capital expenditures.
Surely the main benefit of reserved instances is lower TCO and if you can show you can afford AWS for 3 years a bank would surely loan you the money to pay upfront if you can save say 50% it is simply cheaper even with interest.
For a regional hospital for example, there might be a desire for in-house network and hardware - but perhaps the system for digital patient journals run on Kubernetes and managed databases.
I'd rather have $1 in capex than $10 in opex
Can it run Linux or Windows Data Center as ex-Sun folk they seem to have loved the last open source release before Oracle but now is still community developed variant of SunOS ???.
I would imagine the system architecture is different enough that running Linux as the base OS would take some work, for example it doesn't have an AMIBIOS.
It's not like commodity x86 gear with black-box (often buggy) firmware and layer-upon-layer of hacks and compatibility kludges to present the hardware interface of a late model IBM PC AT.
I can buy these at my local electroncis retailer. So pretty much commodity x86.
This is the type of "PC architecture" cruft many customers have been yearning to ditch for years.
Personally, I have always been annoyed that the BIOS is clunky and every change requires a reboot, taking several minutes. As computers got faster over the years, this has gotten worse, not better. At the core of cloud economics is elasticity: don't pay for a service that you don't use. Wouldn't it be great to power down an idle server, knowing that it can be switched on seconds before you actually need it?
considering you would still need to boot the VMs then, once the Oxide system is up, I’m not sure if this is such a big win.
And at a certain scale you’d probably have something like multiple systems and VMware vMotion or alternatives anyway. So if the ESXi host (for example) takes a while to boot, I wouldn’t care too much.
And, economics of elasticity - you’d still have to buy the Oxide server, even if it’s idle.
To be honoust, I'm using containers most of the time these days but even the full blown windows VMs I'm orchestrating boot in less than 20s, assuming the hypervisor is operational. I think that's about on par with public cloud, no?
> [...] vMotion [...] ESXi.
Is VMware still a thing? Started with virsh, kvm/qemu a decade ago and never looked back.
> And, economics of elasticity - you’d still have to buy the Oxide server, even if it’s idle.
That's a big part of the equation indeed. This is where hyperscalers have an advantage that Oxide at some point in the future might enjoy as well. Interesting to see how much of that they will be willing to share with their customers...
How do you mean Oxide might have that advantage as well in the future? As I understand, you have to buy hardware from them?
As to your question, I meant to say that as volumes and scales economies increase they can source materials far cheaper than regular shops. Possibly similar to AWS, gcs, Azure, akamai etc. It would be nice if they were able and willing to translate some of those scales economies into prices commensurate with comparable public cloud instances.
If you want more insight into all of the things that normally run on "PC architecture" - the 2.5 other kernels/operating systems running underneath the one you think you're running - https://www.youtube.com/watch?v=mUTx61t443A
I imagine they are aware that this isn't a solution for many customers. A John Deere tractor makes a poor minivan. This isn't for you. That's fine. It's not for me either. That's ok. I don't need to poo-poo their efforts and sit and moan about how it's not for me.
The BMC is often a huge problem, its a shifty architecture and extremely unsafe. Meta is paying for u-bmc development to have something a bit better.
Doing things like rack attestation of a whole racks worth of firmware if stacked PC is incredibly hard and so many companies simply don't do it. And doing it with the switch as well is even harder.
Sometimes the firmware runs during operations and takes over your computer causing strange bugs, see SMM. If there is a bug anywhere in that stack, there are 10 layers of closed source vendors that don't care about you.
Costumers don't care if its PC or not, but they do care if the machine is stable, the firmware is bug free and the computer is safe and not infected by viruses. Not being a PC enables that.
Oxide has put immense effort into writing open-source platform initialization code, and built their own open-source BMC/RoT solution.
Not in theory maybe, but in practice. Because as a customer, I would probably also need to put in immense effort to understand and maintain that software myself.
Until you've dealt with a malfunctioning Dell or HP server and have to live with being told "we don't know why it acts that way, we'll try to get the ODM to repro" I don't think you can appreciate how cool Oxide's offering seems.
Which would probably be a lot faster than trying to find someone who could maintain some non-standard firmware (as good as it might be).
Even if I had to replace the server on my own cost it would probably still be cheaper. And it would be easy to replace because it's commodity hardware, that was kind of my point.
Maybe if I had hundreds of servers Dell would have helped me out. At the scale of tens they told me to take what I got. The Customer got a lower performance solution and nobody anywhere could help them for any amount of money, short of replacing the gear.
That's just a performance issue. I've heard horror stories about reliability-- All the way down to disk firmware and RAID controllers. I consider myself lucky.
But how much effort (or money) do you think it would have taken to fix this issue if the NIC firmware was open source?
And with standard hardware, depending on the model, you might have had the option to add dedicated PCIe NICs for example. Not great, but at least something. Now try that with something proprietary (as in non-standard) like this Oxide system.
If they replace the broken component with a working component, then I don’t care how they fix their firmware errors.
If you already get a 50% discount on your on-demand prices for AWS, does buying your own cloud make economical sense?
You're not buying a bag of peanuts and they will likely price it differently if you're buying 1x or 500x.
What is so different about CapEx and a bag of peanuts that defies even providing an approximate range? Number of zeroes, no?
I don’t like it, but it is the norm sadly. It’s just a signal for “it’s expensive and if you even care about the number we don’t want to talk with you”.
It is not, but when it's written publicly you cannot send different prices to different customers, depending on who the customer is
I just hate calling people to determine if we’re going to match, when they could have given me that information up front.
Let's start with their beliefs: 1. Cloud computing is the future of all computing infrastructure.
Yeah, no. There's still gonna be things like mobile phones which will do some stuff locally. And probably also desktops. And probably also on-prem servers.
Which brings us to their belief no. 2: The computer that runs the cloud should be able to be purchased and not merely rented.
If you buy a computer and run virtualized loads on it, then you're not doing cloud computing. As simple as that.
Basically, as others have commented already, this is just something like a blade server system or a hyperconverged system like for example Nutanix is offering. The only real difference seems to be the open source approach.
All text seems to indicate the former while the logo seems to spell the latter?
This 100%!
Not everything needs to be a subscription. Sure there are running costs for operating that cloud computer as well as ongoing software development costs. But having photo albums in the cloud shouldn't cost me $30/month just because of storage - let me buy that hard drive (and maybe compute why not) and pay $10/y for operational costs
How many engineers and how many years does it take to build the infrastructure, write all the software, and deal with all the hardware to provide a photo service that "just works"? How many customers do they need before they break even? And obviously they can't raise VC because a business model that is predicated on making basically no money per customer can never have an exit. How would you bootstrap such a thing?
It feels strange that physical storage costs almost nothing on amazon but the storage attached to a cloud machine is expensive. But run the numbers and you'll see that running a cloud service isn't about hardware costs at all.
> Cloud computing is the future of all computing infrastructure.
Oh god no!
Anywhere I can donate to have this not happen? I want my computing on premise, preferably under my table.
But well, if it _HAS_ to happen, 0xide is probably a lesser evil.
https://oxide.computer/podcasts/on-the-metal
The most recent few episodes have been about corporate open source and they've had excellent guests, like Kelsey Hightower. Definitely the best computer related podcasts out there. Bryan and Adam are great hosts and their humor is always a delight.
Other podcasts I'd recommend: ADSP [1] (if you're into programming), 2.5 admins [2] (if you're into computers). But I have no recommendations about hardware design because AFAIK the podcasts you mention and what Oxide is doing are pretty unique.
[1] Algorithms + Data Structures = Programs https://adspthepodcast.com/
[2] Allan Jude, Jim Salter, Joe Ressington https://2.5admins.com/
The shows at https://www.jupiterbroadcasting.com/ are my personal go-to.
Such as?
I’d say their opinion on “ransomware being a backup problem not a security problem” was enlightening and their backup opinions are also interesting (Push vs. Pull, ZFSes snapshotting, etc.)
Allan’s experience in FreeBSD is welcome too, not every day you get to hear about FreeBSD in prod.
They both skew FOSS too but not in an idealistic way, it’s based on decades of experience.
In a similar vein, Embedded.fm is a great podcast for embedded SW. Though I bet Oxide's take on embedded rust is a lot different than the hosts of that show!
Add few add ons, service contract, support etc so would be there.
I want such a rack but power draw on average is listed around 12kwh. Unbelievable.
Or provide enough white glove after-sales support and written guarantees to peel away low end mainframe customers at a fraction of the price.
Given their management plane/API:
* https://docs.oxide.computer/api/guides/responses
the performance may be about the same, and CapEx as well (or maybe a little higher), but OpEx could be where you make it up in large(r)-scale operations.
And space efficiency is also not to be sneezed at: for some operations DCs/compute can be place anywhere because latency is that big of a deal, but in other places you need to be close to certain things (trading), and real estate can get expensive.
https://oxide.computer/product/specifications
Then yeah, it's ridiculously expensive.
That said compared to competitors it's in the right ballpark, but I have no idea how companies manage to spend so much money for this stuff. I am the founder of my own tech startup and I remember when I was looking at storage solutions and building out computing clusters there were companies charging absolutely insane prices.
I literally just spent about a week of my own time studying and learning as much as I could about it on my own and ended up building out my own custom solution for about 20-25% of the price these other companies were charging. I remember hearing people trying to scare me out of it saying if I did my own solution I'd need to hire full time operations people, and I'd always have to worry about things breaking and maintenance or headaches and nightmares etc...
It's been over 10 years now and absolutely no headaches, no nightmares, and very very minimal maintenance is needed.
Some drills I practiced were setting up RAID 10 and on hardware failure having an email sent to me, so I went through the process of getting RAID 10 working using 4 of the drives, and then I would physically pull a hard drive out of the system as it was running to simulate a failure, and then I swapped in a new hard drive and ensured that the RAID rebuild process took place.
Once I was confident enough, I went to thinkmate.com and bought two JBOD expansion chassis each of which supported 78 drives and filled each of them up with 5 TB Seagate drives to get a total of 390 TB. The JBODs are managed by a 1U server running Ubuntu Server and software RAID using mdadm. I also bought 8 compute servers that were considered high density and could fit in 4U. I got a Cisco router for Internet and I networked everything within the data center together using a used Infiniband switch that I bought off of ebay. I also got a KVM so I could remotely access all of the systems. If there's one thing that I would change today, it would be to use ZFS.
I remember comparing the cost of doing this DIY setup to some other premade solutions like EMC and the cost was astronomical, like 3-4x what I ended up having to pay. I also even remember watching Linus Tech Tips and Level1Techs on Youtube and they both had good content about how they managed their storage that was fairly reasonable but still slightly on the pricey side, nevertheless I learned quite a bit from it.
At any rate, the bottom line is that it's not trivial to learn all this stuff by any means and I remember having some serious frustrations due to just how bad and demoralizing some error messages can be, but it's not thaaaat difficult either and in my situation my company is self funded, no venture capital or outside investors of any kind so every dollar my company spends is a dollar out of my own pocket. You better believe when I'm starting my business I'm not going to just blow 100s of thousands of dollars extra unless I absolutely have to.
About 5 years ago I managed to use my own storage system to ditch Dropbox in favor of Nextcloud, which is just leaps and bounds superior. I remember I got so frustrated with Dropbox because I wanted to just do something as simple as create a sub-directory and grant only read only permissions to some accounts. I also remember wanting to do some simple things like create a fake account with very limited permissions that could be used in our conference room for presentations or demos but the only way to do that with Dropbox would be to create an actual account and have to pay the full price for it.
Nextcloud works amazingly well, has all kinds of cool plugins, and gives our organization a lot of flexibility that Dropbox doesn't and I can make as many accounts as I feel like for whatever reason I feel like.
And not a single mention of backups.
Good luck with that.
I mentioned I use rsync for backups.
So different backups happen in varying directions, with the colocated servers using the main storage system to back themselves up, and the main storage system in turn backing portions of itself onto the colocated servers.
Not everything gets backed up, however, and most of what does get backed up can be compressed quite significantly. We mostly just back things up that can't be recovered from third parties or that are proprietary to our organization.
See pictures: https://pbs.twimg.com/media/FfT7MHoUoAE90QZ?format=jpg&name=...
That strikes me as being in the right ballpark, but it's going to be tough to swallow since that's the lowest level of granularity.
For most orgs you'd be left paying for a lot of excess capacity you couldn't immediately put to use as you migrate workloads in. I guess in ~4 years once you reach steady state and you're retiring / replacing these things it all works out, but if you're migrating from vmware or something else in a traditional blade/chassis world it's not like you can just wave a magic wand and move $500k worth of compute over to this thing at once.
If you're green fielding something, that's a lot of cash to sink in on compute you may not need for some time in the future. Never mind your DR site(s) also needing that much...
Pricing??
Also "Facilities" link in footer: 404
* https://github.com/oxidecomputer
And the debugger is here: https://github.com/oxidecomputer/humility/blob/master/cmd/ri...
I also gave a talk at this year's Open Source Firmware Conference that covers a bunch of debug strategies: https://www.osfc.io/2023/talks/unplugging-the-debugger-live-...
(conference videos aren't online yet, but should be posted early next week)
I design them in Monodraw, pass it through a janky converter I wrote that converts text into a json grid of characters. I then render a number of layers that get combined, which is a mix of the static art layer, and others generated from functions that spit out a similar cell based frame.
If you're interested: https://gist.github.com/benjaminleonard/c913ddbf23fe7a70f9c2...
And for what it's worth there's this ASCII game: https://twitter.com/StoneStoryRPG
So, Humility is like MC/DC?
https://en.wikipedia.org/wiki/Modified_condition/decision_co...
1. The state or quality of being humble; freedom from pride and arrogance; lowliness of mind; a modest estimate of one's own worth; a sense of one's own unworthiness through imperfection and sinfulness; self-abasement; humbleness.
-- Webster's 1918 Dictionary
Our avionics software was originally spec-ed to not NEED a debugger it was to be of such high quality, but the designer added break point op-codes in the VM anyway. I guess he was aware of the concept of humility, too. ;) Like humor, humility seems to be in short supply these days.
Hmm...maybe I should have referenced:
https://en.wikipedia.org/wiki/Sacred_cow_(idiom)
> The motto of the satirical magazine The Realist was "Irreverence is our only sacred cow".
Humor does seem to exist today, it just has a narrow band-pass filter, with apparent focus on "organic" product marketing.
When I see the concept of "extremely high quality" applied to software in the aerospace domain, I tend to think of methods like formal proofs and "semi-formal" methodologies like DO-178C (including MC/DC, not as a tool, but as a critical sub-process). It does appear that Humility adheres to some traditional ideas used in even "safety conscious", real-time embedded work:
> However, Hubris may be more interesting for what it doesn't have. There are no operations for creating or destroying tasks at runtime, no dynamic resource allocation, no driver code running in privileged mode, and no C code in the system. This removes, by construction, a lot of the attack surface normally present in similar systems. [1]
Oxide surely didn't invent all these concepts, we were applying some of this in the early '90s on human-safety-critical embedded projects, and they were most likely used before that. The attack surface eliminated is more than just security, e.g. even self-induced "attacks" from task priority inversion. Maybe the concept of applying them to rack mount servers is novel, but I have no experience there. Techniques like DO-178C go farther and do things like trace requirements to object code and strict coverage criteria.
When you are considering where Hubris/Humility might be applied in aerospace, also note that it would most likely be denied at the proposal stage of talking to a regulatory body like the FAA if you said, "The methodology for our brake controller is cloning a Github repo written by 100X S/W engineers in the Rust language, then modding that to fit." It's a process you have to follow in a documented fashion, from the start. Who knows--I certainly don't know everything--if you do have a identify reproducible process that improves S/W quality, the FAA might be interested.
And as far as eliminating debuggers goes, Boeing did actually do a joint academic/industry project on a zero-bug reduced (subset) Ada compiler (Zbra), which itself would be qualified as a tool instead of the more laborious process of mapping requirements to object code directly. [2]
In the spirit of humor shown on this topic, I would suggest considering a reduced subset of Rust called Iron. :) Regards, and best of luck!
[1] https://oxidecomputer.github.io/hubris/
[2] http://www.sigada.org/conf/sigada2002/SIGAda2002-CDROM/SIGAd...
https://oxidecomputer.github.io/hubris/reference/
* Memory access is protected, but addresses are not virtually mapped. The lack of paging of course ties strongly to the inability to dynamically add tasks.
* They wish they could have read-only shared libraries (viable since there's only one address space) so they could truly be shared, but the current ecosystem assumes mutability is possible (even though in Rust mutable globals are rare).
Can't wait to read its source code, I was curious since reading this thread[2].
[1] https://github.com/oxidecomputer/helios (if this is not found for you, we're in the same boat)
[2] https://news.ycombinator.com/item?id=33337086
I would expect nothing less from that crew. They did amazing things at Sun Microsystems, Inc. (RIP), and they continue to do even greater things now.
https://github.com/oxidecomputer/hubris
And, that one underscore-delimited folder name in this repo just catches the eye, huh?
https://github.com/oxidecomputer/humility
With respect to Humility, I am going to resist the temptation of pointing out why one of those directories has a different nomenclature with respect to its delimiter -- and just leave it at this: if you really want to find some filthy code in Humility, you can do much, much better than that!
[0] https://github.com/oxidecomputer/hubris/commit/651a9546b20ce...
EDIT: "it's not for me" - I'm not working with organizations that have that sort of need directly. Re-reading that phrase, it came across as a bit dismissive of what they've built (which is undoubtedly impressive).
And yeah it's totally chill: this product is not for everyone. No slight taken!
But yes, if multiple teams and they have direct access this rack to provision VMs, maybe yes.
Also - as a developer you don't provision VMs on daily basis. You start a project, you need those resources at the very outset and that's about it?
How is this different than colo space that has been predominant for years? Perhaps simply that it can be purchased more easily and standardized like a typical cloud VM?
I imagine colo’s have something like this, but I _suspect_ operationally it’s a lot more powerful, easy to use and functionally closer to the API’s and behaviour users are used to on current cloud providers.
Now, if someone who actually knows for sure feels like chiming in, I’m happy to be corrected.