HN Explorer

Ask HN: Looking for bare metal hosting company without virtualized NICs

11 points by scrubs ↗ HN
I've been using spot Amazon c5n metal instances to do DPDK NIC I/O development work. About $1.20/hr/machine. Spot instances are awesome for development work, and on the whole, using Amazon is painless.

See https://github.com/rodgarrison/reinvent

However, AWS ENA NICs are virtualized. They are not bonafide plugged-into-the-PCI-bus cards. Therefore the data pathway to/from the NIC is not direct like it would be at home with the NIC right on the motherboard or in the PCI bus. For a cloud company probably virtualized is the only way to go.

Non-virtualized is what I have in mind for the application I have in mind. To that end, anybody know of a decent HW hosting company provides:

- Some kind of Xeon based processors - regular run of the mill COTS NICs (not virtualized) - Bonus: provides or could provide SolarFlare NICs

18 comments

[ 3.5 ms ] story [ 84.7 ms ] thread
wmf
What is it that you actually need? Latency? Jitter?

The "good" news is that most bare metal hosts like Equinix and SoftLayer haven't caught up to Nitro so they use traditional NICs.

And if you can use the OFED API, have you tried the Elastic Fabric Adapter?

scrubs
Mostly looking for low latency.

>The "good" news is that most bare metal hosts ...

Thank you. Will checkout references.

The system I'm working on, for the I/O part, will be a rewrite of: https://www.usenix.org/system/files/nsdi19-kalia.pdf

The idea there is that not so many UDP packets are lost so, with a little help in congestion and packet loss, you can get UDP speed reliably. DPDK, even on a virtualized NIC like AWS, can enqueue UDP packets in ~500ns if the NIC's TXQ isn't full. So compared to, for example, https://blog.cloudflare.com/how-to-receive-a-million-packets... you can get a lot more done in a lot less resources than kernel based work.

But as I say AWS are virtualized NICs. Which is why I was asking for options for bare metal with regular-old-NICs on the PCI bus. So although developing on AWS spot instances is pretty seamless it seems like it defeats the point of using DPDK which I guess would shine even brighter if the virtualized layer was not in the way.

And, no, I have not tried EFA although I've definitely seen it on AWS.

Thank you for time and response.

wmf
If you get this working it would be interesting to compare the latency of AWS vs. something else. It might not be as different as you expect.
fjni
https://metal.equinix.com/ May work for what you need, and they also have spot instances.
nikolqy
Exactly was I was gonna say until I read your response. They’re the barest servers I know of. Oracle cloud or IBM cloud might have what he’s looking for as well.
scrubs
Thank you gentleman. Will check it out.
scrubs
Wanna know what? Wow is my first response!

Amazon's c5n.metals are actually too big for me but it gets most of the virtualization out of the way. And it's about $1.17/hr/box spot. Now there's a ton to like about Amazon: EBS, deployment, snapshots, and a whole bunch of other stuff is 100%. The AMZ linux image + DPDK works and built with no nonsense. Passing around public Amazon AMIs to help people eliminate code builds by just jumping into a ready-to-go-setup is another great upside.

Meanwhile on Equinix I've got a higher CPU frequency on a more gentle size (c3.small.x86) for $.17/hr. It's got two Mellanox NICs and two Intel NICs and although I haven't run any code yet, whatever I can use there is probably fine. I will be darn interested in comparing the two setups.

Thanks.

toast0
I don't know about Oracle, and my IBM Cloud (fomerly SoftLayer) knowledge is a few years out of date now, but IBM cloud has (or had) bare metal servers available by the hour. You've got to pick from a limited menu, so no custom ram size or disk selection like on monthly servers though.
Amfy
how many such machines do you need?
scrubs
2 then 5 when I get into the distributed part of the work.
smackeyacky
Could you get by with second hand physical machines? Something like a HP DL360 G7 has 4 ethernet ports on a single machine, can be bought for $200 or so and has good Linux support. Not sure if the nics count as cots though.
floatinglotus
I cannot imagine trying to do DPDK I/O development in any sort of IaaS. Aren’t you going to have to physically mess around with the cables and Traffic generators all the time? It seems like it would be extremely hard to find one company that offers the NICs you want to test.
scrubs
Thank you for the response but not sure we're on the same page. All I would need is two Xeon boxes with COTS NICs on the same subnet. Having hosted bare metal boxes alleviates me from dealing with cables etc.. As for traffic generators ... I have no need for those. The apps I have in mind will form a distributed system and do their own TX/RX.
egberts1
Been there, done that: remote DPDK development.

  It was for a owner-inserted trading box near a fin-DC (Finance data center).  Has a DPDK NIC.
But it did required your own upfront cost for all the components prior to walk-in.

Learned a bit about data center-specific network protocols remotely.

At the time, AMD was chosen over Intel due to not wanting CPU L2 cache being shared by two CPUs.

rosndo
What if you just look for “dedicated servers”? Pretty much all of them will fit your needs, whereas “bare metal” will only bring up buzzword hosts.

Shops like nforce or FDCservers might fit your needs better than the super overpriced options most people are listing here, and there are hundreds of other options.

andrewf
Disclaimer: I'm at AWS, but not on the EC2 or networking teams, I'm not experienced with their stuff, and they'd be rightly upset if you took me as an authority.

That said, SR-IOV may obviate some or all of your concerns. The hypervisor is bypassed for most purposes, kinda like the way DPDK etc bypass the kernel. Microsoft has a nice diagram at https://docs.microsoft.com/en-us/windows-hardware/drivers/ne...

bobsyouruncle21
Hi there,

You might check out some smaller providers, look more for dedicated servers vs bare metal and that'll get what you need.

I'm not aware of anyone who sells SolarFlare nics directly but maybe check with knownhost.com as they might be able to do something custom for you.

jonathanspw
Jon from KnownHost here - we're always up for custom stuff.

Thanks for the metnion!