Having recently tried to read a lot of old-ish research (10 years maybe, that's old for malware research), I really appreciate the value of the Wayback machine / Archive.org team. Donated €15 at some point just to feel a little better and will decide on another donation later. Was about to post the same link!
Wayback Machine is so critical it should get public funding. It may already but stuff should be in the Congressional Budget. Leave control of it where it is, though.
I saw the page via wayback machine since the site was down at the time. Seems like a cool academic project. I've been getting into FPGAs personally, with this exact application in mind.
I've been really stymied by how to process packets in hardware. The obvious approach seems to be to run an RTOS or even full-fledged linux if your FPGA has hard IP cores on it. But is there a better way? How much performance would one lose? I'm also a bit confused about how to communicate on PCI-Express (I'm a software guy ... so learning about DMA). I have seen soft IP for TCP/IP stacks but it seems too crazy. I'm doing this as a hobby education project btw. It has been great fun so far! Wish there were meetups on this topic.
Generally speaking, FPGAs are for building hardware pipelines with fixed throughput and finite state machines to control things. If your first thought is "Linux", you're almost certainly better off with an off the shelf SoC. If you have a well understood problem (say, filtering packets based on a handful of patterns), you would design a filtering pipeline that would connect between two Ethernet MACs, applying the filtering at full line rate in between. Actually doing tcp/ip involves tracking a bunch of state and allocating very finite resources, but might not be necessary depending on your goal.
The "connecting two ethernet MACs [with] filtering" seems like an interesting project. I guess a "simple" (and possibly useless) idea could be to do filtering based on MAC addresses in the enthernet frame. Is there open IP for this? I guess I'm asking how a hobbyist gets into this :-) If anyone knows of an existing project, I'd appreciate a pointer.
It is definitely possible, me and a few other students did it in a university course. Assuming the FPGA at least has a media access controller (translates bits to analogue signals) it is not to hard to encode/decode ethernet frames.
Sadly, on the top of my head I do not know of any open IP.
We wrote our own, but it is closed.
I don't know about open IP, but it would be fairly straightforward but not trivial. Knowing verilog would be required, though, but it can be learned from a digital design book [1]. From there, you would probably want some experience in using the tools. For example, for Xilinx FPGAs, you could run through the Vivado tutorials. Then to design the IP you would design the state machines and logic to take data from one ethernet controller (for example, the Xilinx ethernet MAC [2]) and send it to another. The actual verilog would not be especially complex, however interfacing with the ethernet controllers and other peripherals would take some time.
Add MACsec while you're at it to immunize you against attacks involving taps or a subset of compromised Ethernet NIC's. Well, a subset of it maybe as it got complicated like some other standards.
In actual fact for Xilinx based FPGA's this is quite straightforward. An example for 10G Base R:
You can get Xilinx's component for their pma/pcs for 10g base-r ethernet for free from vivado and stick one of the macs from open cores on the end of it (probably this: http://opencores.org/project,xge_ll_mac - I used it for prototyping and it seems to work (before creating my own pcs/pma block and mac to cut down the latency)
Once you have that, then you would need to deal with the ethernet frames streaming through the FPGA, probably 64-bits at a time at 156MHz for 10G, so you need to pull out the fields you are interested in (like mac addresses, ip addresses, etc). You can buffer the incoming packet into a FIFO whilst waiting for the stuff you want to filter on. Once you have all your fields you can decide whether you want to pass the packet through to the tx side or not (I usually read the packet out of the FIFO either way and just hold the valid low for packets I don't want to send).
Regarding PCIe - assuming you're using a hard IP MAC/PHY in your fpga, it's a packed-based protocol that basically only has three packets - read requests, write requests and completions. Requests have an address and a length, and target physical memory. Completions have the data and a matching tag from the read request. A dma transaction is just when the endpoint generates the request instead of the host CPU, but otherwise it's basically identical.
Dont be fooled by the appearance of VHDL/Verilog. you are building hardware with them. So writing a TCP/IP stack in VHDL would be almost like physically building a TCP/IP stack with physical logic gate IC's.
I looked into this a few years ago. If you can settle for just sending bytes on an ethernet cable, you can relatively easily TX/RX using UDP Multicast. The FPGA can beam bits straight to the PHY adapter and out they go.
Actually, not quite, not with FPGA. Sure, you can use an HDL to synthesize a custom microchip, in which case you indeed would be "building hardware", but you can also target an FPGA specifically, in which case you would essentially be programming a kind of a computer in a way similar to how it was done on those (early) switchboard-based computers that were configured by manually making wired connections between a fixed set of logical and arithmetical devices. You could say that one would be "building hardware" in such case as well, but since the system could be easily reconfigured at any time so that it could perform a different function, to me it looks more like programming... (I guess, the right way to look at this is as something where the distinction between hardware and software gets blurred to the point that any serious argument about the meaning of words becomes, well, meaningless.)
"encoding of control flow" (program, automaton, turing machines) vs. "encoding of data flow" (PAL configurations, CPLD, FPGA bitstreams, analog computer configuration)
The early computers (for the most part) were von Neumann architecture machines just like modern computers, with an instruction pointer, registers, branching, and so forth that most people associate with conventional software development.
FPGAs are a undifferentiated sea of configurable logic gates with configurable connections between them[0]; it's not quite wiring together bare transistors but it's not that far removed either. None of the elements of a von Neumann architecture are there; if a engineer wants any of those things in their design, they would have to assemble it themselves from the available gates[1]. So, no, building a design for a FPGA has little to do with programming in the sense that most people mean it.
[0] With various special function blocks interspersed at regular intervals depending on the manufacturer and model of the FPGA
> run an RTOS or even full-fledged linux if your FPGA has hard IP cores on it
Letting the OS touch more than a small percentage of the packets would be a gross waste of resources. If you want to write a program to process packets, use a real processor. (Putting an FPGA in a computer and then running a soft core on the FPGA is even sillier for performance.)
The main reason to use an FPGA for this is to build structures that effectively encode your firewall or other layer processing rules in tables in the hardware, so the hardware can make a decision without having to consult the processor. MPLS and VLAN routing is the obvious case, but you can usually keep a short IP routing table in there too. Ideally you'd be able to make this decision early on while recieving a packet so you can start transmitting it while the last bits are still coming in. What you want to avoid is buffering ("bufferbloat").
> What you want to avoid is buffering ("bufferbloat").
Excess buffering is a problem, but a little bit is fine unless you're trying to do high frequency trading. If you don't move packets to an external DRAM, you're in no danger of having bloated buffers with just the memory on the FPGA.
Or you could implement an active queue management algorithm to keep your buffers from inducing too much unnecessary latency even when they are generously sized. There are a bunch of AQMs out there to pick from, of varying complexity. And it doesn't seem like there's near enough research into hardware implementations suitable for use in switches or network co-processors.
There was an article at Ars [1] that explained how Microsoft uses FPGA powered network cards at Azure.
The idea is to move "network decision making" (load balancing; maybe firewalls and other assorted stuff) from the CPU to the FPGA, where this kind of thing can be done faster, and more reliably (e.g. you can have hard timing constraints on a FPGA).
Guess the other players in this field are pure software switches, which can be very flexible, but sometimes slow. And ASICs, which are very fast (potentially faster than FPGAs) but not as flexible.
I always thought the "best-practice" architecture for these sort of high-speed SDN switches was to have a custom ASIC "data plane" (hard realtime) that can be signalled from a CPU "control plane" (running something soft realtime like Erlang.)
Would an FPGA really give advantages over such a setup? I can't imagine it'd beat the CPU at being a control plane—so it'd mostly have to be that it has a lower TCO as a data plane than an ASIC. That might be true, if the code in the data-plane would ever need updates (which, for ASICs, mean new hardware revs.)
Yeah, it would have an advantage: you could deploy it without spending hundreds of thousands of dollars on ASIC production or prototyping. It's the main use for FPGA's. They're better so long as unit price is cheaper than masks and such for ASIC's.
In this case, each customer would have a relatively small number of the switches where volume doesn't justify an ASIC. Might be enough switches & customers to justify licensing a FPGA core with development cost spread out among many customers.
Alternatively you can also re-configure the FPGA for specific loads where ASIC is always fixed. Downside is of course lower speed and higher power consumption.
I shouldve mentioned that. Tks for catch. Reconfiguration is huge selling poinf for them. Gone from HTTP 1 to 1.1 or 2.0? Just upload the new bitstream. Got a better algorithm or bug fix? Upload new bitstream.
ASIC folks be like, "You gonna neef to bring a soldering iron for that update. Or buy our latest model." Haha.
At Azure scale, ASICs would definitely be cheaper but they're still using FPGAs for the flexibility. OTOH AWS is using ASICs so we can see that different clouds have made different tradeoffs.
Probably. Azure scale is a rare example, though. Most of the time a company will be much lower volume. They might consider S-ASIC's like at eASIC before ASIC's. Prototyping cost at eASIC can be as cheap as $50,000 per some writeups thanks to the eBeam machine. With low costs, they're easier to swap out than full on ASIC's too.
I wondered if an FPGA approach would be worth it now that Intel has their DPDK approach.
Searched a bit, and found a terrific comparison of doing the same work three different ways. It compares serving up a Key/Value store using traditional software, then DPDK, then an FPGA based approach: http://www.hoti.org/hoti23/slides/lockwood.pdf
35 comments
[ 2.6 ms ] story [ 78.9 ms ] threadhttp://web.archive.org/web/20160604121207/https://www.libero...
I've been really stymied by how to process packets in hardware. The obvious approach seems to be to run an RTOS or even full-fledged linux if your FPGA has hard IP cores on it. But is there a better way? How much performance would one lose? I'm also a bit confused about how to communicate on PCI-Express (I'm a software guy ... so learning about DMA). I have seen soft IP for TCP/IP stacks but it seems too crazy. I'm doing this as a hobby education project btw. It has been great fun so far! Wish there were meetups on this topic.
[1] Such as https://www.amazon.com/Logic-Design-Verification-SystemVeril... [2] https://www.xilinx.com/support/documentation/ip_documentatio...
https://www.juniper.net/documentation/en_US/junos15.1/topics...
You can get Xilinx's component for their pma/pcs for 10g base-r ethernet for free from vivado and stick one of the macs from open cores on the end of it (probably this: http://opencores.org/project,xge_ll_mac - I used it for prototyping and it seems to work (before creating my own pcs/pma block and mac to cut down the latency)
Once you have that, then you would need to deal with the ethernet frames streaming through the FPGA, probably 64-bits at a time at 156MHz for 10G, so you need to pull out the fields you are interested in (like mac addresses, ip addresses, etc). You can buffer the incoming packet into a FIFO whilst waiting for the stuff you want to filter on. Once you have all your fields you can decide whether you want to pass the packet through to the tx side or not (I usually read the packet out of the FIFO either way and just hold the valid low for packets I don't want to send).
Hope this makes sense!
I looked into this a few years ago. If you can settle for just sending bytes on an ethernet cable, you can relatively easily TX/RX using UDP Multicast. The FPGA can beam bits straight to the PHY adapter and out they go.
Actually, not quite, not with FPGA. Sure, you can use an HDL to synthesize a custom microchip, in which case you indeed would be "building hardware", but you can also target an FPGA specifically, in which case you would essentially be programming a kind of a computer in a way similar to how it was done on those (early) switchboard-based computers that were configured by manually making wired connections between a fixed set of logical and arithmetical devices. You could say that one would be "building hardware" in such case as well, but since the system could be easily reconfigured at any time so that it could perform a different function, to me it looks more like programming... (I guess, the right way to look at this is as something where the distinction between hardware and software gets blurred to the point that any serious argument about the meaning of words becomes, well, meaningless.)
FPGAs are a undifferentiated sea of configurable logic gates with configurable connections between them[0]; it's not quite wiring together bare transistors but it's not that far removed either. None of the elements of a von Neumann architecture are there; if a engineer wants any of those things in their design, they would have to assemble it themselves from the available gates[1]. So, no, building a design for a FPGA has little to do with programming in the sense that most people mean it.
[0] With various special function blocks interspersed at regular intervals depending on the manufacturer and model of the FPGA
[1] or buy a pre-made soft-IP core
No disagreement here, but see, e.g., http://fpgacenter.com/fpga/fpga_prg.php.
Letting the OS touch more than a small percentage of the packets would be a gross waste of resources. If you want to write a program to process packets, use a real processor. (Putting an FPGA in a computer and then running a soft core on the FPGA is even sillier for performance.)
The main reason to use an FPGA for this is to build structures that effectively encode your firewall or other layer processing rules in tables in the hardware, so the hardware can make a decision without having to consult the processor. MPLS and VLAN routing is the obvious case, but you can usually keep a short IP routing table in there too. Ideally you'd be able to make this decision early on while recieving a packet so you can start transmitting it while the last bits are still coming in. What you want to avoid is buffering ("bufferbloat").
Excess buffering is a problem, but a little bit is fine unless you're trying to do high frequency trading. If you don't move packets to an external DRAM, you're in no danger of having bloated buffers with just the memory on the FPGA.
Or you could implement an active queue management algorithm to keep your buffers from inducing too much unnecessary latency even when they are generously sized. There are a bunch of AQMs out there to pick from, of varying complexity. And it doesn't seem like there's near enough research into hardware implementations suitable for use in switches or network co-processors.
The idea is to move "network decision making" (load balancing; maybe firewalls and other assorted stuff) from the CPU to the FPGA, where this kind of thing can be done faster, and more reliably (e.g. you can have hard timing constraints on a FPGA).
Guess the other players in this field are pure software switches, which can be very flexible, but sometimes slow. And ASICs, which are very fast (potentially faster than FPGAs) but not as flexible.
[1] - http://arstechnica.co.uk/information-technology/2016/09/micr...
Would an FPGA really give advantages over such a setup? I can't imagine it'd beat the CPU at being a control plane—so it'd mostly have to be that it has a lower TCO as a data plane than an ASIC. That might be true, if the code in the data-plane would ever need updates (which, for ASICs, mean new hardware revs.)
In this case, each customer would have a relatively small number of the switches where volume doesn't justify an ASIC. Might be enough switches & customers to justify licensing a FPGA core with development cost spread out among many customers.
ASIC folks be like, "You gonna neef to bring a soldering iron for that update. Or buy our latest model." Haha.
Searched a bit, and found a terrific comparison of doing the same work three different ways. It compares serving up a Key/Value store using traditional software, then DPDK, then an FPGA based approach: http://www.hoti.org/hoti23/slides/lockwood.pdf
Skip to slide 22 if you're impatient.
Custom silicon, rather than the FPGA or ASICS they were using when they started the project in 2012.
They tout not only the freed CPU resources, but lower latencies and increased security of isolating the networking from the CPUs running hosted VMs.
https://youtu.be/AyOAjFNPAbA?t=1822