Launch HN: Tensil (YC S19) – Open-Source ML Accelerators

Great question - TVM / OctoML are a great option if you have an off-the-shelf ML model and off-the-shelf hardware. Tensil is different in that you can actually customize the accelerator hardware itself, allowing you to get the best trade-off of performance / accuracy / power usage / cost given your particular ML workload. This is especially useful if you want to avoid degrading the accuracy of your models (e.g. through quantization) to achieve performance targets.

It depends on the model, yes. Here are some examples in the benchmarks section of our docs: https://www.tensil.ai/docs/reference/benchmarks/

We haven't specifically tested on any ICE40 FPGAs yet - if this is something that you'd really like to see, let me know! Taking a look at the lineup, the ICE40 LP8K and LP4K would be suitable for running a very small version of the Tensil accelerator. You'd want to run a small model in order to get reasonable performance.

Generally speaking, FPGAs with some kind of DSP (digital signal processing) capability will work best, since they can most efficiently implement the multiply-accumulate operations needed.

Thank you for the kind words! Just to clarify, the core technology here is free and open source, anyone can use it right now for free. We do have commercialization plans in addition - we may explore things like additional paid features for enterprise use or paid tiers of extra support.

Regarding LSTMs, yes. We're aiming to support all machine learning model architectures: do you have any particular models you're interested in that we should be prototyping with?

For CGRAs, we don't have any immediate plans to explicitly support them. What kind of use case do you have in mind? Generally, any platform that can implement a blob of generated RTL should be something we can work with quite easily.

Just saw your edit re: model compression. One thing that Tensil can do is help you avoid the need to quantize or compress your model entirely! For example, we've found that using a 16-bit fixed point numeric data type preserves almost all the model accuracy while not sacrificing performance thanks to the huge amount of parallelism available on FPGA.

The broader point is that Tensil is extremely flexible, so you can try out lots of different accelerator configurations to find the one that works best for your ML model. Think of it as optimizing the hardware first, then the software if needed.

We're actually working on a tool to manage and automate this hardware architecture search - watch this space!

Thanks! The general answer is that it depends on your model and on which FPGA platform we're talking about, but in a head-to-head benchmark test you'll find results in the ballpark of 2-10x CPU and 0.5-2x GPU. As you point out, the power and cost are big differentiators. The other thing to consider is (as another commenter mentioned) that usually inference on CPU or GPU will require you to do some model quantization or compression, which can degrade model accuracy. Tensil can give you a way around that dilemma, so that you can have great performance without sacrificing accuracy.

Cheers, and great question! FPGAs are pretty amazing devices, but one thing that's been holding them back is how difficult they have been to work with. Typically to actually make use of an FPGA you'd need to have an FPGA expert and an embedded software engineer on your team, along with all the requisite tools and materials.

That has started to change dramatically in the last decade, with open source FPGA toolchains like yosys, runtimes like the PYNQ framework and RTL generator tools like Tensil being developed. When you put these things together, working with FPGAs starts to become as easy as using any other compute platform. For that reason, I think there are lots of applications involving FPGAs that will soon be invented to take advantage of this trend. One could speculate that the reason Intel and AMD are buying up FPGA vendors is because they see the potential there.

As far as head-to-head comparisons go, as long as you're running the workload it was designed for in the environment it was designed for, an ASIC will always be the best possible perf per watt. The question is what happens when you go outside those bounds. Can you take your model, swap out a layer, and have it run just as fast on your Coral or NPU? Probably not, at least right now. But with Tensil, you can re-run your architecture search to find the best accelerator, and take advantage of it right away.

FINN is a very cool project, but usually requires big changes to your model in order to work, e.g. quantizing down to 1 or 2 bit weights. It also works best on large FPGAs which are unsuitable for edge deployments. Tensil works out of the box with any model (no need to quantize / compress) and on small edge FPGAs.

(This comment was originally posted at https://news.ycombinator.com/item?id=30615605, where the question made more sense, but I've moved it into the new thread because it's interesting.)

We're working on our roadmap right now and prioritizing support based on user interest. If there's a particular model or set of models you're interested in accelerating, I'd love to hear about it!

If there's a lot of interest in transformers, we'd aim to offer support in the next couple of months.

In our current demos, the Tensil logic talks to the host through a couple of AXI and AXI Stream interfaces. There are AXI adapters for many other protocols, including PCIe, that should be able to support many different kinds of connectivity. Here's a link to our docs explaining the host<->Tensil connection: https://www.tensil.ai/docs/howto/integrate/#2-connect-the-ax...

Coral is a great project, especially if you are using a completely vanilla off-the-shelf model. However if you've ever tried compiling a custom ML model for it, you know how finicky it can be. There are lots of ways that you can accidentally make it impossible for Coral to run your model, and it can be difficult to figure out what went wrong.

With Tensil, you circumvent that problem by changing the hardware to make it work for your model. If you have made modifications to an off-the-shelf model or have trained your own one from scratch, it might be a better option from the point of view of ease-of-use and even performance.

Apart from the discontinued Intel sticks, are there pytorch compatible USB accelerators on the market?

Glad this helped clarify things for you! The tricky thing about benchmarks is that one of the key benefits of Tensil is the flexibility to find a trade-off between performance, accuracy, cost and power usage that works for you. Benchmarks that only consider performance or performance per watt can be a bit narrow from that point of view. That said, this is a good idea and we'll add some comparisons that we think make sense to the docs!

One more thing to keep a look on in the NN accelerator world is TensTorrent. That thing looks amazing, but it's mostly for datacenter and 'heavy' edge (pcie board, at least 75W, so to measure against Alveo U50/U55 and Tesla T4/A30 and up to 300W so A40/A100).

Generally the comparison between Tensil and any fixed ASIC is going to run along similar lines, which we explain in this comment regarding the Coral accelerator: https://news.ycombinator.com/item?id=30643520#30645318

The big difference is that while those fixed ASICs offer great performance on the set of models they were optimized for, there can be big limitations on their ability to implement other more custom models efficiently. Tensil offers the flexibility to solve that problem.

Definitely! You can see some of our benchmarks here, and we'll be expanding this list soon https://www.tensil.ai/docs/reference/benchmarks/

I replied to your other comment here about commercialization https://news.ycombinator.com/item?id=30652150 I hope it was helpful!

I saw somewhere that 95% of all ML inference tasks is still done on CPU

There are four big categories of ML accelerators. You already familiar with CPUs and GPUs; then there are FPGAs, which offer better performance and efficiency while remaining flexible. Finally there are ASICs (of which the TPU is an example), which offer the best performance and efficiency but retain very little flexibility, meaning if your ML model doesn't work well on an ASIC then your only option is to change your model.

We chose to focus on FPGAs first because with them we can maximize the usefulness of Tensil's flexibility. For example, if you want to change your Tensil architecture, you just re-run the tools and reprogram the FPGA. This wouldn't be possible with an ASIC. That said, we'll be looking for opportunities to offer an ASIC version of our flow so that we can bring that option online for more users.

If you have a device with known performance in mind, you can compare against our benchmarks listed here https://www.tensil.ai/docs/reference/benchmarks/

We'll be expanding this list and adding more comparisons to other platforms in the near future.

Awesome, I think your use case would make a lot of for Tensil. Looking forward to chatting more!

The core technology will always be free and open source, so to commercialize Tensil we're planning to offer a "pro" version which would operate under a paid license and provide features specifically needed by enterprise users. We're also working on a web service that will let you run Tensil's tools in a hosted fashion, with one major feature being the ability to search across a large number of potential architectures to find the best FPGA for your needs. Extra paid support and other services will also be in the mix.

Great questions! With Tensil, all computations are performed on the FPGA. In addition to matrix multiplication Tensil supports SIMD instruction with various operations. The ML activations, average and maximum pooling, normalization, and image resizing use SIMD instruction. Some ML operations, such as padding, are achieved by changing the memory layout. Tensil uses DRAM0 and DRAM1 memory pools (usually in DDR memory) to interact with the host to read model inputs and weights and write outputs. It also uses these pools to offload intermediate results between layers and between tiles within a layer when FPGA does not have sufficient BRAM, which is common on lower-end devices. Tensil compiler takes care of finding the most efficient memory scheduling for given on-FPGA memory size.