For now it's for single GPU inference only.
RTX 3080-10GB should work. You could check https://github.com/facebookincubator/AITemplate/tree/main/ex..., and https://www.reddit.com/r/StableDiffusion/comments/xv7m89/met....
Yes this is correct. batch 16 7.9s / 25 steps, per image 0.49s: it generates 16 images for each prompt within 7.9s, so it's 0.49s per image.
AITemplate only supports fp16 data types with fp16 or fp32 accumulation right now. We are working on supporting more data types and quantization. We don't have an official comparison between AITemplate and tvm / onnx…
We have a bunch of unittests and E2E tests to compare numeric numbers between AITemplate and PyTorch eager.
You could check "AITemplate optimizations" section in the blog (https://ai.facebook.com/blog/gpu-inference-engine-nvidia-amd...), and https://github.com/facebookincubator/AITemplate#more-about-a.... The basic idea is to…
As @haolu7 mentioned, you could take a pre-trained model and use AITemplate to do model inference. All you need to do is to re-write the model using AITemplate frontend and map PyTorch params to AITemplate params.…
tl;dr: Meta is open sourcing AITemplate, an inference engine for both Nvidia and AMD GPUs. Code: https://github.com/facebookincubator/AITemplate. AITemplate delivers much better perf (1.9x ~ 12.8x) compared to PyTorch…
I like HackerNode more because: 1. It contains more columns than only "FrontPage", such as "Jobs" and "Comments"; 2. The UI seems more concise. Although there are not indent between comments, the text is much clearer.…
...............................
For now it's for single GPU inference only.
RTX 3080-10GB should work. You could check https://github.com/facebookincubator/AITemplate/tree/main/ex..., and https://www.reddit.com/r/StableDiffusion/comments/xv7m89/met....
Yes this is correct. batch 16 7.9s / 25 steps, per image 0.49s: it generates 16 images for each prompt within 7.9s, so it's 0.49s per image.
AITemplate only supports fp16 data types with fp16 or fp32 accumulation right now. We are working on supporting more data types and quantization. We don't have an official comparison between AITemplate and tvm / onnx…
We have a bunch of unittests and E2E tests to compare numeric numbers between AITemplate and PyTorch eager.
You could check "AITemplate optimizations" section in the blog (https://ai.facebook.com/blog/gpu-inference-engine-nvidia-amd...), and https://github.com/facebookincubator/AITemplate#more-about-a.... The basic idea is to…
As @haolu7 mentioned, you could take a pre-trained model and use AITemplate to do model inference. All you need to do is to re-write the model using AITemplate frontend and map PyTorch params to AITemplate params.…
tl;dr: Meta is open sourcing AITemplate, an inference engine for both Nvidia and AMD GPUs. Code: https://github.com/facebookincubator/AITemplate. AITemplate delivers much better perf (1.9x ~ 12.8x) compared to PyTorch…
I like HackerNode more because: 1. It contains more columns than only "FrontPage", such as "Jobs" and "Comments"; 2. The UI seems more concise. Although there are not indent between comments, the text is much clearer.…
...............................