Achieving 1,000X CPU Performance Boost with Sparse Models in MLPerf (neuralmagic.com)

Exciting news! Our latest MLPerf™ Inference v3.0 results showcase a 6X improvement in just six months, catapulting our CPU performance to an astonishing 1,000X increase and reducing power consumption by 92%. Our sparsified BERT NLP model achieves lightning-fast speeds of 5,578 items/sec, outperforming ONNX Runtime CPU and NVIDIA T4 GPU, while our streamlined ResNet-50 model leaves competitors in the dust with 19,632 images/sec for image classification. The key to our success is compound sparsity as a model compression technique, which allowed us to trim ResNet-50 from 90.8MB to 11MB and BERT-Large from 1.2GB to a mere 10MB. Harnessing the power of DeepSparse, our inference runtime specifically designed to accelerate sparse models on x86 and ARM CPUs, we've revolutionized AI performance.