The spiraling cost of memory and rising compute costs for training next-gen models have made AI expensive to provide and use. However, running models on your own devices is becoming increasingly popular as a solution, with many smartphones already supporting NPUs for on-device AI processing. Despite this, proprietary APIs and hardware limitations have limited widespread adoption.

Accelerating basic AI workloads on the CPU is an increasingly popular middle ground, targeting the CPU means the tool runs on essentially anything without the development headaches of proprietary drivers and APIs. Newer CPU architectures like Arm's SVE2 and SME2 are designed to support fast math in parallel and matrix execution modes, respectively, significantly reducing memory traffic and increasing throughput for AI workloads.

The recent announcement of AMD and Intel's joint venture to bring AI Compute Extensions (ACE) to future CPUs builds on these advancements. ACE brings native matrix instructions to the x86 ISA, supporting tiny INT4 sub-byte types and conventional BF16 and FP16 data types, making it easier for developers to target CPU-based AI acceleration across phones, laptops, and PCs.

This development is significant for the Edge AI ecosystem, as it will enable more efficient on-device processing and reduce reliance on proprietary GPU or NPU APIs. While existing phones and laptops may not directly benefit from these advancements, future upgrades are likely to incorporate these capabilities, leading to improved performance and user experience.

The integration of CPU matrix execution capabilities into the CPU pipeline, like SME2 and ACE, represents a shift towards architectures that natively support tensor and matrix computation. This will enable faster parallel math processing across vendors on the x86 platform, making life easier for developers looking to bring AI workloads to consumer products.

As CPUs continue to evolve beyond vector SIMD into more capable architectures, they are rapidly becoming far more capable for on-device and low-latency AI inference workloads. The impact of this development will be felt across the Edge AI ecosystem, with potential benefits including improved performance, reduced latency, and increased efficiency.

Source & References