We first started hearing about AI-enabled PCs (AIPCs) from Microsoft. As a platform, PCs may seem a mature and unpromising market, but add AI and some amazing things can happen. Quickly summarize a PowerPoint slide deck or a 30-page Word document. Automatically create a complex filter for an Excel table to then create an equally complex trend chart. All without needing to first spend hours researching methods and plodding through lengthy YouTube videos. This is what Microsoft suggests: an assistant to do all the research for us and to generate a condensed summary or the slides, macros or charts we need. That would be a real advance in personal productivity. But such a capability must be more application-aware than a general purpose chatbot and it must preserve privacy and security, all at very low power. Cloud-based AI is not an option; the AI hardware and software must sit on the PC. Which explains why PC-based AI hardware is growing fast.
Opportunity
The world is still waiting for proven productivity advantages from AI in broad business applications, but in AIPC development is moving much faster. Microsoft’s enthusiastic support for this direction, especially in their latest Surface Pro release building on their CoPilot AI platform is evidence of their commitment. Product releases have already been announced by Dell, HP, Acer, and Asus, and Foxconn is already offering manufacturing support. Qualcomm, Intel, AMD have released or announced AIPC SoCs. None of the big players wants to be late in this race.
This is not surprising. Needham estimates that shipments of AIPCs are expected to grow from 50 million units in 2024 to more than 167 million units by 2027. All for a product class most of us hadn’t known existed until this year! One analyst anticipates initial growth will be driven by high-end business users and content creators, encouraged by intelligent assistant advantages and GenAI image creation. It is not hard to imagine that as these platforms take off more users will want access to similar AI-enhanced capabilities.
Technical demands on AIPC platforms
Clearly an AIPC must be supported by a dedicated AI accelerator. Early generations of these systems run on NPUs offering up to around 40 TOPs in performance. The Needham report suggests >60 TOPs for more advanced AIPCs though they say none are apparent yet. To offer the class of GenAI services needed, these accelerators will be expected to run models with billions of parameters, think Llama2/3-class LLMs for example. Performance must be real-time responsive. Local processing without need to go to the cloud is a plus but the performance offered by general purpose NPUs today is not adequate to provide real-time response in high-end transformer networks where today autonomous driving systems run at 300 TOPs or higher. That’s a different application of course but the networks an AIPC accelerator may run, with speech recognition, prompt processing, and inference/generation will be no less complex.
Memory requirements must be compatible with current expectations for a PC, say tens but certainly not hundreds of GB for a high-end system. Remember that much of that memory is already committed to conventional storage needs (images, videos, other large data files). To host models requiring billions of parameters accelerators must support very aggressive parameter compression, on-the-fly-decompression, and sparsity handling together with off-chip flash backing store behind the DDR memory.
AI is a notorious power hog in datacenters; in laptop applications it must be much more economical with power. Model quantization, parameter compression and robust sparsity handling are essential to meet this goal, as are standard SoC power management techniques: fine-tuned clock gating, DVFS and power islands. Minimizing need for power-hungry internet (cloud) communication is critical, except where absolutely essential (in RAG retrieval for example) is a critical component in reducing power demand.
Differentiation
An important consideration is support for AI model updates, which immediately points out a difference from general purpose GenAI systems in the cloud. AIPCs cannot and will not offer access to an arbitrary range of models; the overhead in downloading new models on-demand would simply be too high. Instead, they will likely support their own pre-selected and limited set of models. These can still be updated in the same way you update to new OS releases. This strategy equally implies that the accelerators need not be user accessible as general-purpose accelerators, which also eliminates need for all the corresponding support infrastructure (SDKs etc.).
In what is likely to be a fast-moving market, hardware differentiation will also be very important factor for PC builders. Building on top of an off-the-shelf platform, no matter how capable, will not offer the advantages that they can exploit in a proprietary platform. Then again, building and maintaining the AI hardware expertise and infrastructure needed to deliver such a platform is a major investment. These builders are most likely work with established embedded AI NPU core providers to help them bring their products to market quickly. They can then focus on the models they want to support, and how these interact with their software and update infrastructure.
Another interesting opportunity to differentiate is in personalization. An assistant can learn over time to tailor responses and services to best meet my most common needs, which will inevitably be different from your most common needs. Managing this experience well can become a major factor in customer satisfaction, recommendations and repeat business.
Takeaways
The AIPC has the potential to be a killer app, redefining how we envision and use our PCs. Making this happen will depend on strong and differentiated AI accelerator solutions inside those PCs, tuned to provide a customer experience which will meet most of their screen time needs without needing to be a general purpose GenAI solution and without needing to go to the cloud for most purposes. At Ceva we’re already engaged with a major PC provider on helping build their next generation product. Give us a call or click on this page.