Microsoft’s new genAI model to power agents in Windows 11

Microsoft is laying the groundwork for Windows 11 to morph into a genAI-driven OS.

The company on Monday announced a critical AI technology that will make it possible to run generative AI (genAI) agents on Windows without Internet connectivity.

Microsoft’s small language model, called Mu, is designed to respond to natural language queries within the Windows OS, the company said in a blog post Monday. Mu takes advantage of the neural processing units (NPUs) of Copilot PCs, Vivek Pradeep, vice president and distinguished engineer for Windows Applied Sciences, said in the post.

Three chip makers — Intel, AMD and Qualcomm — provide NPUs in Copilot PCs prebuilt with Windows 11.

Mu already powers an agent that handles queries in the Settings menus in a preview version of Windows 11 available to early adopters with Copilot+ PCs. The feature is available in the Windows 11 preview version 26200.5651 that shipped  June 13. 

The model provides a better understanding and context of queries, and “has been designed to operate efficiently, delivering high performance while running locally,” Pradeep wrote.

Microsoft is aggressively pushing genAI features into the core of Windows 11 and Microsoft 365. The company introduced a new developer stack called Windows ML 2.0 last month for developers to make AI features accessible in software applications.

The company is also developing feature- or application-specific AI models for Microsoft 365 applications.

The 330-million parameter Mu model is designed to reduce AI computing cycles so it can run locally on Windows 11 PCs.  Laptops have limited hardware and battery life and need a cloud service for AI.

“This involved adjusting model architecture and parameter shapes to better fit the hardware’s parallelism and memory limits,” Pradeep wrote.

The model also generates high-quality responses with a better understanding of queries. Microsoft fine-tuned a custom Mu model for the Settings menu that could respond to ambiguous user queries on system settings. For example, the model can handle queries that do not specify whether to raise brightness on a main or secondary monitor.

The Mu encoder-decoder model breaks down large queries into a more compact representation of information, which is then used to generate responses. That’s different from large language models (LLMs), which are only decoder models and require all of the text to generate responses.

“By separating the input tokens from output tokens, Mu’s one-time encoding greatly reduces computation and memory overhead,” Pradeep said.

The encoder–decoder approach was significantly faster than LLMs such as Microsoft’s Phi-3.5, which is a decoder-only model. “When comparing Mu to a similarly fine-tuned Phi-3.5-mini, we found that Mu is nearly comparable in performance despite being one-tenth of the size,” Pradeep said.

Those gains are crucial for on-device and real-time applications. “Managing the extensive array of Windows settings posed its own challenges, particularly with overlapping functionalities,” Pradeep said.

The response time was under 500 milliseconds, which aligned with “goals for a responsive and reliable agent in Settings that scaled to hundreds of settings,” Pradeep said.

Microsoft has many genAI technologies that include OpenAI’s ChatGPT and its latest homegrown Phi 4 model, which can generate images, video and text.