UK Government Trial of M365 Copilot Finds No Clear Productivity Boost

A UK government trial of Microsoft's M365 Copilot found no clear productivity gains despite user satisfaction with tasks like summarizing meetings and writing emails. While the tool sped up some routine work, it actually slowed down more complex tasks like Excel analysis and PowerPoint creation, often producing lower-quality results. The Register reports: The Department for Business and Trade received 1,000 licenses for use between October and December 2024, with the majority of these allocated to volunteers and 30 percent to randomly selected participants. Some 300 of these people consented to their data being analyzed. An evaluation of time savings, quality assurance, and productivity was then calculated in the assessment (PDF). Overall, 72 percent of users were satisfied or very satisfied with their digital assistant and voiced disappointment when the test ended. However, the reality of productivity gains was more nuanced than Microsoft's marketing materials might suggest. Around two-thirds of the employees in the trial used M365 at least once a week, and 30 percent used it at least once a day -- which doesn't sound like great value for money.

According to the M365 Copilot monitoring dashboard made available in the trial, an average of 72 M365 Copilot actions were taken per user. 'Based on there being 63 working days during the pilot, this is an average of 1.14 M365 Copilot actions taken per user per day,' the study says. Word, Teams, and Outlook were the most used, and Loop and OneNote usage rates were described as 'very low,' less than 1 percent and 3 percent per day, respectively. 'PowerPoint and Excel were slightly more popular; both experienced peak activity of 7 percent of license holders using M365 Copilot in a single day within those applications,' the study states. The three most popular tasks involved transcribing or summarizing a meeting, writing an email, and summarizing written comms. These also had the highest satisfaction levels, we're told.

Participants were asked to record the time taken for each task with M365 Copilot compared to colleagues not involved in the trial. The assessment report adds: 'Observed task sessions showed that M365 Copilot users produced summaries of reports and wrote emails faster and to a higher quality and accuracy than non-users. Time savings observed for writing emails were extremely small. 'However, M365 Copilot users completed Excel data analysis more slowly and to a worse quality and accuracy than non-users, conflicting time savings reported in the diary study for data analysis. PowerPoint slides [were] over 7 minutes faster on average, but to a worse quality and accuracy than non-users.' This means corrective action was required.

A cross-section of participants was asked questions in an interview -- qualitative findings -- and they claimed routine admin tasks could be carried out with greater efficiency with M365 Copilot, letting them 'redirect time towards tasks seen as more strategic or of higher value, while others reported using these time savings to attend training sessions or take a lunchtime walk.' Nevertheless, M365 Copilot did not necessarily make them more productive, the assessment found. This is something Microsoft has worked on with customers to quantify the benefits and justify the greater expense of a license for M365 Copilot.

Read more of this story at Slashdot.