Three months after turning on Copilot for 40 people, the CEO asks for a number. The admin center report says 31 users are "active". IT says people "seem happy". Finance asks how many hours were saved, and nobody can answer, because nobody measured how long those same tasks took before the pilot.
Measuring AI productivity isn't hard, but it requires something fairly dull before you switch on any licenses: a baseline. Without that starting point, any figure that shows up later is just an opinion with decimals. Here is a practical way to do it: which metrics to use, which reports Microsoft gives you to track Copilot adoption, and what the public studies that bothered to measure properly actually found.
Why productivity numbers are so often hype
The most common mistake is trusting what people believe they saved. Nobody is lying on purpose. We are just bad at estimating our own time, and worse with a new tool we're excited about.
In July 2025, METR published a controlled study with experienced developers: with AI tools, tasks took them 19% longer. Afterwards, those same developers estimated that AI had made them 20% faster. It's a study about programming, not Microsoft 365, but it shows how far perception can drift from measurement.
The UK government's Microsoft 365 Copilot trials show the same pattern in office work:
| Study | How it measured | Result |
|---|---|---|
| Cross-government experiment (GDS), 20,000 civil servants, published June 2025 | Self-reported survey | 26 minutes saved per day on average |
| DWP evaluation, 3,549 staff | Treatment group vs. comparison group, with statistical analysis | 19 minutes per day across eight routine tasks |
| HMRC evaluation, 3,000 randomly assigned licenses, published July 2026 | Self-reports adjusted down by about 20% | 2% to 3% of the working week, around 60 minutes a week |
All three found time savings. What changes is the size: the stricter the method, the smaller the number. HMRC even trimmed its own figures, because the people who answer surveys tend to be the keenest users. We like that attitude, and it's the one we recommend copying.
The four metrics that actually work
For a pilot in a mid-sized company, four metrics are enough. With more, the spreadsheet gets abandoned by week three.
- Time per task. Pick five to eight concrete, repetitive tasks (the weekly sales report, committee minutes, a standard tender response) and measure how long they take today. This is the core metric.
- Real adoption. "Used Copilot at least once" isn't enough. You want to know how many people use it week after week, and in which apps.
- Output quality. If the report takes half the time but comes back twice with corrections, nothing was gained. Count revisions, returns or errors caught.
- User perception. Useful, but as a complement. A short survey at the start and at the end, with the same questions.
How to set a baseline before the pilot
It's the step almost everyone skips. It takes two or three weeks, and it has to happen before you assign licenses, not after.
- Define the pilot group and, if you can, a comparison group with similar roles that keeps working without a license over the same period. That's what DWP and HMRC did, and it's what let them separate Copilot's effect from everything else.
- For each chosen task, log the time of at least four or five repetitions. A shared spreadsheet with date, task, person and minutes is enough. You don't need special software.
- Write down how quality is measured for that task: how many versions a document goes through, how many comments the manager makes, how many errors get fixed afterwards.
- Run the initial survey: how much time people feel they lose searching for information, on email and in meetings.
Then, during the pilot, repeat exactly the same measurement. Same tasks, same spreadsheet, same questions.
What Microsoft gives you to measure Copilot adoption
Microsoft has two data sources worth knowing, because they measure different things.
The first is the set of reports in the Microsoft 365 admin center. According to the Copilot usage report documentation, it shows enabled users, active users (those who used any Copilot feature in the period), the active user rate, prompts submitted and adoption by app, over 7, 28, 90 or 180 days. There's a separate report for Copilot Chat that covers users without a license. For the baseline, the readiness report is also useful, since it shows which users meet the technical requirements.
The second is the Copilot Dashboard in Viva Insights. According to Microsoft Learn, it's available to Microsoft 365 business and enterprise customers with Exchange Online and doesn't require a paid Viva Insights license. It organizes data into readiness, adoption, impact and sentiment. Two useful details:
- It classifies users by how consistently they used Copilot over the last 12 weeks. A "habitual" user, for example, is someone who used Copilot in at least 9 of those 12 weeks. That's a more honest metric than "active users".
- Tenants with at least 50 Copilot licenses (or Viva Insights licenses) also get benchmarks, group-level data and built-in surveys.
Be careful with the "assisted hours" the dashboard calculates. They're estimates: Microsoft multiplies Copilot actions by a time-per-action factor (for example, 6 minutes per creation action, based on a study of 147 people), and the documentation itself warns that Copilot is probably not the only cause of the changes it shows. They're good for spotting trends. To defend a budget in front of the board, use the minutes you measured yourself.
How to read the results without fooling yourself
When the pilot ends, compare task by task. A five-column table does the job: task, minutes before, minutes after, change in quality and number of repetitions measured.
A few rules that help:
- Savings aren't even. The UK studies found the biggest gains in drafting, summarizing and searching for information, and clear limitations in complex or data-heavy work. Expect the same, and don't average everything into a single number.
- A few minutes a day, spread across different moments, won't free up a full person. What does change is the team's capacity to do more with the same headcount, and that's how you should present it.
- Separate consistent users from the rest. If the savings come from a third of the group, the sensible decision may be to extend licenses to similar profiles rather than to the whole company.
- If questions come up about what information Copilot can see, sort them out before moving on. At HMRC, 46% of people who didn't use their license cited security and privacy concerns. We wrote about this in Copilot data security.
If you then want to turn this into a business case, the next step is the financial calculation we explain in how to calculate the ROI of Microsoft 365 Copilot. With your own baseline data, that calculation no longer depends on someone else's averages.
Frequently asked questions
How long should a pilot run to measure AI productivity?
At least two or three weeks of baseline and about two or three months of pilot. The GDS experiment ran for three months and the DWP trial for six. The Copilot Dashboard also works with 28-day windows and has up to six days of data delay, so one month isn't enough.
Do I need to buy Viva Insights to measure Copilot adoption?
No. The Copilot Dashboard is available without a paid Viva Insights license for Microsoft 365 business and enterprise customers with Exchange Online. The paid license, and tenants with 50 or more Copilot licenses, add features such as benchmarks and group-level data.
Are user surveys worth it?
They're useful for measuring perception and spotting problems, but they don't replace measuring time per task. The available studies show that self-reporting tends to produce higher numbers than controlled methods.
Measure before you scale
AI productivity is real, but each company has to check how much and where. At Disrupsoft we help you identify the highest-impact scenarios, activate and configure Copilot in your Microsoft 365, train your team using their own work and support adoption over time. If you're about to start a pilot, or already have one without clear numbers, take a look at our Copilot for Microsoft 365 service and let's talk.
