The fair way to compare Claude, Anthropic's AI assistant, with Microsoft Copilot, the assistant Microsoft builds into Microsoft 365, is to run one real workflow through both. Use the same inputs, the same reviewer and the same measures, then compare the finished work. If your team only needs drafting help in documents and your company already pays for Microsoft Copilot licenses, that assistant probably covers it, and you can skip the test.
The two products also overlap. Microsoft's own documentation says it turns Claude models on by default for most commercial cloud customers outside the European Union, the European Free Trade Association (EFTA) countries and the UK. Those models show up inside Microsoft Copilot and Copilot in Microsoft 365 apps. So a fair test compares two products, how each one reaches your work, and which model ran each time.
The Gnar Company connects AI models like Claude to your ERP, CRM and databases through its enterprise AI integration work. Gnar is a Claude Select Services Partner in Anthropic's Claude Partner Network. The worksheet in this post is general advice for running the test yourself.
What exactly are you comparing?#
You are comparing two products and the way each one reaches your files, and sometimes the same model sits inside both. Microsoft Copilot works inside Microsoft 365 apps such as Outlook, and Claude reaches Outlook, SharePoint, OneDrive and Teams through Anthropic's Microsoft 365 connector.
Microsoft has also renamed its two Microsoft 365 Copilot products. Its requirements page says Microsoft 365 Copilot is now named Microsoft Copilot, and Microsoft 365 Copilot Chat is now named Microsoft Copilot Chat. Copilot Chat comes with an eligible Microsoft 365 plan. Microsoft Copilot needs its own license on top of that plan.
The table below puts the questions the worksheet asks next to what each company publishes. The Microsoft column comes from Microsoft's pages, and the Claude column comes from Anthropic's.
| Worksheet question | Microsoft Copilot (Microsoft's pages) | Claude (Anthropic's pages) |
|---|---|---|
| What can reach your email and files? | Microsoft says that without a Copilot license, work data reasoning covers uploaded files. Copilot in Outlook is included either way. With a license, it covers meetings, emails, chats and files. | Anthropic says its Microsoft 365 connector searches Outlook, SharePoint, OneDrive and Teams, using only the access each person already has. |
| What does an admin turn on? | A Copilot license for each user who needs Copilot to reason over meetings, emails, chats, files, and more. Without one, Copilot Chat reasons over work data only in files the user uploads. Claude models are managed under the "AI providers operating as Microsoft subprocessors" setting in the Microsoft 365 admin center. | On Team and Enterprise plans, a Claude organization owner turns on the connector. In every tenant (your company's Microsoft 365 account), a Microsoft Entra Global Administrator (the top admin role in Microsoft's identity service) grants a one time consent. |
| Which model answered? | Microsoft says on-screen indicators show when Claude models are in use in Microsoft Copilot on the web, desktop and mobile. | Record the model if the tool shows it. |
The practical point for the worksheet is that two people "using Copilot" may not be using the same thing. One may hold a Copilot license and one may only have Copilot Chat, and Microsoft's comparison of the two shows they reach different amounts of work data. Write down which version each tester has before the first input runs.
When does the assistant you already have cover it?#
The assistant you already have probably covers drafting help in documents, if your company already pays for Microsoft Copilot licenses. A side by side test is worth it when the work needs more than drafting help, or when teams disagree about which tool does it better.
A test costs the reviewer's hours and the testers' hours. When the decision is small, that time is better spent getting people to use the tool they already have. Gnar's post on why a license on its own is not AI adoption covers it.
A side by side test earns its cost when at least one of these is true:
- The workflow reaches past Microsoft 365. It needs data from an ERP, a CRM or a database, a system that needs a connector set up first.
- The output goes into another system. It lands in a customer's or supplier's system or a system of record, which is more than a drafted document, so an error costs real money.
- People already disagree. Two teams want different tools for the same job, and the argument is holding up a decision.
Work that repeats every week makes a test easier to run, because there are plenty of past cases to use. If none of those is true, the honest answer is to use what you have and revisit the question when the work changes.
Which workflow should you test?#
Test one workflow that repeats, ideally every week, has an owner who can judge the output, and has past examples with answers someone accepted. One workflow measured carefully tells you more than several tried casually, because each tool is judged on the same work.
Gnar's AI strategy work ranks AI ideas with five-dimension opportunity scoring across ROI, implementation cost, data readiness, reasoning risk, and behavior-change cost. A workflow with poor data readiness is likely to struggle in both tools for the same reason, and that tells you little about either tool.
Use this checklist to pick the workflow:
- It repeats. Weekly repeats help, because you have plenty of past cases.
- It has an owner. One person signs off on the output today and can say what "good" means.
- It has a record. Past cases still exist, with the version a person accepted.
- Its inputs can go to both tools. Your data rules allow the same files in both products.
- Its mistakes cost something you can name. A missed term, a wrong quantity, a late reply.
Good candidates in a distribution or manufacturing business look like this: a follow-up email built from a customer's order history, a summary of what changed in a supplier contract, or a check of purchase orders against a supplier's price list. If you do not have a shortlist yet, start with how to find AI opportunities in your business processes.
The worksheet, field by field#
The worksheet has eight blocks: you fill in blocks 1 to 7 before the test starts and block 8 when the last output is scored. Write it once, share it without the held-out inputs (the few cases you keep back until the end) with whoever sets up each tool, and change nothing in the workflow, the inputs, the review standard or the measures after the first input runs.
| Block | What to write |
|---|---|
| 1. Workflow and owner | The workflow in one sentence, and the person who owns its output today. That person is the reviewer. |
| 2. The finished output | What done looks like: an email ready to send, a summary someone files, a table someone acts on. |
| 3. Sample inputs | The list of real past cases, each with an ID, frozen before the test starts. |
| 4. The review standard | The criteria and the score scale, written before anyone sees an output. |
| 5. Measures | What gets recorded for each output: score, edits, errors, time, setup effort and cost. |
| 6. Tool setup | For each tool: product name, license, connectors turned on, the model or version each tool used if the tool shows it, and who set it up. |
| 7. People, order and dates | Who runs each input, which tool they use first, and the start and end dates. |
| 8. Results and decision | One row per input per tool, with the model each output used if the tool shows it, then the decision and the reason for it. |
A shared spreadsheet is enough. Keep the eight blocks fixed, so the test does not drift while it runs.
Where do the sample inputs come from?#
The sample inputs come from real past work, and both tools get the exact same set. Freeze the set before the test starts, so nobody swaps an awkward case out halfway through.
Anthropic's guide to success criteria and evaluations is written for teams building on Claude's API, but its advice on test cases carries over. It says to "design evals that mirror your real-world task distribution" and not to forget edge cases. Its examples of edge cases include irrelevant or nonexistent input data, overly long input, and ambiguous cases where even humans would find it hard to agree.
Build the set in five steps:
- Pull real cases. Take them from recent work, along with the version a person accepted at the time.
- Keep the mix honest. Include ordinary cases and hard ones in roughly the proportion they show up in a normal month.
- Apply one set of redactions. Remove anything your data rules forbid, and remove it the same way for both tools.
- Hold some back. Keep a few cases away from whoever sets up each tool, and run them only at the end. Anthropic's guide calls this a held-out test set.
- Number every case. Each input gets an ID, so the results line up row by row.
How do you write the review standard?#
Write the review standard before anyone sees an output, in terms specific enough that two reviewers would score the same output the same way. The reviewer is the person who owns the work today, because they already know what a usable answer looks like.
Anthropic's guide says good success criteria are specific, measurable, achievable and relevant. It also says qualitative scales, such as rating coherence from 1 to 5, can be valuable if consistently applied along with quantitative measures. And it notes that most use cases need evaluation along several criteria at once.
The criteria below are drawn from Anthropic's list of common success criteria. Rewrite each line for your own workflow.
| Criterion | A 5 looks like | A 1 looks like |
|---|---|---|
| Task fidelity | Every fact matches the source, and nothing the reader needs is missing. | A wrong number or a missed item that the reader would act on. |
| Tone and style | Reads like your team wrote it, at the right length. | Needs a rewrite before anyone outside could see it. |
| Privacy | Contains only what the recipient is allowed to see. | Includes data from a record the recipient should not see. |
| Use of context | Uses the files and history it was given. | Ignores the attached files or invents details. |
Score each output blind. Strip anything that shows which tool wrote it, such as product names or formatting habits, and shuffle the order before the reviewer starts. Blind scoring keeps a reviewer's preference for one tool out of the scores.
What should you measure?#
Measure the finished work directly, with the same instruments applied the same way to both tools. A review score, the edits needed before use and the errors that would have reached a customer all come from your own reviewer, so they count the same way for Claude and for Copilot.
| Measure | How to record it | Why it belongs on the worksheet |
|---|---|---|
| Review score | The score on each criterion from the review standard. | Shows quality on your own work. |
| Edits before use | The number of changes the reviewer made, or a plain yes or no for usable as is. | Rework is where the hours go. |
| Errors that would have shipped | Wrong facts, missed items, or data that should not be there. | These are the expensive failures. |
| Time to a usable output | Minutes from the first request to the accepted output, recorded by the person running it. | Includes the back and forth with the tool. |
| Setup effort | The admin steps and approvals each tool needed for the test group. | Part of the real cost of either tool. |
| Cost | Seat and usage prices from each vendor's quote to you. | Use your own quote, since list prices change. |
Microsoft's own reports are useful for tracking adoption, and Microsoft is careful about what its numbers mean. Its Copilot Dashboard covers readiness, adoption, impact and sentiment. Microsoft says its Copilot assisted hours figure is computed from Copilot actions and multipliers from Microsoft's research, and "should be viewed as a general estimate."
In its business impact report, Microsoft notes that the differences shown "have not been evaluated for statistical significance." It also says Copilot usage is unlikely to be the only variable behind any difference. Those are fair cautions for any tool's numbers. They are also why a side by side test leans on measures your reviewer records for both tools.
One Gnar engagement shows why a measure that stays fixed is worth having. At GSC Enterprises, Gnar ran a four-week Immerse cohort, a mentor-led program, for five engineers using Claude Code, Anthropic's coding tool. That was a mentoring cohort, and it did not compare tools. GSC runs an independent code review agent on every pull request, and the share of pull requests it commented on fell from roughly 100% to less than 10% over the four weeks. The tool stayed the same.
How do you keep the test fair?#
Keep the test fair by holding everything steady except the tool: the same people, the same inputs, the same reviewer, equal setup and a written record of what produced each output. Any difference you did not plan becomes a reason to doubt the result.
Run through this checklist before the first input:
- Same people, split order. The same testers use both tools. Half start with Claude and half start with Copilot, so practice on the first tool is spread evenly across both tools.
- Same reach into your data. If the workflow needs Copilot to work across each tester's own meetings, emails, chats and files, Microsoft's comparison page lists that as "Full" with a Copilot license and "Limited (uploaded files)" without one, so those testers need the license. The Claude side needs the Microsoft 365 connector set up. Anthropic says the connector uses each person's existing permissions and is read only by default.
- Equal setup time. Each tool's owner gets the same window to write instructions and try them, using cases outside the held-out set.
- A record of the model. Microsoft says on-screen indicators show when Claude models are in use in Microsoft Copilot on the web, desktop and mobile. Write the model next to every output if the tool shows it. If Copilot ran a Claude model, the test is comparing two products and their setup more than two models.
- The same access limits. Give the test group the same files, mailboxes and systems in both tools.
- Fixed dates. Name the start and end dates in block 7 and keep them.
How do you read the results?#
Read the results input by input, starting with the failures, before you look at any average. A tool that scores slightly higher on average but fails on the cases that matter most gives a different answer from one that is steady across the set. Use these rules to reach a decision:
- One tool clearly passes. Its outputs meet the standard with fewer edits and no serious errors on the held-out cases. That is your answer for this workflow, and only for this workflow.
- The results are close. Call it a tie. A tie moves the decision to cost, setup effort and where the work already lives.
- Both tools fail. Look at why. If both missed data neither could reach, the problem is access to your systems, and a new tool will not fix it.
- The answer is both. Microsoft offers Claude models inside Copilot, and Anthropic's connector lets Claude read Microsoft 365 with each person's permissions. Some companies will reasonably use each one where its work starts.
Write the decision and the reason in block 8. A year from now, someone will ask why the company picked what it picked.
Where does outside help fit?#
Outside help fits when the workflow you tested needs AI to reach systems outside Microsoft 365, such as an ERP or a CRM, because then the connections are the real work. The Gnar Company builds those connections and keeps them separate from any one model, so the next model can use the same connectors and access rules.
Claude is The Gnar Company's default, and Gnar also builds with Google's Gemini API and on Amazon's AWS Bedrock. Gnar's first step is to list each system AI needs to reach, what it may see and change, and who signs off. That list is a useful input for block 6 of the worksheet, too.
At GSC, MCP connectors for Snowflake, Atlassian, Microsoft 365 and GitHub brought the company's real systems into the loop for its engineers using Claude Code. MCP stands for Model Context Protocol, an open-source standard for connecting AI applications to external systems. It is one way a test that failed for lack of data can be rerun with the data in reach.
When the test points to a build, Gnar Discovery and Design is the first step. Two weeks, $15,000, and you walk out with a build plan that has a fixed price on it. If your own engineers have the capacity and the direction, do it internally and we will say so on the call.
For leaders who want to learn Claude before any test, Gnar runs a free AI workshop series of 8 live sessions, 45 minutes each, on Google Meet.
