Skip to main content
Published in Product Insights

AI Pilot Success Criteria: Scale, Extend, Narrow or Stop

AI Pilot Success Criteria: Scale, Extend, Narrow or Stop
Table of contents
  1. What are AI pilot success criteria?
  2. Why do AI pilots get abandoned after the proof of concept?
  3. Which four decisions can end an AI pilot?
  4. What should the sponsor agree before the pilot starts?
  5. What does each decision need at the readout?
    1. Scale
    2. Extend
    3. Narrow
    4. Stop
  6. How should a CFO price the decision to scale?
  7. How do you check that a good result is real?
  8. What does it look like when the evidence changes the plan?
  9. Where does outside help fit after a pilot?

AI pilot success criteria are the terms, written before a pilot starts, that decide how it ends. Every pilot should end in one of four decisions: scale it, extend it, narrow it to where it worked, or stop it.

Agree in advance what evidence each decision needs and what it will cost. Then the readout, the end-of-pilot review, is a reading of the result against terms everyone signed. Without those terms, it turns into a debate about how the demo felt.

The Gnar Company describes itself in one line: "Custom software built around how your operation actually works, from a firm that will tell you when not to build it." Its enterprise AI integration page explains the work: "Enterprise AI integration connects an AI model, such as Claude, to the systems your company already runs." The same page says "Gnar is a Claude Select Services Partner in Anthropic's Claude Partner Network."

This guide is for the CFO who sponsors an AI pilot that ends soon. It applies to a pilot of any AI tool or model. The four decisions and the terms below are general advice.

What are AI pilot success criteria?#

AI pilot success criteria are written tests that map a pilot's result to a decision. They name the number that matters, its baseline, the result that earns each of the four decisions, the readout date and the person who decides.

The U.S. National Institute of Standards and Technology (NIST) builds the same step into its AI Risk Management Framework. Subcategory MANAGE 1.1 reads: "A determination is made as to whether the AI system achieves its intended purposes and stated objectives and whether its development or deployment should proceed."

The framework, published in January 2023, also lists a benefit it expects users to get. That benefit is "explicit processes for making go/no-go system commissioning and deployment decisions".

Write the criteria before the result exists, so nobody can set the bar to fit the score.

Keep the list short. One deciding number and a cost figure are enough for most pilots. Each extra number is one more place for the readout to argue.

Why do AI pilots get abandoned after the proof of concept?#

Gartner's July 2024 forecast names four reasons, "poor data quality, inadequate risk controls, escalating costs or unclear business value."

That 2024 forecast covered projects through the end of 2025: "At least 30% of generative AI (GenAI) projects will be abandoned after proof of concept by the end of 2025".

A second Gartner release, from June 2025, covers AI agents. It says "Over 40% of agentic AI projects will be canceled by the end of 2027, due to escalating costs, unclear business value or inadequate risk controls".

The 2024 release also says: "Historically, many CFOs have not been comfortable with investing today for indirect value in the future."

Some pilots can only show indirect value, such as staff saying the tool helps. For those, write down before the start which decision that kind of evidence can earn.

Which four decisions can end an AI pilot?#

Four decisions can end an AI pilot: scale, extend, narrow or stop. Each one needs its own evidence and its own next commitment.

Decision What the evidence shows What the CFO funds next
Scale The deciding number cleared its threshold, the cost at full volume is known, and someone owns it after launch The production build, support after launch, and the seats or usage for more people
Extend The result is close to the threshold, and one named question explains the gap One more round with a new end date and the same thresholds
Narrow It worked for some teams, cases or records and not for others Only the part that worked, under a new written scope
Stop The number fell below the band agreed for an extension, a risk has no fix, or nobody will own it Nothing more for this pilot. The budget goes back, or moves to a plan that does not use AI

Gartner's Rita Sallam describes the scale case in the 2024 release. "If the business outcomes meet or exceed expectations, it presents an opportunity to expand investments by scaling GenAI innovation and usage across a broader user base, or implementing it in additional business divisions."

Extend and narrow are the two decisions a sponsor most needs to define in advance. Both spend more money. Both can also stand in for a decision nobody wants to make. Writing their terms down first keeps them honest.

What should the sponsor agree before the pilot starts?#

Agree six things in writing before the pilot starts: the deciding number, its baseline, a threshold for each decision, the readout date, the decider, and the cost of each decision. Put them on one page and have the sponsor and the pilot lead sign it.

  1. The deciding number. Pick one measure of the work itself. Good examples: invoices matched per clerk hour, quotes sent per day, or tickets closed without a second touch.
  2. Its baseline. Measure that number before the pilot, over a period like the one the pilot covers. A quiet month makes a poor baseline for a busy one.
  3. A threshold for each decision. Write the result that earns scale, the band that earns an extension, and the line below which the pilot stops. Say which result points to a narrower scope.
  4. The readout date. Set it on the calendar now. A pilot with no end date has no readout.
  5. The decider. Name one person who makes the call on the readout date.
  6. The cost of each decision. Write what scaling would cost at full volume, what an extension would cost, and what stopping would save.

The last item is the CFO's to ask for. It turns the readout into a funding decision with known amounts. When the scale cost depends on connecting systems, Gnar's integration page says its first step, Gnar Discovery and Design, "maps your systems and ends with a build plan that has a fixed price on it."

If the pilot came out of an AI roadmap, each roadmap line should already carry a cost, a return and an owner. Gnar's guide to an AI roadmap the board will fund puts a number and an owner on every line.

What does each decision need at the readout?#

Each decision needs its own checks on the readout date. Scale needs the threshold cleared and the cost at full volume written down. Extend needs one named question. Narrow needs a new written scope. Stop needs a recorded reason and a person who turns the pilot off.

Scale#

  • The deciding number cleared its threshold against the baseline.
  • The cost at full volume is written down.
  • A named person owns the system after launch, with a budget to run it.
  • The people who will use it at full scale are the kind of people who used it in the pilot.

Extend#

  • The result sits in the band agreed for an extension.
  • One named question explains the gap, such as missing data or an unfinished connection to a system.
  • The extension has a new end date and keeps the same deciding number and thresholds.
  • Allow one extension at most. If that question is still open at the second readout, stop.

Narrow#

  • The result is strong for some teams, cases or records and weak for the rest.
  • The strong part is large enough to be worth running on its own.
  • The new scope is written down before more money is spent.

NIST's framework asks for a documented scope. MAP 3.3 reads: "Targeted application scope is specified and documented based on the system's capability, established context, and AI system categorization." A narrowed pilot is a new scope, so it gets a new document and a new threshold.

Stop#

  • The deciding number fell below the band agreed for an extension, a risk has no fix, or nobody will own the system.
  • The reason is written down in one paragraph, for the next pilot.
  • Someone is named to switch the pilot off and remove its access.
  • The budget goes back, or moves to a plan that does not use AI.

NIST covers the switch-off step. MANAGE 2.4 asks that "responsibilities are assigned and understood, to supersede, disengage, or deactivate AI systems that demonstrate performance or outcomes inconsistent with intended use."

How should a CFO price the decision to scale?#

Price the decision to scale at what the system will cost at full volume. A pilot can run on hand work that a rollout cannot use: records pasted into a chat and an engineer watching every step.

Gartner's Sallam says in the 2024 release that "costs aren't as predictable as other technologies." The 2025 release says, about AI agents: "Integrating agents into legacy systems can be technically complex, often disrupting workflows and requiring costly modifications."

Gnar's integration page describes the hand work in one sentence: "Your AI can't see your orders, customers or files, so people paste them into a chat by hand." A pilot that ran that way did not yet pay for the connection to the order system. A rollout has to.

Cost line In the pilot At full volume
Data access Records pasted or exported by hand A connection to each system, with access rules
Usage A few users and a short window Every user, every day, at the vendor's current price
Review The pilot team checks each output Reviewers in the business, on a set schedule
Support The pilot team fixes problems as they come up A named owner and a support budget
Security Reviewed for the pilot's scope A full review of what the AI can see and change

Ask the pilot team to fill in the right column before the readout. A blank cell there means the scale decision is not ready.

The connection cost depends on the systems involved. Gnar's integration page says "The price depends on how many systems the AI reaches and what it may change in each."

Gnar's post on moving an AI agent from demo to production covers what a production system needs: data, tests, permissions and ownership.

How do you check that a good result is real?#

Check the measuring tool before you trust a big gain. Confirm that nothing about how the number is counted changed during the pilot, and compare it with a period like the baseline.

Gnar's GSC Enterprises case study covers a four-week Claude Code cohort for five engineers on their own production code. Afterwards, the participants ran an internal AI training for 70 GSC leaders. GSC "had already put Claude Code in the hands of its engineering team" before the cohort began.

The case study reports that "PRs flagged by GSC's automated code reviewer fell from roughly 100% to under 10%, with no change to the tool". It also says "The drop was sharp enough that GSC's Business Application Manager checked whether the reviewer was broken."

That check is a habit worth copying at any pilot readout:

  • Confirm the tool or report that produced the number did not change its rules, settings or version during the pilot.
  • Compare the pilot period with a baseline period of the same length and the same kind of work.
  • Read a sample of real outputs, including the failures, before you read the average.
  • Count the time people spent checking and fixing the AI's work against any time it saved.

A result that survives those four checks is one a CFO can defend to the board.

What does it look like when the evidence changes the plan?#

Evidence changes the plan when the data shows a candidate is weaker than it looked, and the recommendation moves to where the value is. Gnar's Counter Culture Coffee case study shows this in a strategy engagement, which is not a pilot.

Gnar built a prioritized AI roadmap for Counter Culture Coffee, a coffee roaster. It found 79 AI and automation opportunities, then scored and ranked each one. The case study lists the scoring: "Every opportunity scored on the same five dimensions: financial ROI, operational efficiency gains, implementation complexity, data readiness, and cultural readiness."

The case study then describes an idea that changed. "The process also tested its own assumptions. Coffee procurement initially looked like a market-timing opportunity." A full year of contract data showed the team was already managing that well. The recommendation shifted to "a single, real-time view of the coffee position across contracts, importers, and warehouses."

Gnar's AI strategy and fluency page lists that as part of the work: "use cases ranked by ROI, an implementation roadmap, and a clear view of what not to build." A pilot readout can do the same job for one use case. The narrow and stop decisions are how it does it.

Where does outside help fit after a pilot?#

Outside help fits when a pilot stalled and the company has no engineering capacity to point at the next step, or its engineers are busy with other work. Gnar's AI strategy and fluency page is "Built for leaders whose pilots stalled and whose AI spend hasn't compounded." It goes on: "We start where the pilots stopped and aim straight at a return you can defend to the board."

Some pilots earn a scale decision but still need their connections built. For them, the first step on Gnar's integration page is "Gnar Discovery and Design, from two weeks and $15,000." It "maps your systems and ends with a build plan that has a fixed price on it." The Discovery page lists three tiers: $15,000 for two weeks, $20,000 for four and $30,000 for six.

A company with its own engineers may not need outside help at all. Gnar's integration page answers that directly: "Often they should. If your own engineers have the capacity and the direction, do it internally. The Gnar Company will say so on the call."

Frequently asked questions

What are AI pilot success criteria?

AI pilot success criteria are written tests, agreed before a pilot starts, that map its result to a decision. They name the deciding number, its baseline, the result that earns each decision, the readout date and the person who decides. NIST's AI Risk Management Framework asks for a related determination in its MANAGE function.

What are the four outcomes of an AI pilot?

The four outcomes are scale, extend, narrow and stop. Scale funds the rollout. Extend buys one more round with one named question and a new end date. Narrow keeps only the teams, cases or records where the pilot worked. Stop ends the spend and records why. Agree what each one needs before the pilot starts.

Should an AI pilot that missed its target get more time?

Yes, once, when one named question explains the gap and the result is close to the threshold. Give the extension a new end date and keep the same deciding number and thresholds. If that question is still open at the second readout, stop the pilot instead of extending it again.

Is stopping an AI pilot a failure?

No. A stop made against terms agreed before the pilot is one of the decisions the pilot was built to reach. NIST's AI Risk Management Framework lists explicit processes for go or no go decisions among the benefits it expects users to get. Record the reason, so the next pilot does not repeat it.

What does The Gnar Company commit to on a build engagement?

Three commitments on every build engagement: a fixed milestone price with any overage on us, a guaranteed outcome meaning the software works as specified or we fix it, and a twelve-month bug-free warranty. It covers Discovery, Build, Forge and Maintain. It does not cover Embed. On Compass, Immerse and Integrate, ask us.

Should our own engineers build what comes after the pilot?

Often, yes. Gnar's integration page answers this question: "Often they should. If your own engineers have the capacity and the direction, do it internally. The Gnar Company will say so on the call. This work is for companies with no engineering capacity of their own to point at this. It also fits teams whose engineers are busy with other work."

Stay In The Loop

Never miss an update. Get the latest blogs, webinars, and expert insights straight to your inbox from The Gnar.