From AI pilot to accepted operating process

A working demo answers one question: can this thing do the task on a good day? A sponsor has a different question. Can the operation rely on it on a bad day, when the input is wrong, the approver is away and the downstream system rejects the update?

Many AI efforts stall in the space between those two questions. BCG’s 2026 survey of more than 115 enterprise executives and more than 75 technology-service providers describes most enterprises as still running targeted agentic pilots, with about a third scaling, and many expecting providers to help build and operationalize priority use cases. It is a vendor-side intent survey, so it says nothing about any Sembium outcome. It does describe the gap this article is about.

What a pilot is, and what it is not

At Sembium a pilot is not a product. It proves one hard stage inside a funded operating-transformation program. The program already includes the baseline, the redesigned process, integration, rollout and measurement. The pilot tests whether the hardest step in that design holds up.

Microsoft’s guidance for AI agents makes a similar point: pilot the hardest steps, and if the pilot does not meet agreed results, pause and reassess. The same guidance says that structured, predictable tasks may not need an agent at all, and that regular code is often faster, cheaper and more reliable for fixed workflows. That matters here, because a good pilot often shows that part of the stage belongs to deterministic rules, not AI.

Five gates between demo and operation

The gates below follow the program logic Sembium proposes. Each has an artifact, an accountable owner and a decision that can be “no”. They describe a method. They are not a record of delivered client work.

Gate 1: Scope and baseline agreed

Before anything is built, the sponsor and process owner agree what is being measured and what counts as success. Microsoft’s guidance is to set baseline goals first, keep the metrics few, and use them as decision gates to continue, change direction or stop.

Artifacts: baseline definition and sample, eligible-case definition, success and failure thresholds, permitted data, permitted actions.

Gate 2: Hard case proven

The test covers the difficult cases: ambiguous documents, conflicting records, missing evidence, unusual customers. It includes the cases where the system should refuse or escalate. A result that only covers typical cases is not evidence for production.

Here the rules/AI/human split is made explicit. Known validations and state changes go to rules. AI is used where interpretation is needed. People keep material decisions.

Artifacts: test set and evaluation rubric, results including known failures, the decision matrix for rules, AI and people, and the human-review workflow.

Gate 3: Controls accepted

IT, security and the process owner review what the system may touch and what happens when it misbehaves. The OWASP 2025 list for LLM applications names prompt injection (LLM01), sensitive information disclosure (LLM02) and excessive agency (LLM06) as risks. In an operation, excessive agency is the practical one: the system can do more than anyone intended because nobody drew the boundary.

NIST’s AI Risk Management Framework describes the functions this gate needs: documented human-oversight processes (Map 3.5), separated roles for human-AI configurations (Govern 3.2), and mechanisms and assigned responsibility to supersede, disengage or deactivate a system whose behavior is inconsistent with intended use (Manage 2.4). It is voluntary guidance and not a certification, and using it does not make anyone compliant.

Artifacts: permission map (least privilege), approval and override design, logging and redaction rules, rollback procedure with an owner, incident contacts, cost and usage limits.

Gate 4: Integrated and hardened

The system connects to the real systems of record through supported interfaces. It writes only what it is allowed to write, and every write leaves evidence. Failure routes exist for the case when a downstream system is unavailable or rejects an update. Monitoring tells the owner when behavior drifts.

NIST ties post-deployment monitoring, incident response and change management together in its Manage function (Manage 4.1 and 4.3), which is a useful reminder that these belong in the release, not after it.

Artifacts: integration contract, monitoring and alert design, failure and retry behavior, runbook, release evidence signed by the relevant owners.

Gate 5: Adopted and owned

A release that nobody uses is a failed release, however clean the architecture. This gate checks that users were trained, that the first cohort works in the new way, that exceptions have a home, and that someone owns the improvement backlog.

Artifacts: adoption plan and cohort scope, training material, support responsibilities, outcome and cost review date, named owners for decisions, incidents and change.

A synthetic example (illustrative, not a client result)

Imagine a fictional B2B supplier whose customer-to-cash flow is slowed by invoice disputes. The program covers the whole path from order to cash. The hardest stage is classifying dispute correspondence and proposing a next step. Everything below is invented to show how the gates work.

• Gate 1: the baseline measures dispute age and rework on a sample of closed cases. The thresholds are agreed with finance before building.
• Gate 2: the test set deliberately includes ambiguous emails and disputes with missing delivery evidence. The system must route those to a person instead of guessing.
• Gate 3: the system can read dispute records and draft a proposal. It cannot issue a credit note or change credit status. Those stay with named approvers, and one named person can switch the classification step off.
• Gate 4: classification results post to the dispute record with an evidence trail. If the posting fails, the case goes to a manual queue and an alert goes to the owner.
• Gate 5: one team of collectors goes first. Their feedback changes the review screen before the next cohort.

Nothing in this example implies a particular saving, duration or accuracy.

Why pilots stall: four patterns

These are Sembium’s working hypotheses from the method, not measured statistics.

1. The pilot owns no part of the process. It sits beside the operation, fed by hand-picked data. Nobody in the real process is accountable for its output, so nobody adopts it.
2. The hard stage was never the hard part. The demo handled the visible task, while the expensive work sat in exceptions, approvals and system updates that the pilot skipped.
3. Controls arrive last. Security and process owners see the design after the demo has already created expectations. Their objections then look like obstruction.
4. Success was never defined. Without a baseline and thresholds, every result can be argued either way, and the decision drifts.

Gates 1 and 3 exist mainly to prevent patterns 4 and 3. Gate 2 addresses pattern 2, and Gate 5 addresses pattern 1.

Who decides what at each gate

The roles are indicative. In your organization the same person may hold several of them, and some may sit with a vendor. What matters is that each gate has a named owner who can say no.

Rollback is part of the release

Teams often treat rollback as something to write when something breaks. It is cheaper and safer as a release criterion. For each action the system can take, the acceptance packet should answer three questions. Who can stop it? What does the operation do while it is stopped? How is a wrong action reversed or compensated?

If the answers are unclear, the permission is too broad or the process around it is unfinished. Narrowing the permission is often the right fix.

What a small senior team can and cannot do here

This approach is designed for a founder-led program with specialists assembled for the engagement. That helps with direct accountability and fewer handoffs. It does not remove the work. Gates 3 to 5 need people on the client side who can approve controls, integrate with existing systems and change how teams work. Specialists for integration, security or change are committed per engagement before delivery promises are made, and their availability depends on the scope.

Stopping is a legitimate outcome

A pilot that fails its criteria has still produced something: evidence about where the process, the data or the approach is not ready. Microsoft’s guidance treats go and no-go decisions as part of the plan. Agreeing the criteria first is what keeps the decision factual.

What this does not prove

• The five gates and their artifacts describe the method Sembium uses. In a program, every threshold is set against your own baseline and agreed with the people who sign off.
• The thresholds in the synthetic example are illustrative. They are not recommended values for your operation.
• The external figures describe surveyed firms in general. They do not predict whether your pilot will scale.
• Passing every gate shows that a process was accepted under agreed criteria. It does not by itself prove a net business case; that needs a baseline and a value model over a stated horizon.

When this applies

This path fits an operation with a recurring process that crosses teams or systems, an executive owner, usable access to data and decision-makers, and a bottleneck worth $1M or more to solve. Scope and investment depend on the domain, systems, controls and specialist needs. If you need only a small standalone automation, or no owner can approve change, the full program is probably the wrong tool.

Got a pilot that works but cannot yet run in your operation? In a 30-minute call we find the gate that is blocking it. You leave with one practical recommendation, not a full diagnosis.

Book a 30-minute bottleneck call