The first production run is a bad place to learn what an agent gets wrong. Something will be wrong. The date format from one vendor, a rule that was written for a policy which changed last year, an attachment that turns out to be a photo of a document. None of that is interesting when it lands in a log. All of it is expensive when it lands in a ledger.
Shadow mode is the cheap way to find those things first. Most teams either skip it or run it in a way that proves nothing, so it is worth being precise about what it is and what it cannot tell you.
What shadow mode is
The agent reads real inputs from the real systems. It does the whole job. It pulls the data, applies the rules, reaches a decision, composes whatever it would have sent. Then it writes its answer to a log and stops. Nothing reaches the system of record. No mail leaves the building. No line gets posted.
Where you stop it matters more than people expect. You do not stop it in the prompt. Telling a model not to actually send the email is a request, and a request gets ignored eventually, usually on the run with the strange input. Block the writes at the connector instead. Give the shadow run read-only credentials, or replace each write function with a recorder that stores the payload it was handed and returns a plausible success. Then the model's decision does not matter, because the capability is not there.
The inputs have to be real, live, and in the order they arrive. A shadow run against a cleaned export tells you the agent handles cleaned exports. Production data carries the half-filled records, the duplicate submitted twice in four minutes, the vendor name spelled three different ways across three systems.
Agree the pass mark before the run
A pass mark decided after seeing the results is not a pass mark. It is a negotiation, and the side that wants to go live wins it. Write the number down first, and make it specific enough to fail.
Start by sorting outcomes into four groups rather than counting a single accuracy figure. The agent and the human agreed. The agent escalated instead of deciding. The agent decided and was wrong. The agent decided, the human record disagrees, and the agent was right. That last group is real and teams forget it exists, which means somebody has to adjudicate the disagreements rather than treat the historic human record as truth. Budget for it. One person, about an hour a day for the length of the run.
The pass mark itself needs a match rate on the decisions that must be identical, a ceiling on confident errors, and an escalation rate the team can absorb.
Confident errors are the number to watch. That is the agent being wrong and saying nothing about it. An agent that escalates too often is irritating and you can tune it. An agent that is quietly wrong twice a week teaches the team to check everything, and once they check everything you have bought a slower process with extra software in it. One confident error on a payment run is worse than fifty escalations.
Derive the escalation ceiling from review capacity, not from ambition. If the workflow handles 500 items a week and the person reviewing exceptions can get through 60, the ceiling is twelve percent. Put twelve percent in the document. When the run comes back at nineteen, you have a decision in front of you instead of an argument.
What to compare
Compare decisions, not prose. For a classification, compare the labels. For a reconciliation, compare the pair of record ids, because two matches can produce the same net position while allocating it to different invoices. For a drafted reply, do not diff the text against what the human wrote. Score it against three yes or no questions. Did it answer what was asked. Did it state anything false. Did it commit the company to something it should not have. Three checkboxes beat a similarity score, and two reviewers will agree on them.
Compare the work the agent did to get there as well. If it made fourteen API calls to reach a decision a person made from one screen, it will still be correct and it will be slow and costly at volume. That is worth knowing before you multiply it by a year.
Now the limit. Shadow mode tests the decision and not the write. Writes have their own failure modes and none of them show up here: rate limits under a bulk day, duplicate records created when a retry fires after a timeout, a status change that notifies two hundred people, a webhook that loops. Test those separately in a sandbox, or with one live item and somebody watching the screen.
How long to run it
Count decisions, not days. Then check whether you have seen the calendar.
A workflow that runs forty times a day gives you around 400 decisions in two weeks, which is enough to see the shape of the failures. A month-end close gives you one run and one sample, which is not a test at all. For those, replay history. Point the agent at the last three or four closes and compare its output against what was actually booked. Replay is weaker than live shadow, because it cannot show you the inputs nobody kept, but one sample is worse.
Whatever the volume, cover one full cycle of whatever drives the variation. Month-end, quarter-end, the week a public holiday shifts every settlement date, the day your largest client submits in bulk. A run that ends before the hard week has tested the easy weeks.
Group the failures by cause
A match rate tells you nothing about the fix. Group the misses by cause and the list usually turns out to be short and specific. One vendor writes dates the other way round. One intake channel strips attachments. One rule encodes a threshold that finance changed in March.
Fix a single cause and the rate can move several points at once. The reverse result is just as useful. If a fifth of the misses have no pattern, the decision probably needs information the agent cannot reach, and that slice of the workflow stays with a person permanently. Better to learn that in week two than to keep tuning against it for a month.
Try not to change the agent mid-run. If you must, mark the cut and count the two halves separately, because every change resets the sample and a blended number hides which version you are promoting.
Promoting to live
Do it in stages, not in one step.
The first live stage keeps a human release on every action. The agent does the work, a person clicks send. This is slower than the old process on paper and it usually is not in practice, because reading a finished draft is faster than composing one. Two weeks of it tells you something you cannot learn any other way, which is whether the person releasing is reading or just clicking.
The second stage moves the release to the escalation path only. Confident decisions go through. Flagged items wait for a human, with the agent's reasoning attached so the human is not starting from nothing.
Keep sampling after go-live. Twenty items a week, checked by hand, for the first month or two. And keep a documented way back: the switch that returns the workflow to people, and the name of the person allowed to throw it. Write the name down. A rollback plan with no owner is a paragraph.
Shadow mode costs a couple of weeks and a chunk of one person's attention. The alternative is finding the same failures with a live ledger and a client on the phone. I would pay the two weeks every time.