A note to this community. Over the last four months, many of you have sent DMs sharing what you are seeing inside your own organisations. Retraining loops that do not hold. Agents nobody owns. Cost numbers that do not add up. Some of you shared company details that I have anonymised and woven into the case studies in this newsletter and in my work. That trust is not lost on me. Thank you.
This week, the book that connects all of these concepts went live. The Pilot Trap: The Category Error Behind Enterprise AI Failure. It explains why thirty years of delivery frameworks fail when applied to AI, builds the operating discipline that replaces them, and gives you a diagnostic you can run across your own portfolio. If you have been reading this newsletter and recognising the problems, the book is where the full discipline comes together.
Amazon link: [LINK]
Last issue I ended with a question. A telco's billing dispute agent was losing $940,000 a year. The CTO said fine-tune the model. The question that was missing was different: who is coaching this agent?
That question landed differently with different readers. Some asked what I meant by coaching. Others said their teams already do it. A few said they had never heard the term applied to AI systems.
This issue explains what it means, why it matters, and how to tell whether your organisation is doing it or just retraining on a loop.
A mid-size financial services firm deployed an AI agent to handle customer onboarding. The agent verified identity documents, cross-referenced regulatory requirements, and set up new accounts. At launch, it processed 84% of applications without human intervention. The remaining 16% were edge cases that required manual review. The team was satisfied. The numbers matched the pilot.
Five months later, the rejection rate climbed to 31%. Valid applications were being flagged and sent to manual review. Customers complained about delays. The operations team raised it with the technology lead. His response was immediate: retrain the model.
They did. The rejection rate dropped to 19% within a week. Two weeks later it was back at 28%. They retrained again. Same pattern. Drop, then climb. Three retraining cycles in four months. Each time, the rejection rate dropped for a week or two before climbing back. The temporary improvement was enough to convince the team that retraining was the right approach. It never occurred to them that the problem was not inside the model. Each cycle consumed roughly two weeks of engineering time. Four cycles was eight weeks of engineering capacity diverted from other projects, none of which appeared on the AI programme's cost ledger.
Nobody asked why the agent was rejecting valid applications. They only asked how to make it stop.
The answer was in the environment, not the model. Two things had changed since deployment. First, the financial regulator updated its guidance on acceptable identity documents. The agent was never recalibrated against the new requirements. It was applying rules it had been trained to recognise at deployment to document types that did not exist when it was built. Second, a major bank in the region introduced a new digital ID format. The agent had never seen it. It treated every application using that format as suspicious.
The model was not broken. It was doing exactly what it had been trained to recognise at deployment. The world around it had changed. The agent had not. And nobody was responsible for checking whether the two still matched.
The fourth time the technology lead proposed retraining, a programme director asked a different question. Instead of retraining every six weeks and hoping, what if someone reviewed the rejection patterns fortnightly, compared them against current regulatory guidance, and made targeted adjustments based on what they found?
That is not retraining. That is coaching.
They started a fortnightly review. One person, ninety minutes every two weeks. They pulled the rejection log, categorised the reasons, and checked them against current regulatory requirements sourced from the compliance team's latest guidance notes. Each adjustment followed the firm's existing change approval process. The difference was that the adjustments were small, informed, and fortnightly instead of large, blind, and quarterly.
In the first review, they discovered 40% of rejections were triggered by the new digital ID format. The fix was not a full retrain. It was a targeted update to the document classification layer using 200 sample documents of the new format. Applied in an afternoon. The next full retraining cycle would have taken two weeks to achieve the same result with less precision.
Within two months, the rejection rate stabilised at 14%. Lower than the original 16% at launch. The retraining loop stopped. The cost of the coaching practice was one person's time for three hours a month. The cost of not coaching had been four retraining cycles, eight weeks of diverted engineering capacity, three months of elevated customer complaints, and an operations team that had quietly started processing applications manually because they no longer trusted the agent. That manual workaround alone added three FTE equivalents in overtime costs that never appeared on the AI programme's ledger.
The distinction between training and coaching is not a language preference. It is a structural difference that changes how the enterprise budgets, staffs, and governs AI in production.
Training is a finite event. It happens before deployment. Success is measured by completion. The assumption underneath it is that the system will behave the same way in production as it did in testing. For deterministic software, that assumption holds. For AI agents, it does not. The environment changes. The data shifts. The agent adapts, sometimes correctly, sometimes not. And there is no alarm that tells you which one is happening.
Coaching is a continuous practice. It happens after deployment. Success is measured by whether the agent is still producing the business outcome it was deployed to serve. The assumption underneath it is the opposite of training: the system's behaviour will change, and someone needs to be watching, interpreting, and calibrating.
Most enterprises budget AI like a project. There is a training line item in the build phase. When the project closes, the budget closes with it. The agent keeps running. Nobody has the funding or the mandate to coach it, because coaching was never a line item.
That is how you end up with the retraining loop. The team has no coaching practice, so the only tool they have is retraining. Retraining fixes the symptom for two weeks. The underlying cause, a changing environment that nobody is tracking against, stays in place. The drift returns. They retrain again.
Monitoring is not coaching. Most enterprises that say "we already do this" are pointing at their dashboards. Dashboards tell you what happened. They show the signal. Automated. Passive. No human judgment required.
Evaluation goes one step further. It compares the signal against a baseline and produces a finding. The rejection rate is 31%, the baseline was 16%, performance has degraded. That finding is useful. But it is still incomplete. It does not ask why the degradation happened, whether the business context changed, what the minimum correction looks like, or what the finding should change about how the agent is monitored next time. Evaluation produces a score. Coaching produces an adjustment and updates the evaluation itself. An enterprise with strong evaluation and no coaching practice knows exactly how far its agents have drifted. It just cannot do anything about it.
The question is not whether you are monitoring your agents. It is not even whether you are evaluating them. The question is whether someone is interpreting the signals, understanding the business context that produced them, making a calibrated adjustment, and feeding what they learned back into how the agent is monitored next time.
If the answer is "when something breaks, an engineer investigates," that is incident response. It is not coaching.
Pick the agent in your organisation with the highest escalation rate or the highest rate of human override.
Ask three questions:
When was this agent's output quality last reviewed against its original business case? Not its uptime. Not its task completion rate. Its output quality measured against the outcome the business expected.
Who conducted that review? Was it a scheduled practice or a response to a complaint?
What changed as a result? Was a specific adjustment made based on current business context, or was the finding recorded and left?
If the review never happened, the agent is not being coached. It is running on whatever it was trained to recognise at deployment, in an environment that has changed since then.
If the review happened but nothing changed, the coaching loop is incomplete. A review without a calibrated response is evaluation, not coaching. It measures the gap without closing it.
If the review happened, a specific adjustment was made, and that adjustment was informed by current business context and fed back into how the agent is monitored, you have the beginning of a coaching practice.
Most teams discover they are in the first category.
How does your organisation respond when an agent's performance degrades? Reply with one answer:
(a) Retrain the model on a schedule.
(b) An engineer investigates and makes a technical fix.
(c) A cross-functional review evaluates the business impact and calibrates the response.
(d) Nobody notices until a customer or downstream team raises it.
One letter. Every response adds to the first cross-enterprise dataset on how organisations actually manage AI agents in production.
The Pilot Trap went live this week. It starts with the question most enterprises have not asked: why do delivery frameworks that worked for thirty years of software fail when applied to AI? The first four chapters diagnose the structural mismatch. The next four build the operating discipline that replaces it, including the full agent coaching model this issue introduces. The final four chapters bring evidence from enterprises that are getting it right, the regulatory deadlines approaching, and a diagnostic you can run across five agents in your portfolio this week.
Amazon link: [LINK]
Next issue: who owns the agent after the build team moves on? Not the sponsor. Not the platform team. Not the vendor. System stewardship is the answer nobody has staffed for yet. Issue #6 is about the role that holds it all together.
Vijayan Seenisamy
Enterprise Agentic AI Systems Delivery | Creator, AI ROF™
Author, The AI Delivery Manager Blueprint and The Pilot Trap
New issues arrive by email first. Subscribe here.
