The most expensive AI failures are the ones nobody notices for months.
I am writing this between sessions at NVIDIA GTC in San Jose. This week, 40,000 people are hearing about the next generation of AI agent platforms. This issue covers the failure mode I find most unsettling, because it requires no mistakes to trigger.
The Failure Anatomy
Eight months ago, a regional logistics company launched an AI-powered demand forecasting agent across twelve product categories. The rollout was textbook: a three-month pilot, accuracy validated at 91%, results documented and presented to the VP of Supply Chain before going live. Early production numbers matched the pilot. At the six-month mark, the VP presented the system’s performance to the board as one of the company’s most successful technology investments.
Nobody in that boardroom knew the system had been quietly wrong for two months.
Four months after go-live, one of the agent’s upstream data feeds changed. A third-party market signals provider updated their methodology for calculating regional demand indicators. They did not notify downstream consumers. The agent kept processing. It absorbed the new data shape without error, without alert, without any visible change in behaviour. Probabilistic systems do not break when inputs shift. They absorb and adjust. Whether that adjustment improves or degrades the output is something you will never know unless you are actively checking. Nobody was checking.
Aggregate forecast accuracy across all twelve categories dropped from 91% to 88%. A three-point move. Within the acceptable range on every dashboard the company had built. Nobody investigated.
But accuracy is an average. Underneath it, one category was collapsing. Seasonal goods forecasting, which had been running at 91% accuracy, dropped to 67% over four months. The agent was overestimating demand by 20 to 40 percent on some SKUs and underestimating on others. The remaining eleven categories held steady, which kept the aggregate number high enough to mask the problem entirely.
The warehouse team noticed first. Not because anyone told them. Because their jobs changed. Orders for seasonal goods started arriving in quantities that did not match what they were seeing on the floor. They began manually adjusting orders without filing a ticket or escalating to anyone. The shift supervisor’s explanation, surfaced during the post-incident review: “We figured it was a temporary glitch. We just handled it.”
For four months, warehouse workers operated as an invisible human fallback layer for an AI system that every dashboard said was performing within acceptable range. Their labour was never visible in the agent’s performance metrics. The system was not failing. It was being quietly subsidised by people.
The discovery was accidental. A finance analyst running a quarterly cost review flagged a 34% spike in warehouse overtime in one region, traced it to seasonal goods, and learned the warehouse team had been manually overriding the system for four months.
There was no crisis meeting. The system had never stopped running. It had just quietly become a different system than the one the board was told about. Excess inventory carrying costs, warehouse overtime, write-downs on perishable stock, and a team that refused to trust the system even after recalibration. The re-forecasting project took six weeks. Rebuilding trust is still in progress.
This is Silent Drift. The system did not break. It shifted, gradually, over months, while every metric designed to catch problems confirmed that nothing was wrong. And nothing in the organisation’s monitoring infrastructure was designed to see the difference between a system that is working and a system that used to work.
I have seen this pattern across four organisations in three industries now. The details change. The structure does not. A system that works, a team that trusts it, and a slow invisible shift that nobody built the infrastructure to detect.
The Framework Lens
The logistics company had monitoring. They could see uptime, throughput, and aggregate accuracy. Their dashboards were full of green indicators. Every check they had built was passing.
That is the problem. They were monitoring the system’s operation. Nobody was evaluating its behaviour. The gap between those two activities is where Silent Drift lives.
In AI ROF, the Observability dimension draws a hard line between these two activities. Monitoring answers the question: is the system running? Evaluation answers a different question entirely: is the system still producing the outputs we validated before deployment? Most enterprises have built robust monitoring. Evaluation infrastructure is rare. I would estimate fewer than one in ten organisations running AI in production have a scheduled process for comparing current outputs against validated baselines. The distinction matters because probabilistic systems can pass every operational health check while quietly producing outputs that have drifted from what was originally approved.
The logistics company measured forecast accuracy at the aggregate level. Twelve categories averaged into one number. That is the equivalent of measuring a hospital’s patient outcomes by averaging survival rates across every department. Cardiology could be in crisis and the number would still look acceptable because orthopaedics and dermatology are holding it up.
Evaluation requires granularity. The right question is not “is the system 88% accurate?” It is “is the system still performing within validated range for each task category it was approved to handle?” Answering that requires different infrastructure than most enterprises have built.
The Five Observability Signals in AI ROF exist because a single aggregate metric cannot surface drift. Signal 1, Reliability, measures task success rate by scenario, not in aggregate. If the logistics company had tracked accuracy by product category, the seasonal goods collapse from 91% to 67% would have surfaced in weeks, not months.
But evaluation infrastructure cannot be bolted on after deployment. The logistics company’s delivery team shipped a system with no category-level baselines, no definition of what a meaningful accuracy drop for a single category would look like, and no one assigned to run output comparisons after go-live. All of that was implicitly deferred to operations. And operations, receiving a system with green dashboards and no evaluation tooling, had no reason to believe anything was wrong.
Drift detection is a delivery responsibility. If the delivery team does not build it, it does not exist. And if it does not exist, the only drift detection mechanism is a human being noticing that something feels wrong. In the logistics company’s case, that human being was a warehouse shift supervisor who quietly worked around the problem for four months before a finance analyst stumbled into it by accident.
That is not an observability model. That is a warehouse worker with good instincts and a finance analyst who happened to be looking at the right spreadsheet in the right quarter.
The Field Test
One test. One system. This week.
Pick one AI system in your organisation that is currently in production. Pull 50 outputs from this week for its most common task type. Then pull 50 outputs for the same task type from 60 days ago.
Compare them. Not aggregate accuracy. Case by case. For the same type of input, is the system producing the same quality and consistency of output it produced two months ago?
If the variance exceeds 5%, you have drift. The question becomes whether anyone would have found it without this test.
If you cannot pull 60-day-old outputs because they were never saved, that is the finding. You have no baseline. You cannot detect drift if you never recorded what normal looked like. Your system could have shifted the week after deployment and nobody would know.
If you are still in pilot, run this test anyway. Compare outputs from the first week to the most recent week. Drift does not wait for production.
One test. If you have drift detection, this confirms it is working. If you do not, you just found the gap before a finance analyst does.
If you run it and find something, I want to hear about it. That is how this newsletter gets sharper. Grounded in what practitioners are actually seeing, not what I think they should be seeing.
The Signal
I am building the first industry benchmark on how enterprises actually discover problems in AI systems running in production. Your one-word answer contributes to that dataset.
This issue’s question: When your team finds a problem with an AI system in production, who typically discovers it first?
(a) The team that built it.
(b) The team that operates it.
(c) A customer or end user.
(d) An unrelated department.
One letter. Drop your answer in the comments. I am aggregating responses across enterprises and will share the pattern in a future issue.
Next issue: I just returned from NVIDIA GTC. Every vendor on the floor was selling the next agent platform. Not one of them mentioned what happens after deployment. I ran one announcement through the framework. The results were instructive.
Vijayan Seenisamy
Enterprise Agentic AI Systems Delivery | Creator, AI ROF™
Author, The AI Delivery Manager Blueprint
New issues arrive by email first. Subscribe here.
