The world's most sophisticated firms are deploying thousands of AI agents. Not one of them can prove those agents are working.
The Failure Anatomy
Two weeks ago, I published an analysis of McKinsey's claim that they had deployed 25,000 AI agents across their operations. The post reached over 150,000 enterprise professionals. What happened in the comments told me more than the article itself.
The reactions split into three camps almost immediately. Some celebrated the number as proof that agentic AI had arrived at scale. Others questioned what McKinsey was actually counting. Were these autonomous agents or glorified LLM workflows? But the most revealing responses came from senior leaders who zeroed in on a question nobody at McKinsey had answered: how do you know those 25,000 are actually working?
A CFO with board-level and special operations experience wrote that enterprises should govern AI agent changes the way they govern revenue recognition, with formal controls, audit trails, and defined thresholds. A senior technical programme manager pushed back on counting deployments entirely: "Stop counting deployments. Start measuring what you actually govern." An engineering leader at Microsoft summarised it in one line: AI maturity is operational discipline, not agent volume.
Three senior professionals. Three different measurement philosophies. Over 400 reactions and 100 comments from leaders at Deloitte, EY, Accenture, JPMorgan Chase, and Microsoft. And not one person, across any of these organisations, could point to a shared standard for what "working" means for an AI agent in production.
That conversation crystallised something I had been seeing from inside enterprise AI delivery for two years. This is not a technology gap. It is a delivery discipline gap. Enterprises are deploying agents using frameworks designed for traditional software, systems where the same input always produces the same output, where failures are visible and reproducible. AI agents are probabilistic. They degrade silently. Their outputs shift as data changes. They produce different answers to the same question on different days. And the measurement systems built over decades of traditional software delivery cannot see any of it.
The data confirmed this is not one company's problem. It is structural. MIT research across 150 executive interviews and 300 public deployments found that 95% of generative AI pilots fail to deliver measurable business impact. Camunda's 2026 survey of 1,150 senior IT leaders found only 11% of agentic AI use cases reached production. S&P Global reported that 42% of companies scrapped most AI initiatives in 2025, more than double the 17% from the year before. Abandonment is accelerating, not slowing.
That McKinsey post, and the hundreds of responses it generated, confirmed a thesis I have been building frameworks around since I started working inside this problem. The delivery discipline for AI agents in production does not exist yet. That conviction led me to write a book, build a delivery framework called AI ROF, and begin advising enterprises on closing the gap. This newsletter is how I share that work: the patterns, the failure modes, and the field tests that separate organisations drifting toward failure from those building real operational discipline.
Every issue follows the same structure. A real failure pattern. A framework lens to diagnose it. A field test you can run this week. And a signal from the field.
The Framework Lens
The measurement void exists because enterprises evaluate AI agents through a single lens. They measure cost, or accuracy, or speed, and call it success. But a system that is cheap and fast while silently violating compliance rules is not succeeding. It is a liability waiting to surface.
Delivery capability for AI agents must be assessed across five distinct dimensions simultaneously. A weakness in any single dimension eventually surfaces as a production failure, often months after deployment when the damage is already compounding.
Business. Most enterprises cannot answer a basic question: is this agent delivering measurable value against a metric the CFO would recognise? The business case that justified the pilot is rarely revisited once the system goes live. Twelve months later, nobody can trace agent output to revenue impact, cost reduction, or customer outcome. The board sees investment. They do not see return.
Integration. Agents do not operate in isolation. They depend on data pipelines, APIs, and upstream systems that change independently. In a pilot, the data is curated and the connections are stable. In production, an upstream schema change at 2am can silently break an agent that processed 10,000 transactions yesterday without error. Integration failure is the silent killer, and it never appears in a demo.
Governance. Air Canada was held legally responsible for misinformation generated by its customer service chatbot. The accountability rested with the enterprise, not the model. Most organisations deploying agents today have no governance structure designed for probabilistic systems. They apply compliance checklists built for traditional software to systems that require continuous oversight. The question is not whether a regulatory review will come. It is whether you would survive one.
Observability. A 2025 industry survey of 1,300 AI professionals found that 89% have some form of agent monitoring, but only 52% run actual evaluations against baselines. Dashboards full of green lights mean nothing if nobody is testing whether the outputs are still correct. Monitoring tells you the system is running. Evaluation tells you the system is working. Most enterprises have the first and lack the second.
Orchestration. Who evaluates agent outputs? How often? What triggers an escalation? What is the release cadence for model updates? Most enterprises have no defined delivery rhythm for AI systems in production. The team that built the agent shipped it and moved on to the next project. Nobody owns what happens after go-live.
These five dimensions are not a checklist. They are the axes across which delivery capability must be assessed and matured continuously. The measurement void closes when enterprises stop asking "is the agent working?" and start asking five separate questions, one for each dimension.
The Field Test
Take 60 seconds this week. Pick one AI system in your organisation, the one closest to production or already live, and answer five questions.
1. Evaluation cadence. When was the last time your team ran a formal evaluation of this agent's output quality against a defined test set? Not monitoring. Not dashboards. An actual comparison of outputs to known-good baselines.
2. Cost per outcome. Do you know the cost per successful outcome for this agent? Not cost per API call. Cost per task completed correctly, including retries, fallbacks, and human escalation.
3. Accountability. Can you name the single person accountable for this agent's reliability, compliance, and cost in production? Not the team that built it. One name.
4. Drift detection (output degradation over time). Does your team have a scheduled process to compare current agent output against a baseline from 30 or 60 days ago? Yes, no, or not sure.
5. CFO explainability. Could you walk your CFO through this agent's business impact, cost structure, and risk profile in under five minutes, with data they would trust?
If you answered "no" or "not sure" to three or more, you have a measurement void. You are not alone. Most enterprise teams would answer the same way. That is the problem this newsletter exists to diagnose.
The Signal
I am collecting one data point per issue from enterprise AI teams worldwide to build a picture no individual organisation can see alone.
This issue's question: How many weeks since your team ran a formal agent evaluation? Zero counts. Never counts. Drop your answer in the comments
I am aggregating responses across enterprises and will share the pattern in a future issue, anonymised and without attribution. This is the beginning of an industry baseline that does not exist yet.
I am building the practitioner toolkit I wish existed when I started this work. Agent Portfolio Dashboard, Governance Cadence Templates, Cost-Per-Task Calculator, Maturity Quick-Assessment, and an Implementation Guide to deploy them all in 30 days.
Launching March 2026. If you want early access and introductory pricing, the waitlist is here: https://vj-x-ai.kit.com/agentic-ai-toolkit
Next issue: The Demo God Curse. Why the perfect pilot is the most dangerous one to scale. If you have ever watched a flawless demo lead directly to a production disaster, that one is for you.
Every issue of this newsletter includes a diagnostic you can run inside your organisation. If the measurement void resonated, my free guide "The First 90 Days as an AI Delivery Manager" goes deeper into the operational structures that close it. I send it to email subscribers along with frameworks that do not appear here. Link in my Featured section to subscribe or download directly here: https://vj-x-ai.kit.com/first-90-days
Vijayan Seenisamy Enterprise Agentic AI Systems Delivery | Creator, AI ROFTM Author, The AI Delivery Manager Blueprint
New issues arrive by email first. Subscribe here.
