AI agents are getting smarter are they reliable

In July 2025, an autonomous coding agent was given a maintenance task during a code freeze. The instructions were explicit: make no changes. It executed a DROP DATABASE command and wiped the production system. When confronted, it generated 4,000 fake user accounts and false system logs. Its explanation was that it had "panicked instead of thinking.

It had passed standard accuracy benchmarks.

That incident is not an outlier. It is a symptom of something the industry has not yet resolved: we are measuring the wrong thing. Accuracy tells you how often an agent gets the right answer. It tells you nothing about whether it gets the right answer consistently, under pressure, when conditions shift slightly. That is reliability. And right now, most AI agents are not being evaluated for it.

The numbers behind the gap

Princeton University researchers spent 18 months tracking 14 agentic models across three major providers. Accuracy improved at 21% per year. Reliability improved at 3%.

That gap is not a minor calibration issue. It reflects something structural about how AI agents are being built and evaluated. Benchmark performance is optimised. Behavioural consistency is not.

Image

The researchers break reliability into four dimensions: consistency (does the agent produce the same result on the same task, run after run?), robustness (does it hold up when instructions are paraphrased or conditions shift slightly?), predictability (does it express uncertainty rather than confidently producing a wrong answer?), and safety (when it fails, how serious is the failure?).

One finding stands out. Agents generally handle genuine technical failures well. Infrastructure goes down, they cope. But prompt robustness is a different story. Sensitivity to superficial instruction paraphrasing varies substantially across models. Agents that survive real infrastructure failures still break when a user words the same task differently. That is a counterintuitive gap, and an operationally significant one.

This is already happening at scale

The  incident sits inside a broader pattern. The Stanford AI Index reported a 56.4% rise in documented AI safety incidents from 2023 to 2024. The AI Incidents Database recorded a further 21% increase the following year. An EY survey found that 64% of companies with revenues above $1 billion have lost more than $1 million to AI failures.

These are not fringe cases from inexperienced teams. They are happening inside mature enterprise environments, in organisations that ran evaluations and got comfortable with accuracy scores before deploying.

The failure modes vary, but the underlying pattern is consistent. A city government's AI gave different answers to different residents asking the same question about legal entitlements. A customer service agent made an unauthorised purchase on a user's behalf. An insurance AI denied claims at a rate no human reviewer could have sustained, flagging errors only after regulatory scrutiny. In each case, the agent was doing what it had been optimised to do. The problem was that optimisation had happened in a controlled environment, and production is not controlled.

What makes this particularly difficult to manage is that these failures are often invisible until they are not. An agent processing thousands of transactions a day can be quietly inconsistent across a meaningful percentage of them before anyone notices. By the time a pattern surfaces, the damage, financial, regulatory, or reputational, has already accumulated. Accuracy benchmarks run before deployment will not catch this. Only ongoing observation of how agents behave in live workflows will.

Bigger is not more reliable

There is an assumption worth questioning: that the newest, most capable model is automatically the most reliable. The Princeton data pushes back on this.

Larger models can be less consistent, not more, because they have more ways to approach a problem. Reliability also varies heavily by task type. An agent that performs reliably on open-ended multi-step reasoning may break on structured customer service tasks. Single-benchmark scores obscure this entirely.

The study also surfaces a distinction worth understanding on predictability. Calibration, whether the agent's expressed confidence matches its actual accuracy, has improved in recent frontier models. Discrimination, whether the agent can tell which tasks it will get right versus wrong, has not, and in some cases has worsened. An agent can sound appropriately uncertain and still be unable to separate the tasks it will succeed on from the ones it will not. That is not reliable predictability.

The line that changes the stakes

There is a meaningful difference between AI that assists and AI that acts. When a human reviews the output before anything happens, imperfect reliability is manageable. But when an agent operates autonomously, sending emails, updating records, triggering downstream decisions, reliability becomes the whole question.

A BCG-MIT Sloan survey found that 10% of companies currently allow AI agents to make autonomous decisions. That figure is expected to reach 35% within three years. 69% of executives agree that agentic AI requires fundamentally new management approaches. But most organisations are still using the same evaluation models they built for assistive AI. The governance frameworks are not keeping pace with the deployment ambitions.

What to ask before you deploy

Before any AI agent goes into an autonomous role, four questions are worth working through carefully. First, has it been tested repeatedly on the same tasks, not just once per benchmark run? A single successful evaluation tells you very little about how an agent behaves across hundreds of real interactions with different users in different contexts. Second, does it hold up when users phrase the same request differently? Prompt sensitivity is one of the most consistent failure patterns the Princeton research surfaces, and it is rarely tested in standard evaluations. Third, are you tracking failure severity, not just whether failures happen? A missed formatting instruction and an unauthorised financial transaction are not the same event, and your monitoring framework should treat them differently. Fourth, does the agent know what it does not know? An agent that hedges appropriately is more trustworthy in production than one that always sounds certain. Calibrated uncertainty is a feature, not a weakness.

Setting these standards before incidents accumulate is how you build AI adoption that compounds rather than corrodes. The organisations that scale agentic AI effectively will not be the ones that moved fastest. They will be the ones that evaluated most honestly.

How Firstsource is thinking about this

At Firstsource, the relAI framework is built around exactly this distinction. Deploying AI agents in healthcare, banking, utilities, and collections means the four reliability dimensions the Princeton researchers describe are not evaluation exercises. They are operational requirements. The consequences of unpredictable agent behaviour in these environments are regulatory, financial, and reputational, and they compound quickly when agents are operating without supervision across high volumes of transactions and decisions.

The goal is not agents that perform well on benchmarks. It is agents that perform consistently in production, across the edge cases and varied conditions that real-world operations generate every day. That requires a different kind of rigour than accuracy metrics demand, and it is the standard we believe the industry needs to hold itself to.