As organizations look to scale their agentic AI initiatives, they’re discovering a hard truth: Traditional command-and-control governance doesn’t work for agent-driven autonomous systems.
Leadership teams need to update their governance playbooks so that they have the proper controls, visibility and accountability in place to capture the benefits of AI agents — while mitigating the added risk.
“The utility of agents conflicts a bit with traditional governance policies because agents can interpret and make decisions by themselves,” says David Guarrera, Principal, Technology Consulting, Ernst & Young LLP. “Their ability to act under uncertainty is their core value, but that creates tension with governance because they can be unpredictable.”
AI agents introduce new forms of risk that organizations did not face with previous generations of emerging technology, such as cloud computing. Those risks include:
- Hallucinations: The tendency of large language models (LLMs) to make things up is well documented. While hallucinations have decreased as the models evolve, even a 98% accuracy rate is not sufficient for business-critical workflows, such as finance.
- Prompt injection: Bad actors can use specially crafted prompts to trick LLMs to override existing rules and execute malicious commands. These types of attacks can create significant privacy, financial and reputational risk, particularly with customer-facing agents.
- Hidden costs: Multi-agent workflows, left unchecked, can get caught in infinite loops that dramatically increase token use and drive up costs. Unexpected costs can eat into ROI projections and erode leadership confidence in AI investments.
Addressing these challenges requires a new model for assessing and mitigating risk. EY Consulting professionals call it a “trust layer” playbook.
“The old paradigm of ‘test and deploy’ is being replaced by ‘monitor and adapt continuously,’” says Guarrera.
This new paradigm of continuous monitoring and adapting requires organizations to build trust into the infrastructure of how the business runs, not just into a policy document that sits adjacent to it. This shifts governance practices from oversight to architecture.
A trust layer playbook for agentic AI centers on three primary principles:
- Establish clear decision rights based on how much freedom an agent has. Every use case for deploying agents requires its own risk assessment. Teams will need to assess permissions and data access for agents — and account for agents that may attempt to actively seek to work around or change permissions in order to complete an assigned task.
- Implement real-time monitoring systems and processes to track AI performance patterns. Observability is critical for identifying hallucinations, infinite loops or other risk signals. Guarrera suggests establishing “accuracy” as a managed metric for measuring performance drift from agents. Real-time monitoring helps encode compliance — with every action signed, traced and reversible, giving humans executive oversight.
- Build escalation paths so that high-impact or strategic calls automatically go to humans. For early-stage agentic AI deployments, it’s better to err on the side of more human oversight throughout the process. “Until you can take the training wheels off, more human in the loop is better,” says.
Governance in the age of agentic AI requires a new playbook. Establishing new governance frameworks that maintain appropriate oversight while enabling AI autonomy moves beyond traditional command-and-control structures. Organizations need frameworks that establish clear boundaries for different levels of AI decision-making, implement real-time monitoring systems that track AI performance patterns, and create escalation protocols that bring humans into the loop for strategic decisions and maintain accountability for all outcomes.
“The goal isn’t to slow things down,” says Guarrera. “It’s to make things safe to go faster.”
Learn more about how EY Consulting teams help organizations establish new AI governance frameworks as part of a holistic Responsible AI strategy.
The views reflected in this article are the views of the author and do not necessarily reflect the views of Ernst & Young LLP or other members of the global EY organization.