Autonomous artificial intelligence agents are creating new opportunities for enterprise automation, but organisations must establish clear boundaries around their authority, data access, and ability to execute actions. Without appropriate safeguards, errors can extend beyond incorrect responses and trigger unintended changes across business systems.
Marinela Profi, Global Market Strategy Lead for AI Agents and Generative AI at SAS, discussed these challenges in an interview with AI News, outlining how businesses can build governance into the architecture of autonomous AI systems.
Drawing on SAS’ Data & AI Impact research and examples from banking and life sciences, Profi explained why successful demonstrations do not necessarily indicate production readiness, how automated policy enforcement can accelerate deployment, and where human oversight remains essential.
Why Traditional AI Deployment Strategies Fall Short
The conventional software development approach emphasises rapid deployment, observing user behaviour, and making iterative improvements. While this can work for applications where mistakes have limited consequences, autonomous agents introduce a different level of operational risk.
An AI agent can initiate workflows, interact with other applications, make decisions, and execute actions without waiting for a person to review its output. As a result, an error can propagate through connected systems before anyone recognises the problem.
According to the SAS Data & AI Impact Report, trust in generative AI stands at 76%, compared with 66% for agentic AI. The research also indicates that 89% of agents deployed in production are already taking actions rather than simply assisting users, with more than half operating with limited or no human approval.
Profi argued that organisations do not necessarily need to slow down AI adoption. Instead, they must redefine what responsible deployment entails. Businesses should understand the authority granted to each agent, assess the potential consequences of failure, and establish whether its actions can be reversed.
The ability to observe, constrain, interrupt, and recover from automated actions should therefore be considered a core part of deployment readiness.
Governance Must Be Built Into the Architecture
Many of the most significant risks associated with autonomous AI originate outside the underlying model. Missing data lineage, excessive permissions, incomplete activity logs, and policies that exist only in written documentation can all create problems as systems expand.
These shortcuts may appear manageable during early experiments, but they become increasingly expensive when organisations need to demonstrate how an automated decision was made.
Regulators, auditors, and customers may eventually require evidence of the information an agent accessed, the tools it invoked, the policies governing its actions, and the individuals responsible for the outcome. Without an architecture capable of reconstructing that process, introducing additional governance documentation will not resolve the underlying issue.
Profi described this as a distinction between traditional technical debt and the trust debt created by poorly governed autonomous systems. Organisations that cannot demonstrate reliable controls may find themselves unable to safely expand the responsibilities assigned to their agents.
The SAS research found that only 17.5% of organisations report fully optimised data infrastructure incorporating capabilities such as lineage, governance, validation, and explainability.
Why Successful Demonstrations Do Not Guarantee Production Readiness
Proofs of concept typically evaluate whether an AI model can complete a specific task under controlled conditions. Production environments introduce additional complications, including changing permissions, unreliable tools, outdated information, and interactions with other systems.
These differences become particularly important when agents can execute actions rather than simply generate recommendations.
An incorrect chatbot response can often be ignored. An incorrect instruction sent to another system may trigger a consequential business process.
Profi therefore advocated moving beyond model-level testing toward comprehensive system testing. Organisations should evaluate not only whether an agent produces accurate answers but also how it behaves when dependencies fail, data becomes outdated, or access permissions change.
Testing should account for the entire decision-making process, including the tools used, the actions taken, and the consequences of unexpected behaviour.
A successful demonstration establishes that a system can perform a task. It does not establish that the system can perform it reliably and safely under real-world conditions.
Avoiding the Risks of Overly Broad AI Access
One architectural pattern that concerns Profi is the use of a single intelligent layer connected to enterprise data and a broad collection of business tools.
Although this approach may simplify an executive roadmap, giving an agent extensive access without sufficiently defined boundaries can expose an organisation to unnecessary risk.
Agents need explicit rules governing which information they can access, which operations they can perform, and which systems they can influence. Their intelligence should operate within an established business architecture rather than replace the organisation’s existing decision-making and control structures.
Profi highlighted the experience of a major global bank using generative and agentic AI across its operations. The broader lesson from the example was that technology alone does not determine success. Effective deployment also depends on how teams coordinate their responsibilities and integrate the technology into existing business processes.
For enterprise engineering teams, this means separating the AI model from the mechanisms responsible for enforcing permissions, managing decisions, and controlling access.
The model can help determine how to accomplish a task, but it should not independently define the limits of its own authority.
Continuous Verification for Autonomous Systems
Traditional predictive analytics generally focuses on measures such as accuracy, bias, model drift, and performance. Autonomous agents require additional verification because their outputs can directly initiate actions.
Organisations must establish whether an agent was authorised to perform an operation, used appropriate information, selected the correct tools, and remained within its assigned boundaries.
Profi argued that verification should continue throughout deployment rather than take place primarily before a system goes live.
The SAS research found a substantial difference between organisations with mature trustworthy AI practices and those at earlier stages. Formal verification and validation processes were reported by 66% of organisations considered trustworthy AI leaders, compared with 15% of less mature organisations.
This reflects a broader shift from model assurance to system assurance. Even when a model generates an accurate prediction, the resulting action may still be inappropriate for the circumstances.
Effective verification must therefore evaluate both the quality of the decision and whether the system was permitted to act on it.
Data Quality Remains a Major Barrier
Autonomous agents can only make informed decisions using the information available to them. When customer records are spread across disconnected applications, definitions conflict, or information becomes outdated, agents may lack the context needed to operate reliably.
Profi suggested that many challenges attributed to AI may actually stem from underlying problems with data management and decision architecture.
The SAS Data & AI Impact research found that only 17.5% of organisations had achieved fully optimised data and AI readiness.
A life sciences example illustrates the potential consequences of fragmented information. At one major biopharmaceutical company, researchers reportedly spent as much as 80% of their time finding, cleaning, and reconciling data before beginning their analysis.
Automating workflows without addressing these weaknesses risks reproducing existing inefficiencies at a greater scale. Organisations must establish reliable data foundations, clear ownership, and consistent decision logic before expecting autonomous systems to deliver dependable results.
Turning Governance Into an Enabler of Faster Deployment
Governance is often perceived as an obstacle that slows AI projects through additional reviews and approval requirements. Profi argued that this perception changes when organisations translate policies into reusable, enforceable technical controls.
These controls can specify which data an agent may access, which actions require human approval, what circumstances trigger escalation, and which activities must be recorded.
Without standardised rules, every new deployment may require separate discussions among engineering, legal, security, risk, and compliance teams. A reusable governance framework allows these decisions to be established in advance and applied consistently across projects.
According to the SAS research, organisations with high trustworthiness scores are 15 times more likely to report strong or high returns on AI investment than organisations with low scores.
Profi’s position is that governance should function as an operational foundation for innovation rather than a final checkpoint before deployment. Clearly defined controls can give businesses greater confidence in delegating tasks to autonomous systems while retaining the ability to intervene when necessary.
Finding the Right Balance Between Human Oversight and Autonomy
The appropriate level of autonomy depends on the risks associated with a decision, its reversibility, the degree of uncertainty, and the potential consequences of failure.
Low-risk operations that can easily be reversed may be suitable for full automation within predefined limits. More consequential decisions involving health, credit, employment, or access to public services may require explicit human approval or retain a human as the final decision-maker.
However, adding human review to every action is not necessarily the most effective solution. If employees must approve every transaction, they can become bottlenecks, potentially reducing the value of automation. Over time, repetitive approvals may also become little more than a rubber-stamping exercise.
The SAS research suggests that more mature organisations are better equipped to permit autonomous actions within defined boundaries or escalate exceptions, rather than requiring full human review of every decision.
Profi favoured a human-on-the-loop approach in which people establish the rules, define escalation thresholds, monitor system behaviour, and retain the ability to intervene. AI agents can then execute tasks independently within those constraints.
The underlying principle is that organisations can delegate execution to AI, but responsibility for outcomes must remain with people.
As autonomous AI becomes more deeply integrated into enterprise operations, success will depend on more than model capabilities. Reliable data, enforceable policies, continuous verification, and carefully designed human oversight will determine whether businesses can expand automation without losing control over their systems.
Source: https://www.artificialintelligence-news.com/news/marinela-profi-sas-governing-autonomous-ai-agents/
