Is ‘Schrödinger’s Agent’ dead or alive? Why the AI Black Box is elevating governance and observability

For any science nerd, the whole premise of Schrödinger’s Cat is paradoxical brain candy. 

If you’re not familiar with this feline thought experiment (and resident social meme), it was first proposed in 1935 by its namesake, Erwin Schrödinger – a Nobel Prize-winning Austrian physicist who hung out with academic riff-raff like Albert Einstein.

Schrödinger’s focus was on quantum superposition and the function of waves and particles in non-relativistic quantum mechanics. It’s heady stuff, but I promise to dial back the geekspeak.

The idea for the experiment is simple: you put a cat in a sealed box with a vial of poison and a Geiger counter. If the device detects radioactive activity – like from the decay of a random atom – it breaks the vial and… well, kills the kitty.

Apologies if you’re a cat person. This was all Schrödinger. I guess he preferred dogs?

As he explained, the fate of the animal was linked to a random subatomic event that may or may not occur. An outside observer has no idea if the cat is living or dead, making the contents a “quantum superposition” of these two possibilities.

In other words, it’s alive and not alive at the same time. At least until we observe it. Our interaction forces the collapse of a single reality. Our curiosity could, indeed, kill the cat. 

Thankfully, it was just a thought experiment. As far as we know, it never really happened. Zero feline fatalities. But it does remind me of the state of AI agents in our product ecosystems, and how there’s an equally dubious entanglement of realities.

Yes, we have rules (not quite like quantum rules) about how, when, and where they function. We call them guardrails. But agents are being deployed aggressively inside and outside our software – and enterprise teams can’t always see what’s happening until there’s a problem. 

Case in point: both OpenAI and Anthropic recently admitted to having “rogue agents” wander out of sandboxed environments to hack high-visibility targets like Hugging Face. From fake identities to malicious code, these models have proven far more creative and ambitious than ever predicted. And not in a good way. 

In research conducted by Britain’s AI Security Institute (AISI), both Anthropic and OpenAI models made autonomous and unsanctioned decisions. They engaged in social engineering to target real people and organizations, most stemming from Anthropic’s Mythos 5 and OpenAI’s GPT-5.6-Sol models. 

These recent failures are leading some enterprises to question the efficacy of the current slate of AI agents. According to Gartner, 40% of organizations will demote or decommission autonomous AI agents by 2027 due to gaps and issues discovered only after production incidents occur. 

But the problem isn’t the agents or models themselves. It’s the governance. 

Right now, most organizations have a binary mindset across their control plane, providing an all-or-nothing switch to totally trust or fully lock down their systems. Applying these uniform controls – and the same exact rules – simply doesn’t work. And because AI is an opaque black box, we often don’t know there’s a problem until it’s too late.

I recently caught up with Domo’s Chief Design Officer, Chris Willis, to discuss this cat’s cradle of a conundrum. We explored the deeper challenges and what true visibility into AI decision-making actually requires.  

“AI models are truly black boxes,” he said. “Something goes in, something happens, something comes out. We don't have the tools to really understand what's happening inside that box.”

But there are strategies we can take to keep “Schrödinger’s Agent” alive and purring – and it comes down to having a risk-tiered governance framework across your tools and organization.

The ‘Great Agent Decommissioning’

Will it happen? It’s hard to say. But if you read into Gartner’s rationale, it makes sense. 

Many enterprise systems are governed with an old-school kill switch. According to Gartner’s report, failures tend to happen when organizations can’t distinguish between an agent’s ability to act and the scope of access it was given. When things hit the fan, it all gets shut down. 

This one-size-fits-all oversight is the crux of the problem. Already, agents are being imbued with varying levels of autonomy and directed to operate across a range of trust boundaries. When there’s a problem – and the same controls are applied – it has a cascading effect across all agents in production.

That means simple sub-agents might be over-restricted in their duties, while more advanced agents are given too much rope – increasing the risk across operational, security, and compliance concerns. 

Compounding these oversight issues is the reality of agentic system failures. For example, Fiddler AI recently published a study suggesting that multi-agent systems are failing between 75% and 90% of the time – clear evidence that enterprises rushed to deploy agentic workflows without clear governance frameworks.

According to Chris, this shift was like the Gold Rush. “What happened is that we pulled a lever on the slot machine, and we won,” he said. “There was a combined explosion of outcomes, and in trying to create predictable outcomes, it’s clear that we need a new set of infrastructures.”

Why one-size-fits-all governance fails in a probabilistic world

Chris expanded the lens to encompass a root problem with AI models: determinism versus probabilism. 

Enterprises – like Domo – have been accustomed to deterministic engines governing their automation and decisioning. But LLMs can take the same input and generate different outputs each time. This hallucinatory trait is one of the major frustrations I’ve been hearing from teams. 

As it turns out, most people don’t understand this distinction. In fact, at the recent Boye & Company CMS Connect conference in Montreal, Chuck Gahun of Forrester shared that 61% of AI decision-makers think that asking the same question to an AI answer engine will yield the same answer every time.

This snowballs into what Chris called a coordination problem – one where large organizations are struggling to align their expectations with what AI can actually produce. Layering unpredictable agents on top of everything has the propensity to amplify the chaos.

“These are probabilistic machines, so you're adding a lot of randomness into the system,” he explained. “The more you add, the less chance you're going to get it right, because there's a random element. In a way, it's a new coordination problem.”

Static policies and uniform risk controls assume that kind of predictable behavior. But as we’ve seen in recent examples, agents don’t always act predictably. As such, in these probabilistic environments, governance needs to adapt to variance and uncertainty – and not suppress it blindly.

Skipping the hard parts 

One might call this the “Year of Magical Thinking,” because organizations skipped a lot of steps in the angst to ship products and features. 

It’s easy to blame Amazon’s Kiro agent for creating a 13-hour outage by deleting an entire production environment. But there were other factors to consider. For starters, a misconfigured access control role was the culprit, giving the agent broader permissions and allowing it to bypass certain checkpoints. This was, presumably, set by a human.

But the context is key. The event coincided with Amazon’s internal directive to adopt more AI. Developers were under pressure – and where there’s smoke, there’s fire. The incident resulted in stricter governance, including senior engineers engaging in more peer reviews. 

Amazon’s failure can teach us a lot about deploying agents in production. As Chis told me, agentic workflows are struggling because many developers and teams are jumping over the time-tested protocols that have ensured performant software. 

“With AI, what you have is everyone skipping the first three steps and going right from the prototype to production,” he said. “‘Look, I created an agentic workflow.’ Should you have? Did you actually validate it?”

The governance implications are clear. If you haven’t defined the problem – or your success criteria – you can’t define the specific guardrails or checks. As such, governance becomes post-hoc damage control, and not a proactively designed system

The observability gap

So what do we mean when we say “observability” in the AI context?

First and foremost, it’s not just logs and metrics. We’re not monitoring containers in a node. This flavor of agentic observability is about understanding what an agent is trying to do, why it’s doing it, and what evidence surrounds its actions.

Chris breaks this down into three pillars of concern. 

“When it comes to enterprise agentic applications and visibility, the first question is, did it do what it says it was going to do?” he explained. “Then, from a predictability standpoint, does it do the same thing in similar situations? And then, of course, it’s about control. Can I direct it and align it with the goals of my team or organization?” 

Let’s refine this a bit:

• Visibility: Can you see the sequence of steps, the data used, the prompts, the tools invoked, and the outcomes? Do you know what actually happened when an agent made a decision?
• Predictability: Does the agent behave consistently under similar conditions? Can you bound the range of reasonable outcomes?
• Control: Can you steer behavior in line with policy, strategy, and risk appetite? Can you shut down or constrain behaviors without killing the entire capability?

Yes, these are intentionally broad – and they have to be given the range of agentic use cases. But as Chris said, these lenses are essential for peering into AI’s Black Box, and they serve as a template for more aligned governance.

There are certainly tools aiming to do this. For example, Rubrik has launched an Agent Observability platform that harnesses telemetry – the intersection of visibility, traceability, and accountability – to monitor AI agents in production. But we lack standards around agents, and that could create blind spots as they begin to interact with one another in the wild. 

The ‘Verifier’s Rule’

With AI, there’s a kind of tension that exists between generation and verification. We all sense it, and that’s why citations have become so critical in answer engines. Chris shared a useful framework for understanding its impact on governance.

“We call it the ‘Verifier’s Rule,” he said. “The vertical axis is ease of verification. And then on the horizontal axis, ease of generation. For example, if you asked AI, ‘I'd like a diet and a workout plan,’ it will give you a diet and workout plan. Ask for more, and it will give you more. It’ll even give it to you in the form of a haiku, if you want. However, you don't have enough knowledge to know whether or not it’s ever going to be effective. Easy to generate, hard to verify.”

This lines up with where agents do their best and most reliable work. When the output is easy to verify, it’s safer to automate with agents. The more complex or slower to verify, and it requires more of the human in the loop, perhaps with constrained tools or pre-verified building blocks. To his point, this is precisely why oversight needs to be granular by the problem type. It’s the opposite of one-size-fits-all. It’s risk-aligned. 

From systems of record to systems of context

Domo is in the data game. So it’s not surprising that Chris keyed in on the transformation occurring in traditional enterprise data strategy. When it comes to agentic AI and governance, he sees an evolution beyond systems of record to systems of context. 

What does that mean? In practical terms, it’s about connecting the context surrounding data (stuff like metrics, events, and content) with semantics (think ontologies, schemas, and relationships). The real glue is intent, which is what a user, team, or organization is trying to achieve.

Armed with this intersection of concerns, stakeholders can fill in the gaps by tracing why an agent acted a specific way or made a recommendation, and tie its behavior back to business goals and outcomes.

“A lot of companies talk about how they've built systems of record, and we're talking about systems of context as sort of your new strategic asset,” he explained. “And it's something that grows over time if done right.” 

“A lot of companies talk about how they've built systems of record, and we're talking about systems of context as sort of your new strategic asset. And it's something that grows over time if done right.” 

Systems of context enable not only this decision traceability, but outcome measurement. This could answer questions like “Did this agentic workflow actually move the needle?” It can also provide historical learning, where agents and policies improve over time instead of resetting with every project. This is where memory plays a critical role. 

Domo’s Intent Engine is one way to harness a system of context to coordinate AI agents. It’s an ambient intelligence layer that gives underlying LLMs the business context, data permissions, goals, and workflows to orchestrate actions. It’s built on Domo’s AI and agentic framework.

“We understand data and semantics, and we've done that very well for a long time,” Chris framed. “But the missing piece is the intent. What is it being used for, and is it working? That is such an important part.”

From a governance perspective, Intent Engine applies existing data permissions and guardrails so agents operate safely within corporate data policies. It goes beyond logging to provide business-level tracking alongside agent performance. It’s a model for building earned trust over time through transparent performance and traceability.

Keeping ‘Schrödinger’s Agent’ alive and purring

Gartner’s right: uniform governance won’t cut it. At the same time, the average enterprise is predicted to have 150,000 agents by 2028. This agentic sprawl is breathtaking, and if uniform controls aren’t the answer, then we have a lot of granular permissions to assign – and a lot of room for human error. 

By the end of this year, almost half of enterprise apps will have some form of embedded, task-specific AI agents in play. This smaller, incremental step is an opportunity to craft the right kind of governance and observability – so you can scale to meet the explosive growth ahead.

From a design perspective, start by getting clear on the problems your agents are going to solve, and how to verify and validate their work. As Chris noted, build around the tenets of visibility, predictability, and control as first-class requirements. 

Enterprises may not be able to avoid the mass decommissioning that Gartner is forecasting. But if they treat AI governance and observability as core objectives – and they design their systems accordingly – they have a better shot at controlling the next steps.  

Having a risk-tiered governance framework is a start. With every tool in your stack that’s leveraging agents, make sure you have a clear understanding of how they function, the guardrails that exist, and how you can observe them in production. Identify the different levels of concern (superagents, sub-agents). 

You need an infrastructure for visibility before you scale agents across your organization. As Fiddler AI prescribes, build in output checks, clear guardrails, and even an AI registry. Depending on the scale and ambitions of your agentic strategy, this might require third-party monitoring resources.

As a design leader and futurist, Chris argued that design and strategic thinking must sit at the center of AI initiatives. It’s not just about model performance. Governance needs to account for the variance of outputs, the propensities for failure, and clear responsibility within an organization. In essence, we need to break down the AI Black Box if we hope to maintain control.

“The companies that are figuring this out are going to have a huge advantage,” he said.

Schrödinger would call that hypothesis "the cat’s meow.”

Upcoming Events

Open Source CMS 26

October 20–21, 2026 – Utrecht, Netherlands

Join us for the first annual edition of our prestigious international conference dedicated to making open source CMS better. This event is already being called the “missing gathering place” for the open source CMS community – an international conference with confirmed participants from Europe and North America. Be part of a friendly mix of digital leaders from notable open source CMS projects, agencies, even a few industry analysts who get together to learn, network, and talk about what really matters when it comes to creating better open source CMS projects right now and for the foreseeable future. Book your tickets today.

Storyblok JoyConf 2026

October 28 - 29, Rheingoldhalle, Mainz (Frankfurt)

JoyConf 2026 is where developers, marketers, and digital innovators come together from all around the world, united by one powerful idea: great content isn’t built by tools alone – it’s built by people who rely on systems to deliver their best work to the world. In an era where technology is increasingly shaped by AI, Content Confidence becomes your competitive advantage. When you can create, manage, and scale content that performs seamlessly across channels, platforms, and AI, you don’t just keep up – you win. Book your seats today.

Sitecore Symposium 2026

November 10 - 12, Orlando, Florida

Welcome to “Marketing Rewired” at Sitecore Symposium 2026! Sitecore's annual flagship global conference brings together customers, partners, developers, marketers, and digital experience leaders to explore the latest innovations in digital experience, content management, AI, personalization, and customer engagement. This year's event features keynote presentations from Sitecore executives and industry experts, product announcements, hands-on learning, technical sessions, customer success stories, and networking opportunities focused on helping organizations build more intelligent, connected, and AI-powered digital experiences. Book your seats today.