About Our Contributing Expert
Darpan Vyas | Product Manager
Darpan Vyas is a Technical Product Manager at The Modern Data Company, focused on building enterprise data infrastructure and data products. With a computer science background, he specializes in taking complex, ambiguous ideas from zero to one and turning them into scalable, usable products.
Darpan works across the data product stack, spanning data movement, metadata, governance, transformation, quality, semantics, and consumption. In his current company, he has played a key role in taking the data movement engine from concept to general availability, supporting batch and CDC across diverse enterprise data sources.
His interests sit at the intersection of data platforms, product strategy, and AI-ready infrastructure, with a particular focus on how enterprise data systems can provide the context, governance, and operational foundations needed for reliable AI and agentic systems. We’re thrilled to feature his insights on Modern Data 101.
We actively collaborate with data experts to bring the best resources to a 20,000+ strong community of data leaders and practitioners. If you have something to share, reach out!
🫴🏻 Share your ideas and work: community@moderndata101.com
*Note: Opinions expressed in contributions are not our own and are only curated by us for broader access and discussion. All submissions are vetted for quality & relevance. We keep it information-first and do not support any promotions, paid or otherwise!
What Makes an AI Agent Reliable
Start with a principle that has nothing to do with AI. The performance of any component depends on the system it operates inside. A well-indexed table returns bad answers if the query planner routes around the index. A well-designed API fails in production if the network between caller and service is unreliable. Competence at the component level is necessary but never sufficient; the surrounding structure decides whether that competence becomes an observable outcome.
Language models are a component. A raw model reasons and generates text, nothing more. Whether that reasoning becomes a working system depends on what surrounds it: what instructions it is given, what tools it can invoke, what state it can read or write, what permissions bound its actions, what feedback tells it when it is wrong, what limits stop it before a mistake compounds. Call this surrounding structure the runtime. The relationship is simple enough to state as an equation:
Agent = model + harness (system around the model)
When an agent performs reliably, not once in a demonstration but across thousands of unsupervised executions, the runtime deserves most of the credit. A 2026 study out of Stanford, MIT, and KRAFTON reached a related conclusion from a different angle: an automated system that searched over the code surrounding a fixed model discovered configurations that outperformed hand-built defaults on coding, math, and classification tasks, without touching a single model weight.
Vivek Trivedy documented a sharper version of the same finding. His team ran an experiment on Terminal Bench 2.0, a benchmark that scores how well coding agents complete real terminal tasks.
They held the model constant, GPT-5.2-Codex, and changed everything around it: the prompt, the available tools, the middleware, how context was injected, and how output was verified. In his own write-up, the score moved from 52.8 percent to 66.5 percent. The ranking moved from outside the top thirty to the top five. No new parameters were trained.
The system around a fixed intelligence was rebuilt, and the outcome changed accordingly.
Coding agents made this pattern visible first, because coding benchmarks are easy to score. The pattern itself generalizes well past code, a point several practitioners have since made in more depth.
An enterprise agent operating on organizational data needs the same categories of support: instructions it can trust, tools with bounded scope, permissions it cannot exceed, context relevant to the task at hand, a way to verify its own output, and a record of what it did.
Most of the infrastructure needed to supply these things is not missing from the modern data platform. It already exists, built originally for other purposes: governance, lineage, access control, observability.
The open question is not whether an enterprise has the pieces. It is whether those pieces are wired into the agent’s execution path, or sitting beside it as documentation nobody consults at the moment it would matter.
That distinction, between owning a control and putting it in the loop, is where most enterprise AI architecture will succeed or fail.
What Your Data Platform Already Has What AI Agents Need
Break the runtime into its load-bearing parts, and each one has a direct counterpart already sitting inside a mature data platform, particularly one built around data contracts rather than one-off pipelines.
Instructions, in the coding-agent sense, define conventions and constraints the agent should follow without being told every time. In a data platform, this is the job of the semantic layer and its active metadata.
A semantic layer is, at bottom, an organizational agreement about meaning: what net revenue includes and excludes, which source system is authoritative for a given entity, whether a metric is valid for a particular region or time window. Agreements like this are simple to write down once and surprisingly hard to keep enforced as an organization scales.
For instance, one implementation of this idea exposes that agreement as a queryable model rather than a document that only humans read. An agent asking about revenue should resolve the same definition every time, regardless of which session or which tool issued the question.
This is not just an engineering convenience. A 2026 paired benchmark across three frontier models found that giving a model a governed semantic-layer document alongside the raw schema raised first-shot accuracy by 17 to 23 percentage points, and closed nearly all of the performance gap between the three models, because they were reasoning over pre-verified business logic instead of reconstructing it from raw tables.
Context management, in a coding runtime, selects the smallest useful slice of information rather than dumping an entire codebase into the prompt. The data-platform equivalent is a service that assembles the relevant definitions, lineage, and prior usage for the specific task in front of the agent, not a full catalog export.
More metadata is not automatically more useful. Irrelevant fields compete for the same attention a model has available for the fields that actually matter.
Tools, in both settings, are bounded ways to act. In an enterprise context, a tool should be a governed interface to a data product: a typed, contracted surface with a declared owner and a declared consumer, not a raw warehouse credential handed to the agent with instructions to be careful.
A data product built this way already carries most of what an agent would otherwise have to reconstruct on its own: schema, freshness guarantee, lineage, and the semantics that explain what the numbers mean.
Hooks and policy gates, in the coding-agent world, run deterministic checks before or after an action executes. In the data world, a contract and an access policy already do this, provided they run at the moment of use rather than only at design time.
A policy should validate schema, freshness, purpose, entitlement, and budget before a query executes, and it should return a structured reason when it fails, something the agent can act on, not a message meant only for a human reading a log later.
Sandboxes limit blast radius. Scoped compute, short-lived credentials, row- and column-level controls, and isolated workspaces already do this inside most data platforms; the requirement is that an agent’s session inherits the same scoping a careful analyst would be given, not broader access granted for convenience.
Evaluators and observability test whether a result matches intent. Lineage, execution traces, data quality signals, and cost telemetry are the evidence plane already generated by most platforms, originally for entirely different reasons.
The mapping is useful, but an analogy is not an architecture. A control that exists inside a platform and a control that participates in every consequential decision an agent makes are two different things. The gap between them is closed only by wiring, not by naming.
A Real Example: How an AI Agent Should Handle a Customer Refund Request
Consider an agent asked by a support manager to investigate a spike in refund requests and recommend which affected customers qualify for a proactive credit.
The model can decompose that request into subtasks. It should not decide, on its own authority, what “refund,” “affected,” or “qualify” mean, and it should not have unrestricted access to customer records or the standing authority to issue money.
Here is what the surrounding system needs to do, in order.
Bind identity and purpose. The request carries the manager’s identity, the agent’s own identity, a declared purpose (support investigation), and a correlation ID that will tie every downstream action back to this request. The agent acts as a named principal, not as an anonymous extension of someone else’s access.
Resolve semantics. A context service returns the approved refund metric, the market definition for Germany, the relevant time zone, and the authoritative order and payment data products along with their current freshness status.
Plan with bounded tools. The agent can query approved aggregates, inspect lineage, and request a customer-level cohort through a defined interface. It cannot issue arbitrary queries against the warehouse.
Evaluate policy before execution. The cohort request reaches a policy decision point, a concept formalized in attribute-based access control standards long before agents existed. The decision weighs the manager’s entitlements, the stated purpose, the sensitivity of the requested fields, and any geographic restriction. The outcome is one of allow, deny, mask, or require approval, not a plain yes.
Meter the work. Query cost, tokens consumed, retries, and any downstream API calls are attributed to this specific workflow and checked against a budget while the work is still in progress, not after the invoice arrives.
Verify the result. The agent’s explanation is checked against source freshness, the actual query results, and the approved definition of the metric. A separate evaluator can independently test whether the stated conclusion is actually supported by what was retrieved.
Gate the action. Drafting a credit policy is low risk and can proceed on its own. Issuing an actual credit is not, and it should require human approval and execute through a narrowly scoped, transaction-specific tool with hard limits on amount and customer count.
Record the chain. Identity, the version of every metadata definition consulted, every policy decision, every query, every cost, every approval, and every output are logged under the same correlation ID established in step one.
None of these eight steps is exotic. Every one of them already exists in some form inside a mature data platform: identity and purpose binding through access management, semantic resolution through a semantic layer, bounded tools through governed data products, policy decisions through an access-control engine such as Bifrost, metering through cost attribution, verification through lineage and quality checks, gating through approval workflows, and recording through existing audit logs. What changes is that these controls now sit inside the agent’s execution path instead of beside it.
What Your Data Platform Can’t Control on Its Own
There is a boundary worth stating plainly, because overstating the claim is a fast way to lose credibility with the people who have to implement it.
A data platform cannot govern an action it never observes. If the refund agent can call a SaaS admin API directly, browse the open web, or post to a messaging channel outside governed infrastructure, then data-layer controls have nothing to act on. Identity federation, an API gateway in front of the agent, application-level authorization, network policy, and human approval for out-of-band actions may all need to sit outside the data platform while still belonging to the complete system around the model.
The accurate claim is narrower than “the data platform is the entire control system for AI,” and it is more credible for being narrower. For data-intensive agents, the platform supplies the context, data access, execution, and evidence core of the system. A separate authorization fabric or agent gateway has to connect that core to any tool that is not a data interface.
Microsoft’s own reference architecture for agent authorization follows exactly this pattern: a policy enforcement point intercepts a proposed action, a policy decision point evaluates identity and context, and the tool executes only after an allow decision or a completed approval.
This boundary also explains why a catalog, on its own, is not a runtime. A catalog informs a person who chooses to consult it. A runtime intervenes in execution whether or not anyone is watching. A quality alert that lands in a messaging channel after an agent has already acted is observability, useful for the next incident, but it is not a preventive control for this one. A well-written access policy is guidance until the moment it is checked automatically, before the action it governs, rather than after.
How to Keep Improving AI Agent Governance Over Time
The discipline of building a system around a fixed model treats failure as an input rather than a hiccup.
If an agent acts on a stale table, the fix is not a reminder to check freshness manually next time. The tool the agent calls should reject stale inputs as a matter of contract. If an agent keeps invoking an expensive model for a task a cheaper one handles adequately, that routing decision should become policy rather than a one-off correction.
If a legitimate workflow keeps colliding with a permission boundary that was never designed with agents in mind, the answer is usually a narrower, purpose-built tool, not a broader credential handed out to make the friction disappear.
Each incident should resolve into one of four durable changes: a clearer semantic definition, a narrower or more useful tool, a new deterministic check, or a revised boundary between what the agent may do autonomously and what still requires a human decision. This is the ratchet. Failures accumulate into permanent improvements to the operating environment rather than into a growing pile of postmortems nobody revisits. Modern Data 101 has made a related argument about durable definitions: a semantic layer only holds its value if the organization treats every correction as a permanent edit to the shared definition, not a one-time fix.
The goal is not to collect rules indefinitely. Every control encodes an assumption about what a model, a tool, or an organization cannot yet be trusted to do unsupervised. As that assumption changes, obsolete controls should be retired even as new ones are added for whatever failure mode appears next.
Models will keep changing in quality, price, and latency, often on a timescale measured in months. Whatever advantage a given model provides today is rented, and will be matched or exceeded by a competitor eventually. The system that surrounds the model, its semantics, its policies, its accumulated operating knowledge, its governed interfaces, and its record of what has already gone wrong once, is not rented. It stays.
MD101 Support ☎️
If you have any queries about the piece, feel free to connect with the author(s). Or connect with the MD101 team directly at community@moderndata101.com 🧡
Author Connect 💬
Got questions? Find Darpan on LinkedIn or drop a comment below. 💬
More from Darpan
On a similar concept from the same author: Why Your Data Platform Isn’t Wired for AI
References
Vivek Trivedy, Improving Deep Agents with Harness Engineering, LangChain
Vivek Trivedy, The Anatomy of an Agent Harness, LangChain
Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn, Meta-Harness: End-to-End Optimization of Model Harnesses, Stanford / MIT / KRAFTON, arXiv 2026
O’Reilly Radar, Agent Harness Engineering, Addy Osmani
DataOS, Vulcan: Semantic Modeling
HumanLayer, Skill Issue: Harness Engineering for Coding Agents
Anthropic Engineering, Harness design for long-running agents
Microsoft Security Community Hub, Authorization and Governance for AI Agents: Runtime Authorization Beyond Identity at Scale
NIST, SP 800-162: Guide to Attribute-Based Access Control (ABAC) Definition and Considerations
Modern Data 101, The Contract-driven Data Platform
Modern Data 101, Data Products: The Essential Context for Enterprise AI
Modern Data 101, The Semantic Layer Blackhole: From Initial Build to Perpetual Governance
Modern Data 101, Memory Blueprint for AI
Michael Rumiantsau et al., Semantic Layers for Reliable LLM-Powered Data Analytics: A Paired Benchmark of Accuracy and Hallucination Across Three Frontier Models, arXiv 2026
The refund example is the part I kept coming back to.
A correlation ID tells you what happened after the fact. It doesn’t stop an approved action from changing between review and execution.
I’d tie the approval to the exact customer list, amounts, and policy version. Change any of those and the approval dies.
Does your design bind approval to that exact payload, or only log everything under the same ID?