Vlad Radziuk | Account CTO

Vlad Radziuk is a transformation leader and systems thinker working at Nordcloud, an IBM company. He operates at the intersection of modernisation, competence centres, and AI-driven enterprise foundations with a strong focus on ontologies, knowledge graphs, and structured modelling.

With an interdisciplinary background spanning software engineering, cloud-native architecture, and enterprise data strategy, Vlad has built and led innovation units and competence centres of up to 50 professionals. As Account CTO and former Practice Lead, he combines delivery leadership with long-term architectural vision, shaping AI-powered portfolios, establishing internal knowledge ecosystems, and guiding complex transformation programs.

Beyond corporate leadership, Vlad is an active contributor to the data and knowledge graph community. He writes and researches topics such as the business case for ontologies, cross-domain metadata management, digital sovereignty, and the economics of AI integration. His thinking reflects a systemic lens: technology only creates value when aligned with governance, culture, and long-term strategy. We’re thrilled to feature his unique insights on Modern Data 101!

We actively collaborate with data experts to bring the best resources to a 15,000+ strong community of data leaders and practitioners. If you have something to share, reach out!
🫴🏻 Share your ideas and work: community@moderndata101.com

*Note: Opinions expressed in contributions are not our own and are only curated by us for broader access and discussion. All submissions are vetted for quality & relevance. We keep it information-first and do not support any promotions, paid or otherwise.

Let’s Dive In

What might sound like a bunch of buzzwords actually turned out to be a valuable lesson in systems thinking. I heard this statement during a discussion with a group of enterprise architects about how data mesh could support decentralizing their enterprise.

The challenges they described were familiar: bottlenecks, siloed data, and the all-too-common lament, “We’re sitting on mountains of data but extracting little value from it.”

If you’re a data specialist, these issues likely sound familiar - they’re part of our daily reality.

Then, the conversation took an unexpected turn.
One architect emphasized the need for a holistic approach when transforming organizations toward a platform-based setup with federated governance and decentralized ownership. That’s when he dropped that line.

It was a powerful reminder for me: real-world systems are complex and interdependent. Inspired by this insight, I decided to outline a high-level playbook for developing systematic approaches to decentralized data management methodologies like data mesh.

This playbook will probably not be interesting for everyone, but if you’re looking to understand the broader context and make better decisions (or offer better advice) for your internal or external stakeholders, it’s worth stepping back and examining the bigger picture beyond just the data layer.

Reasoning behind the decentralization of organizations

Decentralization, as the core idea of the data mesh methodology, isn’t an end in itself, but a means to drive better data quality and unlock greater value from it. But the question that occupies me is:

How can organizations maximize benefits from decentralization?

Theory suggests that decentralized organizations move faster, motivate employees more effectively, and foster innovation. My own experience confirms this. When an organization’s culture is focused on delivering customer value, having multiple centers of power can lead to remarkable outcomes.

For example, when I led a competence center for application development (operating outside the company’s traditional hierarchy), I was able to create new narratives, uncover new opportunities, and engage employees in ways that a top-down approach simply wouldn’t allow.

Another example that fascinated me recently
Local leaders in a multinational consulting firm, operating in a smaller market, developed a strategy to sustain growth even during years with a slower macroeconomy. They began serving local startups with attractive pricing, which gave them indirect access to venture capital networks and diversified their pipeline.

Many of us have seen similar success stories, but let’s be honest: often, these are driven by the enthusiasm of individuals rather than institutionalized decentralization. What interests me is how organizations can systematically enable decentralization at scale. What platforms, cultures, and organizational designs are needed to make this work?

Decentralization isn’t just about restructuring. It’s about how an organization operates. If we want to decentralize an enterprise, there must be a clear intention behind it. The shift should be leveraged holistically, not just to improve data value, but to transform the way we work.

In terms of platforms, Netflix can serve as a poster child with their data mesh architecture, knowledge graph, and Unified Data Architecture, which support various domains serving different parts of its platform, including films, series, games, live events, and ads. They regularly post articles about their architecture in their engineering blog (Netflix TechBlog), which I highly recommend reading.

Culture-wise, there is one company that made decentralization its source of competitive advantage and became a role model: a Chinese player in the IoT area called Haier.

They developed a philosophy called RenDanHeYi, after which they gave autonomy to their business units (e.g., a unit producing refrigerators, or other kinds of home appliances), which started to serve as “microenterprises”.

Each of these internal companies got “three rights”:

1. the right to make independent decisions,

2. the right to hire talent and

3. the right to distribute profit

So they can quickly make decisions based on market changes, take responsibility for their own gains and losses, and share value in accordance with the agreement.

Instead of a centralized strategy, Haier’s approach is customer-driven: each team identifies real needs and builds solutions, and coordinates through ecosystems (e.g., “microenterprises” can form alliances to achieve bigger goals, and for higher-level strategy)

Haier still has some centralized institutions that fund high-risk, long-term bets. Although I could not find much about the internal technological setup of Haier, I still brought up this example to illustrate how important it is to think systematically when planning decentralization, and not to just focus on technology in general or data in particular.

Still, this piece is primarily for data professionals, so let’s go back to what we know best and deduce other aspects from our shared vision of the bright data future.

What does it actually mean for data to be FAIR?

FAIR data (data that adheres to the principles of findability, accessibility, interoperability, and reusability) is a cornerstone of modern data management. For organizations aiming to implement FAIR, a data product culture (often supported by methodologies like data mesh) and explicit concept modeling (using ontologies) are essential.

This shift moves the focus from tables to business concepts: entities with clear definitions, data instances, and relationships to other concepts.

The FAIR principles provide a clear North Star for how data should be managed to deliver business value. Let’s consider logistics as an example. FAIR data would align with master data (locations, products, partners) and reference data (units of measure, currency codes).

Each data product or data point would have a persistent identifier (PID) or URI, enabling actions like scanning a pallet’s PID to instantly retrieve its contents, origin, and destination from a shared database.

Data would be fully described using knowledge languages or ontologies. For instance, a controlled vocabulary would standardize status terms like “Received,” “Picked,” or “Shipped” (aligned with the SCOR model), while relationships like “Product → StoredIn → WarehouseBin” would be explicitly defined.

I believe FAIR principles offer a powerful lens for examining the holistic decentralization of organizations.

🔖 Related Read(s)

Let’s imagine the shiny world of FAIR data: how would it look from the user's perspective? Users can find the data they need by typing, for example, concept names, synonyms, or related concepts into the search bar of a data catalog.

What’s even more compelling is that data is aggregated by business concepts, allowing users to navigate from concept to concept, much like on Wikipedia, where you might start with “black tea” and end up reading about Einstein’s biography.

In this case, however, users could search for a product, explore its sold instances, track shipping status, and receive recommendations for similar products. This seamless experience is made possible by combining complete domain ownership of data with federated governance.

But what does this mean when we step outside the data perspective?

1. Users traverse the enterprise network across domains. They don’t just access the knowledge space of different domains (e.g., by navigating a knowledge graph through a graphical interface, clicking on PIDs to move between concepts), but also can also consume data or access software functionality, provided they have the necessary permissions.

2. Domains own their assets, meaning they control business logic, access policies, and boundaries. In essence, they decide who can “visit” and interact with their data, much like setting the rules for guests in their own home.

3. Because domains fully own their business logic, the definitions of business concepts and processes can be traced: from knowledge bases and controlled vocabularies, through data, all the way to software.

4. Since domains provide their assets as products, they are responsible for the full lifecycle: maintenance, updates, monitoring, and beyond.

As you can see, decentralization, driven by data mesh and ontology-based data management, has far-reaching implications for how IT operates. The FAIR criteria don’t just guide data management. They also highlight what’s needed for domains to thrive in a decentralized environment.

Domain as a quantum of decentralized organizations

Decentralization often means empowering domains, but what exactly is a domain? It could be a department, business unit, team, platform team, technical capability, or something else entirely. At its core, a domain is any logical boundary with clear ownership, responsibilities, and alignment to business goals.

In the context of data, a domain refers to an organizational unit that not only produces data but also owns it throughout its entire lifecycle, from creation to consumption. However, ownership doesn’t begin with the data itself. It starts earlier, with business logic and language, a principle well-captured by the domain-driven design (DDD) methodology.

Domains have boundaries, which define their unique knowledge models, language, and rules (something we often call business logic). These boundaries extend beyond abstract concepts and include tangible assets that support domain operations, such as software and data. As a result, domains exist across multiple layers:

• Knowledge layer: The conceptual world of definitions, rules, and language.

• Operational layer: The runtime environment of applications, services, and microservices.

• Analytical layer: The world of data products and insights.

To successfully decentralize an organization, all these layers must be addressed, with each playing a critical role in enabling autonomy, clarity, and alignment.

If we abstract away from data or software and focus instead on business capabilities, something interesting emerges: those capabilities remain consistent, whether viewed through an operational or analytical lens.

The language, definitions, and rules (in other words, the knowledge model) persist across all layers. This alignment means that software and data can be managed in surprisingly similar ways. And, fortunately, we already have the tools to do it.

API-first approach and communication over the web

First of all, in an ideal FAIR world, the way we write software is different from how enterprises did that for decades, as we push business logic towards knowledge models (ontologies) and data, away from software. As Martynas Jusevičius told recently:

“Ora Lassila has claimed that all non-RDF data is legacy data. I would go one step further and claim that all software not built natively on and for RDF is legacy software.” (source).

I find this point interesting, but I personally have yet to check what it actually means and to what contexts it applies. On the contrary, domain-driven design (DDD) is pretty widely adopted nowadays and delivers value by many means, with one of them being modeling the code after business concepts, not technical vocabulary.

Another interesting relationship between software and data is that FAIR ultimately requires data products following similar principles that software engineers seem to have widely adopted: modular development (microservices) and communication over HTTP APIs.

And then it also becomes clear why the architect mentioned at the beginning of the article said, “Data mesh without service mesh does not make sense”. Yes, this phrase was cut from the context. The guy worked at a digital-native company before joining a traditional enterprise, so for him, it is easy to judge.

Not all data products are built like web services, for a number of reasons, and a service mesh would not be ultimately necessary to start implementing a data mesh. But still, the connection between the two concepts does exist and should not be ignored.

Data mesh tools focus on data-specific needs (metadata, lineage, access policies), while service mesh focuses on network-level concerns (encryption, load balancing, observability). One lives in the runtime world of microservices, the other in the organizational world of data domains.

Data mesh could rely on service mesh to provide the underlying networking and security fabric that connects domain-owned data products. Without service mesh, data mesh domains could struggle to enforce access controls, manage traffic, and ensure secure communication. On the contrary, service mesh ensures APIs feeding AI are reliable, secure, and observable.

With data being governed using the data mesh principles, with specialized tools and platforms supporting this, and with service mesh enabling services to talk to each other in a secure, observable way, it is easier to draw the technical boundaries between domains.

With that said, it is also important to mention that technology alone is not sufficient, as teams have to adopt the product mindset when it comes to both data and software. Handling them as a product means not only developing them, but also managing them along the whole lifecycle, including operations, monitoring, and so on.

Communication between the domains

It appears natural that different domains would like to communicate with each other: stages in a supply chain depend on each other, production facilities sometimes need to know the results of the R&D activities, and recommendation engine domain needs to pull historical data for past transactions of a customer.

While different parts of organizations would like to communicate on the architecture level, we would like to prevent tight coupling between domains: scenarios when components are highly dependent on each other, making them less flexible and harder to maintain.

The problem of coupling has many aspects: technological (e.g., programming languages), data format (binary, Parquet, CSV), interaction style (async vs sync, REST vs. GraphQL), and many others. Event-driven architectures seem to be a good way to minimize the coupling.

What is actually an event?

As a quantum of event-driven architectures, it is a signal that the state of the system changed. One necessary property of it is that it is immutable. As a consequence of these two properties, in a decentralized architecture, an event can serve as a contract for integrating different systems.

Then producing and consuming domains can agree upfront on a schema of the events. What attributes would a good schema contain? At the very least, an event needs to contain notification about change.

But the more self-contained an event is, the more successful an integration can be. Thus, we would like to see not only the fact that something changed, but also the nature of the changes (what data was modified) and the context.

In the best case, a domain event should be stateless, immutable, modeled as close to the domain as possible, belong to one of the types from the controlled vocabulary/taxonomy, be scoped to a bounded context, and describe the intent behind the change.

Such an event can ensure interoperability and thus serve as a good “currency” in a decentralized architecture. Furthermore, an event with ID can help with deduplication.

• Producing domains are not only responsible for adding context to their events, but also for versioning and ensuring backward compatibility of it (for some time),

• while consuming domains must take ownership of consumer-related logic.

It is considered a good practice when producers know their consumers and do not give out secrets of their life on a wall at the central square - e.g., they can have a policy that only users with specified roles or keys can read the information in the event payload.

As communication between domains happens over networks, I think it makes sense to touch on some networking aspects, too. If domains are responsible for managing their own software, data assets, and their domain events, it follows naturally that they should also take on more responsibility for managing their network and communication streams.

While it’s unrealistic for domains to manage the entire networking infrastructure, they can realistically own their access policies and monitor their networks using modern observability tools.

Service mesh empowers domains to configure their own ingress/egress rules, retries, timeouts, and load balancing. With observability tools, domains can independently track latency, errors, and traffic - without relying on a central team.

Managing policies (e.g., access or firewall rules) as code enables versioning, testing, and deployment via CI/CD pipelines, ideally in a tool-agnostic way.

Boundaries between domains are enforced at two levels.

Thanks to the network isolation (isolated virtual networks and micro-segmentation) on one side and event-driven architecture on the other, we can decouple workloads and services, enabling independent scaling.

Instead of service A polling service B, service B emits events when changes occur, embodying a “choreography” mindset (vs. “orchestration”) and enabling service-level decentralization. Back on the network layer, tools like PrivateLink and enforced encryption for all inter-domain communication ensure secure connectivity between domains.

The difficult part: human behaviour and business architecture

When decentralizing organizations, you need a systematic approach that will touch different aspects of domain operations. In this text, I touched on the main principles, such as:

• decentralized ownership over software, data, and network assets;

• self-service platform operated by centralized platform teams;

• cross-functional collaboration happening on multiple levels, especially in governance;

• computational governance, policies as code, infrastructure as code, versioned resources and workflows;

• API-first design;

• event-driven architecture.

But as compelling as your approach may sound on paper, reality is far more challenging. You’ll face legacy cultures rooted in the industrial era, organizational inertia, and deep-seated biases.

You’ll need to strike the right balance between centralization and decentralization - one that aligns with your organization’s unique context. And you’ll have to prepare people for new roles, like data stewards, which require both technical and cultural shifts.

To be honest, this is more art than science.

Start with a business problem.

I’ve seen many teams eager to improve how their IT organizations operate, confident that they have management buy-in. But too often, their plans were disconnected from real business problems.

Instead of addressing tangible pain points, they focused on crafting the perfect architecture (on paper). That’s not how transformative changes like decentralization work.

You have to start with a specific, agreed-upon business problem - something concrete that management cares about. For example, instead of deploying an entire data mesh infrastructure upfront, you might begin with a data catalog to help the R&D team gain visibility into the company’s data assets.

This approach achieves two critical things: real buy-in (when your architecture solves a specific, urgent problem, stakeholders see its value immediately) and purpose (real problems provide a clear sense of direction: something abstract architecture diagrams often lack). Start small, solve a real problem, and build momentum from there. That’s how meaningful change happens.

Search for early adopters, build alliances.

In any organization, even the quietest ones, there are always people who are eager to learn and try out new things.

I covered the topic of engaging employees in another article of mine: Every employee is a superhero - when they understand the economics of their jobs | LinkedIn Also don’t be afraid of building alliances across the company to ensure the success of your initiative. Again, the goal is to find people who, for some reasons, are interested in your version of the future.

Decentralization is primarily a business endeavour, not technical one.

I described the case of Haier, which reshaped its whole operational model in a decentralized spirit.

Authority allows the domain to make decisions closer to the outside world, to customers. But they also have to have the right capabilities to support their judgment. One cannot work without the other. And what is most important is that only a culture built on trust, on a positive perception of humans, can allow for a true decentralization, resulting in better decisions.

When I was writing this piece, published a text where he distinguishes between delivery-centric vs. value-centric organizations. While the first ones aim at predictable outcomes and controlled workflows, the latter, which remind me of classical Japanese management practices: aim to grow value over time by continuously improving themselves.

But this learning can only happen close to where the problems emerge. Which, in its turn, requires having true ownership over one's own workflows and assets. Thus, by focusing on the technical side, we risk again hoping for a magic tool (or self-service platform) to solve our problems. But as it usually is, we need to do some homework on the organizational level first.

MD101 Connect ☎️

If you have any queries about the piece, feel free to connect with the author(s). Or connect with the MD101 team directly at community@moderndata101.com 🧡

Author Connect 💬

Connect with Vlad on LinkedIn 💬

From MD101 team 🧡

The Data Product Playbook

Here’s your own copy of the Actionable Data Product Playbook. With 3500+ downloads so far and quality feedback, we are thrilled with the response to this 6-week guide we’ve built with industry experts and practitioners. Stay tuned to moderndata101.com for more actionable resources from us!

DOWNLOAD!

Alejandro Aboy | Senior Data & AI Engineer, Writer

Alejandro is a Senior Data & AI Engineer and the voice behind The Pipe & The Line, a technical newsletter where pipelines meet product thinking. With a career that began in marketing and web analytics, he transitioned into modern data engineering, bringing with him a sharp instinct for business context, experimentation, and stakeholder alignment.

At Workpath, Alejandro serves as the main owner of the Data Platform & AI stack. He builds AI agents with RAG architectures (Pgvector on Aurora), designs MCP-based tool ecosystems, and implements production-grade AI observability. His work spans dbt modeling, orchestration with Airflow, Slack-to-Tableau automation, and secure backend design for agentic systems.

Alejandro is deeply engaged in the open data community. He has collaborated with OpenMetadata on agentic data modeling and metadata-first architectures, co-authoring end-to-end implementations that connect lineage, governance, and AI tooling through MCP. His writing is frequently featured by Modern Data 101 and other leading data publications. We’re thrilled to feature his unique insights on Modern Data 101!

We actively collaborate with data experts to bring the best resources to a 20,000+ strong community of data leaders and practitioners. If you have something to share, reach out!
🫴🏻 Share your ideas and work: community@moderndata101.com

*Note: Opinions expressed in contributions are not our own and are only curated by us for broader access and discussion. All submissions are vetted for quality & relevance. We keep it information-first and do not support any promotions, paid or otherwise.

Let’s Dive In

TL;DR

• An AI agent in production meets every formal definition of a data product. It has users, serves analytical data, needs quality monitoring, evolves through versioned schemas, and breaks silently when you ignore it.

• Observability is not logging. It’s product analytics. Every conversation is a user session. Every tool call is a feature interaction. Every retry is a frustration you can measure.

• Undetected errors are invisible churn. Your agent can score perfectly on generic metrics while hallucinating links, suggesting impossible actions, and losing users you’ll never hear from.

• The feedback loop IS the roadmap. Manual annotation sessions, clustered by patterns, connected to tickets. Not dashboards. Decisions.

• Security is a product contract, not a checklist. 4 defense layers, human-in-the-loop, and scoped permissions. A data product nobody trusts is a data product nobody uses.

• Your agent can be its own onboarding tool. Enable it to explain its own features through conversation. Documentation becomes discoverability. Every “how do I...” question is a product signal.

• The teams winning aren’t the ones with the best prompts. They’re the ones treating agent behavior as a continuous product input.

The Subtle Shift that Went Unnoticed

You shipped your agent. Observability is green. Traces are flowing. Users are chatting. But can you answer these?

1. What are users actually trying to do with it?

2. Which tools should you optimize or kill?

3. What feature requests are hiding in conversations you never read?

Not being able to answer any of this means you have an experiment running in production. Most teams assigned agents to ML engineers or product managers. In my opinion, they are partially right and partially wrong.

The infrastructure underneath: RAG pipelines, semantic layers, observability traces, that’s data engineering territory. And the vision on top screams product thinking.

Your Agent Already Is a Data Product

As we can read in this post, we usually describe data products as assets that stakeholders should be able to discover, access, trust, and interact with in a secure fashion.

You can’t carry that vision without observability, clean architectures, and proactive quality monitoring. Let’s see how your AI Agent can fit conceptually in this definition:

AI Observability Is Product Analytics

Most teams connect to Opik or Langfuse and call it done. But that’s just logging. Product analytics is a different ballgame. It answers: What should I build next? Where should I put my focus?

Every conversation is a user session.

When I started reading conversations manually at Workpath, I found users hacking the agent for use cases we never designed. 50% wanted bulk operations we didn’t support, and others were asking for exports that didn’t exist.

None of this showed up in any dashboard. The tools were not raising errors because everything was failing successfully.

Every tool call is a feature interaction.

We had 50 tools. Users called 5 regularly. Some tools were ghosts because the agent decided to never use them. Others had high usage but huge failure rates.

That’s a product prioritization alert: fix the broken popular ones, kill the unused ones, and merge the ones that always get called together.

Every retry is a measurable frustration.

• When a user rephrases the same question three times, that’s friction instead of a “conversation.”

• When they abandon after “I can’t help with that,” that’s churn.

• When they ask for something five times and give up, that’s a feature request screaming right at your face.

Undetected Errors Are Invisible Churn

My agent had perfect hallucination scores. Then I started reading conversations manually and found:

• Agent hallucinating documentation links after fake-searching the docs

• Agent inferring wrong user information and making incorrect tool calls

• Agent suggesting follow-up actions that literally didn’t exist

Generic metrics (e.g., Hallucination, AnswerRelevance, ContextPrecision) are too abstract to catch this. They tell you something is “good” or “bad” without telling you what to fix.

What works

Binary evaluation criteria specific to your use case. “Did the agent call search_knowledge before citing a URL? Does the URL in the response match the tool output?”

Run it with LLM-as-Judge ten times, and you get the same answer. And if you want a bulletproof validation, ask different colleagues in your company. That’s testable.

Those undetected bugs are signals about what your agent promises and can’t deliver. Each one is either a ticket to fix something or a scope clarification for what the agent should and shouldn’t do.

A data product that fails silently is worse than one that fails loudly. At least loud failures get tickets.

Security as a Product Contract

Security isn’t a checklist you bolt on before launch. For a data product, trustworthiness is a foundational property. If users don’t trust the product, they stop using it.

Your agent accepts natural language from anyone. That’s the largest attack surface in your product. I wrote an entire series on building jailbreak-proof AI agents, and the core idea is defense in depth across 4 layers:

• Input validation is a product boundary definition. You’re deciding what your agent will and won’t accept.

• Guardrails are a scope definition. Topical guardrails keep the agent on-task. PII masking isn’t compliance is built-in trust. Users need to know their data isn’t leaking into responses.

• Output control prevents the agent from fabricating data that looks real. Structured outputs are the difference between an agent that invents a documentation link and one that only cites what it actually found.

• Architecture is where engineering meets product. Least privilege, scoped tools, observability. An agent can’t drop your production table if it doesn’t have permissions.

Then there’s human-in-the-loop. I’ve been evaluating Safety Operations to see if my agents comply with asking users before destructive operations. The agent asking, “Are you sure you want to delete this?” is good UX. It also happens to be good security.

Teaching Your Agent What It Can Do

Discoverability is the first data product property we listed. Traditional data products have catalogs. AI agents have something better: they can explain themselves through conversation.

Users don’t read documentation. Savvy ones ask the agent, “What can you do?” and if the answer is vague, you lose them before they even try the feature they came for.

At Workpath, we enabled Zendesk knowledge base articles as a RAG source. When users ask “how do I create a goal?” or “what can you help me with?”, the agent searches those articles and explains features, workflows, and guidance in conversation. The agent seeks to teach users on demand.

Instead of static docs nobody reads, you have a conversational interface that meets users where they are and explains exactly what they need.

Track those patterns and find discoverability signals you can leverage. If the agent can’t explain a feature clearly, either the knowledge base article is bad, or the feature itself is poorly designed.

Every “how do I...” question is a product signal about what’s confusing, what’s undocumented, and what users actually want to use.

The Feedback Loop Becomes the Roadmap

The process I developed after 1000+ manual (yes, 1 by 1) conversation reviews:

• Before each annotation session, prepare. Look for anomalies to pick up outliers, such as latency, too many tool calls, long-lasting conversations, and high token consumptions. Don’t read “some chats” and call it a day.

• During annotation, collect structured signals. Is this a bug? A missing feature? A scope violation? A prompt gap? Each annotation should map to an action type, not just a thumbs-up or thumbs-down.

• After annotation, materialize the roadmap. Connect patterns to JIRA tickets. Update documentation. Cross-reference against the backlog to validate if the most concerning patterns are already being considered.

I started generating a bunch of GitHub PRs just by reading conversations, taking notes, and applying those annotations as Claude Code plans.

None of these came from user interviews. None came from JIRA. They came from reading what users actually did with the product.

The loop: Conversations → Structured Annotation → Clustered Patterns → Tickets → Ship → Measure → Repeat

Final Words

A robust data product encodes business meaning, enforces quality, traces change, and supports continuous evolution. It earns trust through security. And it teaches users what it can do instead of waiting for them to figure it out.

I haven’t seen a moment like this in my career, where users are telling you what to build next through every single conversation.

The teams winning at AI agents are the ones with the best feedback loops, the strongest trust contracts, and agents that know how to explain themselves.

MD101 Support ☎️

If you have any queries about the piece, feel free to connect with the author(s). Or connect with the MD101 team directly at community@moderndata101.com 🧡

Author Connect

Connect with Alejandro Aboy on LinkedIn 💬 | Or find more of his work on Substack:

The Pipe & The Line

Hands-on guides, tools, and experiments to sharpen your Data & AI Engineering skills from someone who learned it all in the wild.

By Alejandro Aboy

Antonio Sánchez Gómez | Data Management & Governance Specialist

Antonio is a Data Management & Governance Specialist at Roche, where he focuses on building scalable Master Data Management (MDM) and Data Governance operating models. His work centres on establishing clear data ownership, decision rights, and reusable governance structures that transform master data from technical assets into trusted business capabilities.

Antonio designs and implements governance frameworks that create single sources of truth across critical domains such as customer, product, and pricing. He advocates for governance as an operational discipline, embedding stewardship workflows, quality thresholds, and accountability directly into enterprise execution. His perspective positions Data Governance as a revenue enabler that accelerates product launches, strengthens regulatory confidence, and removes friction from core business processes.

With a background spanning data engineering, analytics, and big data across organisations such as the International Olympic Committee and financial services programs at BBVA via Minsait and Deloitte, Antonio combines deep technical expertise with enterprise governance design. We’re thrilled to feature his unique insights on Modern Data 101!

We actively collaborate with data experts to bring the best resources to a 15,000+ strong community of data leaders and practitioners. If you have something to share, reach out!
🫴🏻 Share your ideas and work: community@moderndata101.com

*Note: Opinions expressed in contributions are not our own and are only curated by us for broader access and discussion. All submissions are vetted for quality & relevance. We keep it information-first and do not support any promotions, paid or otherwise.

Let’s Dive In

Data Ownership is one of the most frequently defined roles in Data Governance, and one of the least effectively exercised.

Despite mature frameworks, operating models, and role catalogs, many organizations still struggle to translate “ownership” into real authority, accountability, and business consequence. In most enterprises, Data Owners exist formally but operate with limited decision power.

Such as accepting data-related risk, approving quality thresholds, authorizing usage, or resolving conflicts between value and control. Ownership becomes visible on paper but is absent where governance must operate.

This is not primarily a problem of role definition, but a problem of decision rights.

Governance fails because authority is not explicitly designed into the operating model. When decision rights are unclear, decisions drift toward operational teams, escalation becomes informal or avoided, and accountability dissolves across organizational boundaries. The result is governance that appears complete but is structurally incapable of governing.

This article reframes Data Ownership as a decision-centric role. Ownership is not a collection of activities, but a formally empowered authority embedded in the enterprise decision system. Only when decision rights are explicit, bounded, and institutionally supported does Data Governance function as an operational capability rather than an aspirational framework.

🔖 Related Read

Why Data Ownership Remains Ineffective

The persistent failure of Data Ownership is not due to a lack of documentation, but due to a systemic misunderstanding of what ownership entails in complex organizations.

In many cases, Data Owners are appointed based on hierarchy or domain proximity, without a corresponding transfer of decision authority. Governance programs emphasize responsibilities (reviews, forums, endorsements) while remaining silent on what Data Owners are actually entitled to decide.

Matrixed organizational structures amplify this issue.
Decisions around data quality, access, or structural change cut across domains and systems. When decision rights are not explicit, decisions are either pushed downward, endlessly deferred through consensus, or resolved informally by whoever has operational leverage.

The Data Owner is bypassed not out of negligence, but because their authority was never operationalized.

Until Data Ownership is designed around decision rights rather than activities, it will remain symbolic rather than functional.

From Role Definitions to Decision Authority

Governance operates through decisions, not tasks. Outcomes are shaped by who can approve thresholds, accept risk, authorize usage, or prioritize remediation, not by who attends meetings or produces reports.

Many organizations assign Data Owners extensive accountability without granting authority to enforce standards, block usage, or resolve conflicts. This structural asymmetry (responsibility without decision rights) erodes both the credibility of the role and the effectiveness of governance.

A decision-rights perspective forces a different question:

What decisions is the Data Owner empowered to make, under which conditions, and with what consequences?

Decision rights must be explicit, bounded, and supported by formal escalation paths. Authority that exists only through personal influence collapses under pressure. Moving from role descriptions to decision design is not a semantic refinement; it is a structural redesign from governance as documentation to governance as an executable system.

What Ownership Means in an Enterprise Context

Enterprise ownership combines authority, accountability, and risk acceptance.

Data Ownership is inseparable from risk. Decisions on quality thresholds, usage permissions, access models, or structural change all imply acceptance of residual financial, operational, regulatory, or reputational risk.

Data Owners do not own only data artifacts, but also the business outcomes enabled by data. This is why ownership cannot sit exclusively with technical or operational roles. Authority must rest with those accountable for business performance.

Ownership must also be institutionally recognized. Informal authority collapses under competing priorities. Decision rights must be documented, escalation paths defined, and accountability mechanisms enforced. Without this, ownership defaults into coordination rather than governance.

Core Decision Domains of the Data Owner

In practice, Data Owner decision rights consistently fall into a limited set of domains:

• Data Quality: Defining acceptable quality thresholds, approving exceptions, and accepting residual risk, not monitoring metrics or fixing defects.

• Data Usage: Authorizing or restricting use based on value, sensitivity, contractual obligations, or regulatory exposure.

• Data Access: approving access models and exceptions. Provisioning is operational while authorization is a business decision.

• Structural & Lifecycle Change: approving changes to models, hierarchies, or reference data, balancing short-term delivery against long-term consistency.

• Risk Acceptance: making explicit decisions about tolerated exposure and escalation.

These domains define the practical scope of Data Ownership. Without explicit authority in these areas, ownership cannot function operationally.

Data Owner vs Data Steward: The Decision Boundary

The most common governance failure is not role confusion, but misallocated decision authority.

Data Stewards are frequently asked to approve thresholds, accept deviations, or close issues: decisions that carry business risk they are not accountable for. Conversely, some organizations reduce Data Owners to passive signatories, disengaged from real decisions.

When governance models describe who does what but not who decides what, decisions inevitably migrate toward execution. This is a structural inevitability.

Effective governance requires a clear separation: Data Owners decide and Data Stewards execute, monitor, and escalate.

🔖 Related Read

The Cost of Undefined Decision Rights

Undefined decision rights produce systemic consequences:

• Organizationally: Accountability erodes, escalation becomes performative, and ownership is perceived as optional.

• Operationally: Decisions stall, exceptions are inconsistent, and teams default to risk aversion or workarounds.

• From a business perspective, organizations oscillate between over-control and under-control, slowing delivery or exposing themselves to unmanaged risk.

Most critically, risk ownership becomes invisible.

Designing Decision Rights into the Operating Model

Making decision rights explicit requires intentional design, not more documentation.

Effective models start from recurring decision points (quality thresholds, usage authorization, access exceptions, structural change) and explicitly assign authority, scope, and escalation criteria. Decision rights must be embedded into workflows, not layered on top.

Escalation thresholds must be predefined to prevent operational roles from absorbing risk by default. Authority must be institutionally reinforced through leadership mandate, alignment with risk management, and performance accountability.

Finally, decisions must be visible and traceable. Governance matures when risk acceptance is deliberate rather than accidental.

How Decision Rights Evolve with Maturity

Decision rights evolve as governance matures:

• Early stages: Corrective, visible, and frequent intervention to establish legitimacy.

• Stabilization: Focus shifts to approving standards and tolerances, enabling operational autonomy within boundaries.

• Advanced maturity: Decisions become strategic and anticipatory, enabling new capabilities and managing trade-offs between innovation and control.

Misalignment occurs when mature expectations are applied to immature structures—or when decision models remain overly prescriptive after governance stabilizes.

Signals of Effective Data Ownership

Functional Data Ownership is visible through behavior:

• Clear, timely decisions at escalation points

• Explicit, traceable risk acceptance

• Consistent decision patterns across similar cases

• Confident execution by operational roles

• Reduced decision latency and predictable outcomes

Where these signals are absent, ownership exists structurally but not operationally.

Final Note

Governance works when decisions have owners. Data Governance succeeds or fails on the clarity of decision ownership.

Frameworks provide structure, but they do not govern. Governance becomes real only when authority to decide is explicitly assigned, supported, and exercised.

Until organizations design Data Ownership around decision rights, governance will remain an intention rather than an operational capability. Governance works when decisions have owners.

MD101 Support ☎️

If you have any queries about the piece, feel free to connect with the author(s). Or connect with the MD101 team directly at community@moderndata101.com 🧡

Author Connect

Connect with Antonio Sánchez Gómez on LinkedIn 💬

From MD101 team 🧡

The Modern Data Masterclass: Learn from 5 Masters in Data Products, Agentic Ecosystems, and Data Adoption!

With our latest community milestone, we opened up The Modern Data Masterclass for all to tune in and find countless insights from top data experts in the field. We are extremely appreciative of the time and effort they’ve dedicatedly shared with us to make this happen for the data community.

Master Modern Data, One Session at a Time!

Explore classes

Jessica Talisman

Knowledge Infrastructure Strategist, Founder & Advocate of The Ontology Pipeline Framework

Jessica Talisman is a Semantic Engineer, Information Architect, and knowledge infrastructure strategist dedicated to building information systems. With more than 25 years of experience in enterprise architecture, e-commerce content systems, digital libraries, and knowledge management, Jessica specialises in transforming fragmented information into coherent, machine-readable knowledge systems.

She is the founder of the Ontology Pipeline, a structured framework for building semantic knowledge infrastructures from first principles. The Ontology Pipeline emphasises progressive context-building: moving from controlled vocabularies to taxonomies, thesauri, ontologies, and ultimately fully realised knowledge graphs.

Professionally, Jessica has led semantic architecture initiatives at organisations including Adobe, where she architected an RDF-based knowledge graph to support Adobe’s Digital Experience ecosystem, and Amazon, where she worked in information architecture and taxonomy. She also founded Contextually LLC, providing consulting and coaching services in ontology modelling, NLP integration, and knowledge governance.

An educator and thought leader, Jessica publishes regularly on her Substack newsletter, Intentional Arrangement, where her writing frequently explores the relationship between semantic systems and AI. We’re thrilled to feature her unique insights on Modern Data 101!

We actively collaborate with data experts to bring the best resources to a 15,000+ strong community of data leaders and practitioners. If you have something to share, reach out!
🫴🏻 Share your ideas and work: community@moderndata101.com

*Note: Opinions expressed in contributions are not our own and are only curated by us for broader access and discussion. All submissions are vetted for quality & relevance. We keep it information-first and do not support any promotions, paid or otherwise.

Let’s Dive In

The Opportunity Before Us

Ontologists and semantic engineers face a persistent professional challenge to demonstrate that our work produces tangible value. We build controlled vocabularies, develop metadata standards, structure taxonomies, and design ontologies. Yet too often, these artifacts remain siloed rather than operational infrastructure.

Business stakeholders often struggle to connect our deliverables to measurable outcomes due to a lack of understanding about the mechanics of semantic systems.

Entity resolution changes this equation. When semantic infrastructure directly enables entity reconciliation (determining who is who and who is related to whom across data sources), our work becomes measurably operational.

Match rates improve. False positives decrease. Disambiguation succeeds where string matching fails. The thesaurus we build from entity resolution results becomes part of the critical infrastructure, which makes the use case work.

The emergence of tools like Senzing’s sz-semantics library and specifically, the Senzing thesaurus creates an inflection point for our profession. These tools leverage the Ontology Pipeline: the systematic methodology I have articulated for constructing semantic knowledge management systems to shape logic through executable code.

For taxonomists, ontologists and semantic engineers, this means our frameworks now have concrete tooling, with a methodology and deployable artifacts. While elegantly simple in its shape and code base, the Senzing thesaurus demonstrates how essential our expertise becomes as neurosymbolic AI solutions based upon Semantic Web principles emerge in the public repositories of enterprises.

Why Entity Resolution Needs Us

Entity resolution engines excel at resolving multiple records from different data sources based on evidence, for which Senzing uses entity-centric learning. The system architecture supports the detection of names, addresses, phone numbers, identifiers, and other features that constitute entity profiles.

They handle messy, inconsistent data with remarkable sophistication, accommodating variations in spelling, formatting, and completeness that plague enterprise data integration. What they do not do or what they cannot do without semantic infrastructure is interpret what matches mean within a knowledge domain.

When entity resolution operates without our work, results remain technically useful but semantically isolated. The output captures what matches, but not why it matters. Two records merge because phone numbers align,

• but what does that merged entity represent in the domain?

• What relationships does it participate in?

• What constraints govern its behavior?

Semantic engineering and ontologies can model and architect domains to answer these questions, thereby adding rich context and meaning to the underlying data.

On the way to building out supporting, semantic infrastructure, the Ontology Pipeline becomes essential. The pipeline’s iterative stages (controlled vocabulary, metadata standards, taxonomy, thesaurus, ontology, knowledge graph) prepare data and information for semantic interpretation.

Each stage builds towards maturing semantic systems to support the meaningful classification of resolved entities. Discovered relationships can be mapped to edges, where the ambiguity inherent in real-world data resolves through principled knowledge organization.

Entity resolution produces candidate matches. But as ontologists and semantic engineers know all too well, the semantic infrastructure determines what those matches mean.

The Ontology Pipeline as Professional Framework

The Ontology Pipeline provides the technical methodology and professional framework for scoping, estimating, and delivering semantic engineering projects. When business stakeholders ask what ontology work costs or how long it takes, the pipeline offers concrete answers. Each stage has defined inputs, outputs, and success criteria. Each stage maps to identifiable skills and time investments.

To build resilient systems that fit into the expectations of organizations, the pipeline’s defined stages translate to tangible project deliverables. Controlled vocabulary construction produces deduplicated, defined term lists with clear scope and cardinality.

Metadata standards development delivers schema documentation with element definitions, value constraints, and validation rules, informing semantic engineering products of how data traverses complex data ecosystems.

Taxonomy construction yields hierarchical concept schemes in SKOS format with defined depth and breadth.

The thesaurus extends the taxonomy by adding associative relationships, synonyms, and cross-references that handle domain ambiguity. Ontology development introduces classes, properties, attributes, relations and logical constraints. Knowledge graph deployment represents an architecture that synthesizes all stages into a dynamic and queryable knowledge infrastructure.

Each deliverable can stand on its own, and be easily purposed for specific needs and use cases. Each can be reviewed, validated, and versioned. Each contributes measurably to the semantic system’s capabilities.

When entity resolution connects to this pipeline, stakeholders and the business can understand exactly how ontology work enables data integration outcomes.

The controlled vocabulary ensures consistent class, entity and attribute naming conventions and definitions. Metadata standards define schema constructs and logic while also revealing the mapping logic for entities, classes and attributes. The taxonomy classifies entities, lending inference by way of a hierarchy that shapes parent-child relations.

The thesaurus adds more contextual understanding to the taxonomy by encoding alternative labels, hidden labels, relationships and documentation properties. The ontology defines semantic interpretations while providing logical reasoning. The knowledge graph can include all parts of the Ontology Pipeline, making it all queryable.

This is how we justify investment and build buy-in. Trust is formed when we move beyond abstract arguments and demonstrate how each pipeline stage directly enables entity resolution quality and contextual understanding for machines and humans.

The Thesaurus Stage, The Sweet Spot

Of all the Ontology Pipeline stages, the thesaurus is the low-fidelity powerhouse that can provide the most value to entity resolution. A thesaurus handles ambiguity by forming associative relationships between terms, going beyond the parent-child relationships within a hierarchy.

It captures synonyms, near-synonyms, variant spellings, acronyms, cross-domain equivalences and linked data enrichment. This is what entity resolution needs but cannot generate on its own.

When Senzing determines that records from different sources describe the same person, it performs a matching function. When the semantic thesaurus assigns that resolved entity a URI, classifies it within a taxonomy, and links alternative names as skos:altLabel relationships, it performs a knowledge organization function. The thesaurus transforms raw matches into semantically meaningful entities that participate in broader information ecosystems.

The sz-semantics library implements this transformation directly. Its Thesaurus class takes entity resolution results and a domain taxonomy as inputs, then generates RDF fragments that extend the taxonomy with resolved instance data. Alternative names discovered through matching become thesaurus relationships. Entity types map to taxonomy concepts. The output serializes a SKOS-compliant RDF formatted corpus, and applies these values to entity resolution outcomes.

For semantic engineers, this means our thesaurus expertise directly determines entity resolution quality.

The more precise our taxonomy classification, the more meaningful the resolved entity profiles. The more complete our associative relationships, the more discoverable the connections between entities. The thesaurus can provide the necessary support to imbue data with rich context and meaning.

Human-in-the-Loop: Expertise Becomes Essential

Automated entity resolution and semantic transformation cannot replace human judgment. The Strwythura project, which implements the complete Ontology Pipeline workflow, explicitly includes a human-in-the-loop curation stage.

As noted in The Strwythura project’s GitHub readme, the human-in-the-loop (HITL) is an essential step, whereby

“humans curate definitions for entities extracted from the unstructured content...There is a ‘cascade’ effect, where the quality of entity definitions poses a limiting factor for how well the system can perform relation resolution downstream.”

Senzing formalizes HITL because it is instrumental to the success of semantically supported entity resolution, highlighting the critical importance of semantic quality and expert oversight.

What’s involved with curation exactly?
Entity resolution produces candidate matches with varying confidence levels. The semantic thesaurus assigns these entities to taxonomic categories based on available evidence.

Some assignments will be correct while others will require refinement.

• Which synonyms should be promoted to preferred labels?

• Which alternative names represent genuine aliases versus data entry errors?

• Which taxonomic classifications capture the entity’s primary identity versus incidental attributes?

These are knowledge organization questions that semantic engineers answer through thesaurus construction and enrichment. Our training in classification theory, facet analysis, and vocabulary management prepares us to make exactly these judgments.

Our understanding of domain context enables us to recognize when automated assignments miss nuance. Our familiarity with SKOS and ontology engineering allows us to encode corrections in standard, machine readable, interoperable formats.

HITL curation prioritizes data quality and semantic integrity according to thesaurus and ontology standards. Entity resolution produces the matching framework while the thesaurus supports the capture of rich semantics and context, to ultimately enable reliability and accuracy.

While automated transformation produces candidates and matches, the thesaurus produces validated knowledge. As ontologists and semantic engineers, the Senzing thesaurus was engineered by ontologists with design decisions and judgment that defines our expertise.

NIEM: A Case Study in Pipeline Application

The National Information Exchange Model (NIEM) presents a compelling case study for ontologists applying the pipeline to entity resolution. NIEM already institutionalizes the first two pipeline stages: it provides controlled vocabularies for government information exchange and metadata standards that define how those vocabularies apply to data elements. For semantic engineers working in the public sector, NIEM represents existing infrastructure ready to be extended, with the objective.

Entity resolution challenges are at the core of NIEM use cases. Law enforcement agencies share suspect information across jurisdictions. Emergency management coordinates resources across organizational boundaries.

Immigration systems reconcile traveler identities from multiple sources. Each scenario involves determining who is who when data quality varies, terminology differs, and completeness fluctuates.

Through the Senzing RDF thesaurus, we map SKOS concept schemes to NIEM data components, organizing elements, types and attributes through matching frameworks based upon vocabulary concept definitions and semantic relationships.

We build thesauri that capture the alternative terminologies different agencies use for the same concepts. We develop ontologies that constrain how NIEM-compliant data participates in entity resolution.

The NIEM-Senzing mapping exercise is semantic engineering work. Aligning NIEM element definitions with Senzing feature types (NAME_FULL to nc:PersonName, ADDR_LINE1 to nc:LocationStreet, organizational identifiers to NIEM organization components) constitutes a crosswalk, leveraging thesaurus concepts and properties to support resilient mapping and transformer logic.

This mapping becomes a metadata standard, enabling entity resolution results to flow directly into NIEM-compliant information exchanges.

For ontologists working with the NIEM ecosystem, entity resolution integration represents an opportunity to demonstrate that semantic engineering enables interoperability outcomes.

Federated entity resolution across agencies becomes possible because semantic infrastructure ensures consistent definitions and interpretation. The thesaurus captures variant agency terminologies while the ontology enforces exchange constraints; the knowledge graph provides queryable access. Our work makes it work.

The Value of Ontology Work

Semantic knowledge systems are rapidly emerging as essential for achieving highly performant AI applications and services. Large language models require clean, well-structured, semantically enriched data to provide accurate and reliable results.

Knowledge graphs underpin retrieval-augmented generation systems that ground LLM outputs in verifiable facts. Entity resolution ensures that the entities referenced in AI responses correspond to accurate, real-world subjects.

The Ontology Pipeline positions semantic engineers at the center of this infrastructure. By following the pipeline’s rigorous, iterative process (cleaning, preparation, reconciliation, modeling, testing, enrichment, enforcement), we ensure the data quality that AI systems require.

Entity resolution validates that the knowledge we organize corresponds to reality. The semantic thesaurus ensures that the vocabulary AI systems use is consistent and unambiguous.

Tools like sz-semantics and Strwythura demonstrate that our frameworks support complex ecosystems, with or without AI. The Thesaurus class generates SKOS from entity resolution results. The pipeline workflow produces entity-resolved knowledge graphs ready for GraphRAG integration. Interactive visualization makes our work accessible to stakeholders who need to understand what the semantic system contains.

This architecture demonstrates the value of semantics in supporting complex knowledge infrastructures, and moreover, a professional opportunity. Semantic engineers and ontologists are essential to making semantic architectures such as Senzing’s a reality.

Entity resolution needs semantic engineers because matches without meaning are just data. The Ontology Pipeline provides the methodology; these tools provide the implementation; our expertise provides the quality, context and meaning. Ontologists make the knowledge work.

Getting Started With Designing for Infrastructure

For semantic engineers ready to apply the Ontology Pipeline to entity resolution, the work involves the familiar in addition to new tools, methodologies and applications.

Audit your existing semantic assets

What controlled vocabularies, metadata standards, taxonomies, and ontologies already exist in your organization? These represent completed pipeline stages. Assess their quality, coverage, and the SKOS compliance of controlled vocabularies. Identify gaps that entity resolution integration would expose.

Learn the Senzing Vocabulary and NIEM Vocabularies

The Senzing Entity Specification documents available entities and features: NAME_FIRST, NAME_LAST, ADDR_LINE1, PHONE, and dozens of others. Familiarize yourself with the NIEM standard, paying attention to the object-oriented logic of NIEM. NIEM organizes its standard by types, elements and attributes.

For further guidance, the NIEM Naming and Design Rules (NDR) specifies guidelines for creating data models, namespaces, schemas, and messages, in addition to guidelines for naming conventions, syntax, semantics and RDF crosswalks.

Understanding this vocabulary enables you to design crosswalks between your domain’s metadata elements and entity resolution features. This semantic engineering work is extremely effective at supporting interoperability while preserving rich semantics, entity reconciliation and mapping logic.

Structure Your Domain Thesaurus in SKOS

If your existing taxonomies use spreadsheets or proprietary formats, migrate them to SKOS. After your taxonomy is SKOS compliant, add alternative labels, hidden labels and definitions (if absent) to create your SKOS thesaurus. The SKOS taxonomy or thesaurus produces the TTL file that sz-semantics requires—and positions your taxonomy or thesaurus for broader semantic web interoperability.

Plan for Thesaurus Extension

Entity resolution will discover synonyms, variant spellings and acronyms that your current vocabulary does not include. These newly discovered terms can be integrated into a domain taxonomy or thesaurus and represented as alternative labels.

To support this work, establish governance processes for incorporating these discoveries. Define criteria for promoting matches to canonical semantic definitions. This is where your expertise in vocabulary management will be handy, and essential to operations.

Engage With the Tooling

The sz-semantics and Strwythura repositories are open source and documented. Work through the tutorials. Understand how the thesaurus class transforms entity resolution results. Identify where your semantic assets would integrate into the workflow. Prototype with sample data before proposing production integration.

Build the Business Case

Document how each pipeline stage contributes to entity resolution quality. Estimate time and cost for each deliverable. Define metrics that demonstrate semantic infrastructure value, such as match rate improvements, false positive reductions, and disambiguation successes. Speak the language of measurable outcomes.

The Moment

The Ontology Pipeline offers the framework to support semantic engineering, from documentation practices to infrastructure discipline. Entity resolution provides the use case that demonstrates this transformation concretely.

Controlled vocabularies enable consistent entities, classes relations, properties and attributes. Taxonomies classify resolved entities meaningfully, as concept classes. The thesaurus disambiguates and documents alternative names and documentation properties.

Ontologies encode the logical constraints and semantic interpretations. Ontologists and semantic engineers are critical to making this work happen, to build robust and resilient semantic knowledge infrastructures.

Tools like sz-semantics implement the pipeline stages as executable code. The thesaurus we build becomes the infrastructure that entity resolution depends upon. The SKOS concepts we define become the URIs that identify resolved entities globally. The associative relationships we curate become the semantic edges that make knowledge graphs queryable.

Entity resolution benefits from the semantic infrastructures we build, to transform matches into knowledge.

As new frameworks and tools emerge, there are exciting opportunities to participate in new innovations that support semantic interoperability, integrations between systems and semantic data transformations.

Operationalizing RDF and semantic web standards to negotiate data mapping logic, enable transformations, and support entity resolution tasks is a fantastic opportunity for ontologists and semantic engineers that will increase demands for our skillsets.

For ontologists seeking to demonstrate the practical value of their discipline, entity reconciliation provides a compelling application domain, and sz-semantics provides the tools to realize it.

If you have any queries about the piece, feel free to connect with the author(s). Or feel free to connect with the MD101 team directly at community@moderndata101.com 🧡

Author Connect

Connect with Jessica Talisman on LinkedIn 💬 | Or dive into her work on Substack:

Intentional Arrangement

I am an Information Architect and Semantic Engineer. I have been organizing and building digital knowledge ecosystems since 1997. Intentional Arrangement is the name of my book Intentional Arrangement, to be published in 2026!

By Jessica Talisman

From MD101 team 🧡

🌎 Global Modern Data Report 2026

The Modern Data Report 2026 is a first-principles examination of why AI adoption stalls inside otherwise data-rich enterprises. Grounded in direct signals from practitioners and leaders, it exposes the structural gaps between data availability and decision activation.

With hundreds of datapoints from 500+ data leaders and experts from across 64 countries, this report reframes AI readiness away from models and tooling, and toward the conditions required and/or desired for reliable action.

Get your copy

References

DerwenAI. (2024-2026). Strwythura: Construct an entity-resolved knowledge graph from structured data sources and unstructured content sources. GitHub. https://github.com/DerwenAI/strwythura

Senzing. (2025). sz-semantics: Transform JSON output from Senzing SDK for use with graph technologies, semantics, and downstream LLM integration. GitHub. https://github.com/senzing-garage/sz-semantics

Senzing. (2025). Senzing Entity Specification. https://senzing.com/docs/entity_specification/

Talisman, J. (2025). The Ontology Pipeline: A Semantic Knowledge Management Framework. Intentional Arrangement. https://jessicatalisman.substack.com/p/the-ontology-pipeline

Talisman, J. (2025). Knowledge Graph Lite, Part III: The Mighty Thesaurus, SKOS in Action. Intentional Arrangement.

Tesfaye, L., & Jonas, J. (2025). Expert Analysis: How Do I Get Started with a Semantic Layer? Enterprise Knowledge.

W3C. (2009). SKOS Simple Knowledge Organization System Reference. https://www.w3.org/TR/skos-reference/

ANSI/NISO Z39.19-2005 (R2010). Guidelines for the Construction, Format, and Management of Monolingual Controlled Vocabularies.

ISO 25964-1. Information and documentation — Thesauri and interoperability with other vocabularies.

Vu Trinh | Data Engineer

Vu Trinh is a Data Engineer and educator known for breaking down complex data engineering concepts into simple, structured explanations. Currently working at Amanotes, a music game company, Vu combines hands-on industry experience with a deep passion for learning and teaching.

He is the creator of VuTrinh, a fast-growing newsletter read by 17,000+ data engineers worldwide. Through long-form articles and curated weekly resources, Vu explores everything from Apache Spark internals and lakehouse table formats to BigQuery pricing models and the evolution of data engineering itself. His writing is practical, research-driven, and grounded in first principles.

Based in Ho Chi Minh City, Vu’s mission is simple: cut through shallow content and help engineers truly understand how data systems work under the hood. As he often says, his mom started reading his articles to support her son, and now she can design data architectures and write ETL scripts! We’re thrilled to feature his unique insights on Modern Data 101!

We actively collaborate with data experts to bring the best resources to a 15,000+ strong community of data leaders and practitioners. If you have something to share, reach out!
🫴🏻 Share your ideas and work: community@moderndata101.com

*Note: Opinions expressed in contributions are not our own and are only curated by us for broader access and discussion. All submissions are vetted for quality & relevance. We keep it information-first and do not support any promotions, paid or otherwise.

Let’s Dive In

Apache Spark now cares about the ultra-low latency.

When it comes to stream processing, everybody usually brings Spark and Flink to the table. Flink is famous for its true stream processing: every record is processed continuously, nearly as soon as it arrives. Spark Structured Streaming, on the other hand, relies on its batch-processing engine to process a continuous stream in microbatches.

The latency is indeed higher than Flink’s; however, Spark has the advantage of its user base: many users already use Spark, and if they want to switch from batch to stream processing, they only need to tweak the code a bit.

Thus, you commonly see advice like this: “Only if you actually need millisecond latency, invest in Flink. Spark is still a good choice for second-tier latency use cases, given its simplicity and familiarity (Flink has a steep learning curve and requires significant resources to manage).

However, in the latest version, Spark introduced a new mode for Spark Structured Streaming that promises to overcome the limitations of the original approach and achieve millisecond latency, similar to Flink. The cofounder of Databricks said that this new mode could offer the latency competitive to Flink:

Source

In this article, we will dive into Spark Structured Streaming’s real-time mode to see how this feature could change the “latency game” in stream processing. We will first briefly review Spark and Spark Structured Streaming to understand their limitations in low-latency scenarios better; only then will we see how the new streaming feature addresses them.

This article only has a brief overview of Spark and Spark Structured Streaming. For the deep dives, you can check these two articles:

• If you’re learning Apache Spark, this article is for you

• Everything you need to know about Spark Structured Streaming

Spark Cluster

Spark is a distributed processing engine. Thus, there must be a cluster of nodes.

Every Spark application is associated with a Spark cluster, which comprises a driver and a set of executors. All of these nodes are JVM processes that consume resources from a physical cluster, which consists of physical machines and is managed by the Cluster Manager process.

When you submit your application, the cluster manager first initiates the driver on one of the physical machines (if you run it in cluster mode). The driver contacts the cluster manager to launch the executor.

Anatomy

Spark has hierarchical processing units:

• Job: A job represents a series of transformations applied to data. It encompasses the entire workflow from start to finish.

• Stage: A stage is a job segment executed without data shuffling. A job is split into different stages when a transformation requires shuffling across partitions.

  • DAG: In Spark, RDD dependencies are used to build a Directed Acyclic Graph (DAG) of stages for a Spark job. The DAG ensures that stages are scheduled in topological order.

• Task: A task is the smallest unit of execution within Spark. Each stage is divided into multiple tasks that process data in parallel across different partitions.

Scheduling

To provide the executor with tasks to run, the driver must first plan execution based on the user-defined logic. Then the driver starts scheduling the execution according to the plan.

This scheduling process involves some components (which also live inside the driver):

• The DAGScheduler for stage-oriented scheduling

• The TaskScheduler for task-oriented scheduling

• The SchedulerBackend interacts with the cluster manager and provides resources to the TaskScheduler.

The DAGScheduler schedules stages according to the DAG’s topological order. Each stage is submitted once all its upstream dependencies are completed. If stages don’t have dependencies on each other, they can be run in parallel.

Remember this: Stages are scheduled in topological order.

The DAGScheduler creates fully independent, unprocessed tasks for that stage. Then, the DAGScheduler sends them to the TaskScheduler. This component is responsible for scheduling those tasks on available executors. It requests resources from the SchedulerBackend to schedule tasks.

The Micro-batch Structured Streaming

Overview

After the revision on how Spark works, it is straightforward to understand how Spark Structured Streaming operates.

Its core design principle is to treat a continuous stream as a bounded dataset. When we start a streaming application in Spark Structured Streaming, we create a long-running Spark application (a Spark cluster).

The Driver process remains active at all times, managing the entire streaming query lifecycle. Each stream will have a trigger. This trigger determines when Spark checks for new data. When the trigger fires, the Spark engine does the following:

• It queries the source (e.g., asks Kafka “what are the latest offsets?”).

• It identifies the new data that has arrived since the last batch (e.g., Kafka offsets 1001 to 5000).

• This chunk of new data is a micro-batch. Internally, Spark treats it as a static DataFrame.

• And for every micro-batch, things will happen pretty much like how batch processing works in Spark: the driver, parse the logic, create/optimize plan, and schedule the stages in order.

The Delay

Although the current Spark Structured Streaming approach can leverage Spark’s robust batch engine for stream processing, it has some disadvantages, especially regarding the low latency performance:

• First, the stages are scheduled in order, which means the stage can only be scheduled when all its upstream stages are completed.

• For every batch, the driver needs to coordinate the communication between the nodes, parse, plan, schedule the stages/tasks, and commit the output. These overheads are included in each record-processing latency.

That’s why some online sources suggest that if you prefer super-low latency, you should go with Flink rather than Spark.

The Real-time Mode

However, the Spark 4.1 release (Dec 16, 2025) released a very cool feature: Structured Streaming Real-Time Mode (RTM).

Databricks introduced this feature in 2025 August and later contributed it to the community version. (Source)

The RTM targets millisecond latency for use cases, while the original micro-batch is suitable for seconds-latency use cases. RTM achieves this through these clever changes (not exhaustive, but I believe these are the crucial changes)

Consuming Fresh Data

In micro batch mode, the source reading process occurs only when a trigger is present. In real-time mode, the engine keeps reading fresh, available data from the source in longer-running batches (default: 5 minutes).

Stages Scheduled Concurrently

Unlike in (micro)batch processing, where stages are scheduled sequentially, RTM schedules all stages concurrently, eliminating wait time for upstream stages before scheduling a new one.

Longer batches and concurrent scheduling of stages mean lower coordination and planning overhead.

Streaming Shuffle

If you work with Spark, you must know about data shuffle. There are two transformations in Spark: narrow and wide dependencies.

The first is where each partition in the child RDD has a limited number of dependencies on partitions in the parent RDD. These partitions may depend on a single parent (e.g., the map operator) or on a specific subset of parent partitions known beforehand (e.g., with coalesce).

This means that operations like map and filter do not require data shuffling. RDD operations with narrow dependencies are pipelined into a single set of tasks per stage (e.g., map and filter can be baked into a single stage)

The latter requires data to be partitioned in a specific way, where a single partition of a parent RDD contributes to multiple partitions of the child RDD. This typically occurs with operations such as groupByKey, reduceByKey, or join, which involve data shuffling. Consequently, wide dependencies result in stage boundaries in Spark’s execution plan.

Thus, when a shuffle occurs, data must be exchanged between executors.

When thinking about Spark, people often assume that every operation must run in memory. That’s not 100% true. Although Spark tries to keep data processing in memory and spills only when needed, data shuffling occurs on the executor’s local disk.

The “map” tasks will write shuffle data to the executor’s local disk. They first write data to a memory buffer. When the buffer is full, they spill data to temporary disk files and merge them into the final shuffle files.

The “reduce” tasks communicate with the shuffle service, pull data from the target executor’s local disk into the memory buffer, and then start the processing.

However, exchanging data like this might not meet the RTM latency requirement.

Thus, RTM allows data to be passed between stages as soon as it is produced using an in-memory streaming shuffle.

Note: I only know that streaming shuffle allows for faster data exchange between stages in memory. I can’t find the details of the streaming shuffle implementation anywhere at the moment. Hope more resources are released in the future.

Checkpointing

In micro-batch mode, checkpointing is done straightforwardly. During the planning phase, the driver already knows which data ranges (the batch) the cluster must process, and the start and end of that batch are written to the checkpoint location before processing.

However, in real-time mode, things become more challenging because the driver doesn’t know the batch’s end boundary beforehand.

Thus, the strategy must change a bit.

Checkpointing occurs only at the end of batch processing, rather than at the beginning.

If the batches have a wider data range, RTM checkpointing occurs less frequently, reducing overhead but increasing the read workload because the engine must read a broader range of data from the source when failures occur.

If the batch is smaller, checkpointing occurs more often, which harms the low-latency. When failure happens, the recovery process might be faster as it reads a narrower range of data from the source.

My Thoughts

The key thing about this feature is that the user only needs to make a minor change to the micro-batch code.

The user needs to change the trigger to RealTimeTrigger. Source

This convenience allows users to seamlessly switch to the new mode if the latency performance of the original Spark Structured Streaming does not satisfy them anymore

For me, this is a game-changer.

In the past, the only way to achieve that lower-latency performance was to use Flink. However, managing, operating, and developing Flink requires significant effort (as much as you use Spark).

Now with RTM, users can consider this new mode instead of Flink. If users can leverage RTM, they don’t need more resources to manage Flink and still stay in the Spark ecosystem, which is a big win for any organization that has been investing in Spark (e.g., engineers who are only familiar with Spark)

Although the real adoption of RTM is still being observed, and Flink won’t be replaced anytime soon (its true stream processing capabilities, robust watermarking, and state management still make it a strong stream processing engine), RTM will change how companies choose a stream processing engine.

Final Note

I opened this article out of curiosity about Spark Structured Streaming’s real-time mode, which promises to deliver competitive low-latency performance compared to Flink. Then we have a brief review of how Spark works as a cluster, its processing units, and the scheduling process. Next, we revisit Spark Structured Streaming, which processes continuous data in small batches, each handled in the same way as Spark batch processing.

From there, we understand the factors that impact the latencies of the micro-batch processing approach. Finally, we see how the real-time mode solves these problems by consuming fresh data, scheduling stages concurrently, using a streaming shuffle, and employing a different checkpointing strategy.

Thank you for reading this far. See you in my next articles.

MD101 Support ☎️

If you have any queries about the piece, feel free to connect with the author(s). Or feel free to connect with the MD101 team directly at community@moderndata101.com 🧡

Author Connect

Connect with Vu Trinh on LinkedIn 💬 | Or dive into his work on Substack:

VuTrinh.

My mom reads my articles to support her son. Now, she can design a data architecture and write ETL scripts.

By Vu Trinh

From MD101 team 🧡

🌎 Global Modern Data Report 2026

The Modern Data Report 2026 is a first-principles examination of why AI adoption stalls inside otherwise data-rich enterprises. Grounded in direct signals from practitioners and leaders, it exposes the structural gaps between data availability and decision activation.

With hundreds of datapoints from 500+ data leaders and experts from across 64 countries, this report reframes AI readiness away from models and tooling, and toward the conditions required and/or desired for reliable action.

Get your copy

References

[1] Databricks, Real-Time Mode Technical Deep Dive: How We Built Sub-300 Millisecond Streaming Into Apache Spark™ (2025)

[2] Databricks, Real-time mode in Structured Streaming Documentation

Adapted from concepts shared by Author(s) | Curated by Modern Data 101

Why Unified Standards Emerged

Before we speak of protocols, platforms, or products, we must reduce the system to its irreducible parts. Strip away branding, tooling, and terminology, and what remains is intelligence attempting to act in an environment.

Intelligence, by definition, operates on context. Remove structured context, and the model substitutes probability for truth. But context alone is insufficient. Intelligence must also have a defined surface through which it can act.

And even action requires constraint. Without that, intelligence becomes either paralysed or dangerous.

So any system that enables AI must solve three things:

• structured context,

• controlled action, and

• bounded execution.

Systems Scale Only Through Abstraction

When integrations multiply without a unifying layer, the architecture begins to decay. Point-to-point connections proliferate until the system resembles a fragile web of dependencies rather than a coherent structure.

As this happens, context fragments. Each integration carries its own assumptions, schema, and semantics. What was once a shared understanding becomes a localised interpretation. Security follows the same pattern. Without a standard interface, every connection defines its own authentication model, permission logic, and failure modes.

Historically, the escape from this entropy has always been the same: standardised interfaces that abstract variability.

• HTTP abstracted the variability of machines into a universal language of web communication.

• SQL abstracted storage engines into a shared grammar for querying data.

• Kubernetes abstracted infrastructure into declarative orchestration.

MCP (Model Context Protocol) performs the same move for intelligence. It standardises the interaction surface between AI and the systems it must reason about and act upon. It is not another integration, but the abstraction that makes integrations composable.

MCP as a Unified Standard

AI interaction can only do three things: read, act, or be guided. Everything else is a composition of these. MCP formalises this into three orthogonal primitives.

These primitives are independent by design. Each has a clear owner, a clear boundary, and a clear responsibility. Together, they form a minimal but complete interaction model.

1. Tools (Action Surface)

Tools are the verbs used by the system. They are executable capabilities that allow intelligence to affect the external world. Like write to a database, call an API, modify a file, trigger a workflow.

They are model-controlled, meaning the model decides when to invoke them. But they are not free-form. They are constrained by typed inputs defined through JSON Schema.

Every invocation is auditable. Action is no longer implicit in code hidden behind prompts as it becomes explicit, structured, and reviewable.

2. Resources (Ground Truth)

Resources are the nouns used by the system. They represent passive read-only contextual data: the ground truth against which reasoning occurs. Examples include file contents, database schemas, or API documentation.

They are application-controlled. The host determines what context is relevant and safe to provide, anchoring the model’s reasoning in bounded reality.

3. Prompts (Workflow Templates)

Prompts are like structured intent. They are reusable workflow templates that guide how tools and resources should be composed.

They are user-controlled. Invocation is explicit, not inferred, preserving agency in complex operations. They help structure interactions and provide a consistent way to initiate complex tasks by telling the model how to work with specific tools and resources.

Unified Architecture Pattern Inside MCP

MCP unifies AI interaction across two independent dimensions: semantics and transport. This separation is what allows the protocol to scale without collapsing into coupling.

A. Context Layer (Data Layer)

At the data context layer, MCP defines meaning. It uses a structured message format, ensuring that communication is not ad hoc but typed and predictable.

The MCP primitives (tools, resources, prompts) exist here as structured capabilities. They are discoverable through explicit methods like tools/list and resources/list, which means capability is declared.

Version negotiation also belongs to this layer. Before execution, both sides agree on what they understand. Compatibility becomes a first-class concern rather than an afterthought.

B. Transport Layer

Beneath context and semantics is transport. MCP supports multiple communication channels. Authentication mechanisms operate here. Security is enforced at the communication boundary, independent of semantic meaning. Transport is interchangeable, while meaning is not.

…

By decoupling these, MCP mirrors every mature architecture. Abstraction at the right boundary is what makes a system extensible. MCP applies that approach to intelligence itself.

Data Products: The Unified Standard (For Purpose)

A Data Product is unified around purpose.

What Makes a Data Product a Unified Standard

A standard exists when variability is constrained by contract. A Data Product becomes a unified standard when it defines what must remain stable and independent, even as everything around it changes state.

It standardises:

• Ownership: There is a clearly defined steward. Responsibility is not diffused across teams or hidden inside pipelines.

• Accountability: Quality, availability, and fitness-for-purpose are measurable and enforced.

• Interface: Access is intentional and documented. Consumers interact through a defined surface instead of using implicit tribal knowledge.

• Consumer obligation: The producer acknowledges the needs of a specific audience. Reuse is designed.

• Lifecycle: Creation, evolution, deprecation → each phase is governed. The product has a temporal boundary.

Architectural Resonance: Data Product ↔ MCP

When two systems are built from sound first principles of data, they will mirror each other. Data Products and MCP are visibly resonating for us because they solve adjacent problems with the same architectural language.

• A Data Product defines an interface for consumers. In MCP, that interface becomes a tool schema (a typed contract that intelligence can safely invoke).

• A Data Product provides context shaped for a specific purpose. In MCP, that context is formalised as resources.

• A Data Product encodes workflows that guide how it should be used. In MCP, prompts structure execution, turning intent into a repeatable interaction pattern.

• Ownership in a Data Product establishes bounded responsibility. In MCP, the server boundary enforces controlled scope, defining where authority begins and ends.

• Observability ensures that a Data Product can be trusted. In MCP, activity logs and auditable tool calls make every action inspectable.

MCP operationalises Data Products for AI. It translates purpose-bound economic units into machine-consumable contracts.

Role of MCP in Data Product Development

If Data Products are built for purpose, they must eventually be built for intelligence. The moment AI becomes a consumer, the interaction surface must be formalised, and MCP provides that formalisation.

MCP enables:

1. AI-native Data Product interfaces: Instead of wrapping products in ad hoc prompts or brittle middleware, it exposes typed, discoverable contracts that intelligence can reliably invoke.

2. Declarative action surfaces: Tools are no longer hidden behind code or implicit workflows. They are explicit capabilities with defined schemas and auditable execution.

3. Safe execution boundaries: Action is constrained by permissions, approvals, and scoped servers, ensuring that intelligence operates within governed limits.

4. Dynamic context retrieval: Through structured resources and templates, products expose live, parameterised ground truth rather than static extracts.

5. Multi-server orchestration: A single host can compose multiple Data Products into a unified capability surface, without collapsing ownership boundaries.

With MCP, Data Products become plug-and-play intelligence nodes. The contract is stable, the boundaries are clear, and intelligence can scale without architectural entropy.

Unified Standard for Builders: DDP

If Data Products unify around purpose, something must unify how they are built. Scale cannot depend on artisanal engineering.

A Data Developer Platform (DDP) standardises the conditions under which Data Products are created, governed, and evolved. It reduces variance at the builder layer so that the purpose can be delivered reliably.

A DDP standardises:

• Modelling: Data structures are designed through shared conventions.

• Governance: Policies are embedded into workflows rather than enforced retroactively.

• Observability: Health, quality, and performance are measurable by default.

• Access control: Permissions are embedded and reproducible.

• CI/CD: Change is versioned, tested, and promoted.

• Environment management: Development, staging, and production operate under consistent rules.

• Metadata: Context about data is first-class, not an afterthought.

• Product lifecycle: Creation, iteration, and deprecation follow defined processes.

DDP’s Core Layers

A Data Developer Platform resolves into distinct layers:

• Compute Layer: Where transformations execute, and logic materialises into state.

• Storage Layer: Where data persists, versioned and retrievable.

• Metadata Layer: Where meaning about data (lineage, ownership, definitions) is formalised.

• Orchestration Layer: Where dependencies, timing, and execution order are coordinated.

• Governance Layer: Where policy, quality, and compliance are enforced.

• Interface Layer: Where products expose their contracts to consumers.

• Experience Layer: Where builders and stakeholders interact with the system.

Each layer isolates a fundamental concern. Together, they transform scattered tooling into coherent infrastructure. DDP ensures that builders can reliably produce standardised Data Products.

MCP ↔ DDP: Architectural Interlock

MCP connects intelligence to Data Developer Platforms. If DDP industrialises data production, MCP industrialises AI interaction with that production. The two systems interlock at their boundaries.

MCPs standardise how intelligence reads from, reasons about, and acts upon the data developer platform capabilities.

DDP governs the infrastructure. MCP governs AI interaction with it. Together, they close the loop between governed data production and intelligence-driven operations at scale.

Three Unified Standards, Three Planes

The architecture resolves into three distinct planes. Each plane solves a different problem, yet all follow the same structural logic.

Purpose (Data Product)

The first plane is economic. Data Products are unified around outcome. They exist to serve a defined consumer, with measurable utility and bounded responsibility.

Platform (DDP)

The second plane is industrial. The Data Developer Platform is unified around builder productivity and governance. It standardises how products are created, evolved, and controlled at scale.

Intelligence Interface (MCP)

The third plane is cognitive. MCP is unified around AI-system interoperability. It standardises how intelligence reads context, invokes capability, and operates within boundaries.

Across all three planes, the same architectural language appears:

• Interfaces

• Contracts

• Ownership

• Boundaries

• Discovery

• Versioning

• Observability

These are structural requirements of scalable systems which make the convergence click.

Business Purpose → Architectural Language → Direct Value

Architecture is the translation between business intent and operational reality. The resonance of the three systems that manage economic value, infrastructure, and communication with humans and machines enables us to streamline and plan value like never before.

Reduced Integration Cost

A unified AI interface eliminates bespoke connectors. Instead of rebuilding integration logic for every tool and dataset, intelligence interacts through a stable contract. Cost declines because variability is abstracted.

Faster Productisation

When Data Products expose typed interfaces and governed context, they are AI-ready by design. Intelligence does not need to reverse-engineer meaning.

Safer Automation

Automation without boundaries is the most fragile governance model that unfortunately exists in many organisations to date. MCP introduces human-in-the-loop checkpoints, typed tool validation, and scoped execution roots, orchestrated by DDP.

Composable Intelligence

Multiple servers can be orchestrated as a single capability surface. Intelligence no longer operates within isolated silos but across coordinated domains. Composition replaces integration as the scaling mechanism.

Governance Alignment

Permissions operate at the tool level. Every action is logged, auditable, and reviewable. Approvals are structured, not implicit. Governance is embedded in execution.

Platform Leverage

A Data Developer Platform ensures reusable metadata, standardised modelling, and governed data exposure. It stabilises the production layer. MCP ensures standardised AI execution and stabilises the interaction layer.

Final Note

MCP is the standardisation of intelligence interaction. Data Products are economic interfaces tied directly to business value. A Data Developer Platform make the industrialisation layer.

When aligned, the organisation moves from Pipelines to Products to Programmable intelligence surfaces.

Author Connect 💬

Find me on LinkedIn 🙌🏻

Find me on LinkedIn 🤝🏻

MD101 Support ☎️

If you have any queries about the piece, feel free to connect with the author(s). Or feel free to connect with the MD101 team directly at community@moderndata101.com 🧡

From MD101 team 🧡

🌎 Global Modern Data Report 2026

The Modern Data Report 2026 is a first-principles examination of why AI adoption stalls inside otherwise data-rich enterprises. Grounded in direct signals from practitioners and leaders, it exposes the structural gaps between data availability and decision activation.

With hundreds of datapoints from 500+ data leaders and experts from across 64 countries, this report reframes AI readiness away from models and tooling, and toward the conditions required and/or desired for reliable action.

Get your copy

John Wernfeldt | Managing Director: Data, Analytics & AI

John Wernfeldt works with CDOs and senior data leaders who are under pressure to “be AI-ready” while still struggling with trust, ownership, and decision clarity in their data. His focus is practical data governance and foundations, the kind that reduce rework, end metric debates, and give executives confidence to act.

Based in Stockholm, John is Managing Director at Northridge Analytics, where he helps organisations turn data and analytics into tangible business value, and President of DAMA Sweden. His background spans data strategy, governance, architecture, and analytics leadership roles at Gartner, Capgemini Invent, and KPMG.

Known for cutting through hype, John emphasises clear decision rights, fit-for-purpose quality, and governance embedded into delivery. His work helps organisations move from fragmented, fragile data to trusted foundations that scale with analytics and AI. We’re thrilled to feature his unique insights on Modern Data 101!

We actively collaborate with data experts to bring the best resources to a 15,000+ strong community of data leaders and practitioners. If you have something to share, reach out!
🫴🏻 Share your ideas and work: community@moderndata101.com

*Note: Opinions expressed in contributions are not our own and are only curated by us for broader access and discussion. All submissions are vetted for quality & relevance. We keep it information-first and do not support any promotions, paid or otherwise.

Let’s Dive In

I’ve never seen a “data quality issue” that wasn’t, at its core, an ownership problem. I’ve seen this play out more times than I can count.

A number looks off, someone flags it, everyone agrees it’s “important.” And then the questions begin. Sales uses it in forecasts, finance uses it in reporting, marketing uses it in KPIs, analytics builds dashboards on top, and IT moves the data and gets asked to fix it.

When a metric breaks, the same explanations always show up

When the number is questioned, the conversation goes in predictable directions.

A recent survey on the state of data and AI in enterprises conclusively suggests how the same metric gets corrupted as it changes hands. Because no one in particular owns the health of the metric. Many of us touch the same metric, and we end up seeing different versions.

93% of respondents say they encounter conflicting metrics. Nearly half do not fully trust their own data, and 68% explicitly state that it is not trustworthy enough for AI

- Findings from the Modern Data Report 2026

There could be many reasons behind this experience. For example, the definition is unclear, the source system changed, there was manual input, and the data quality dropped. All of these things are often true, but they’re also mostly irrelevant.

Because the only question that actually matters never gets answered:

When that person doesn’t exist, the discussion goes nowhere.

The cycle that follows is always the same

When ownership is missing, organisations fall into a loop that looks like this:

The metric breaks, and a hero steps in to fix it. People argue about whose fault it is, and a temporary workaround is put in place. Then everyone moves on. Only until next month.

This is why so many teams believe they have reporting problems, BI problems, or tooling problems. What they truly have is a governance gap.

Why “data quality” becomes the wrong label

Calling this a data quality issue is convenient. It allows the organisation to:

• Treat the problem as technical

• Push responsibility toward IT or analytics

• Focus on fixing symptoms instead of decisions

Data quality becomes a proxy for unresolved ownership. But quality cannot exist in a vacuum. It only improves when someone is accountable for the outcome. Until then, every fix is temporary.

The problem is decision authority, not data. At its core, a metric is not a technical artifact. It’s a decision artifact. It exists so someone can make a decision, explain it, and stand behind it.

If no one has the authority to:

• Define the metric

• Approve changes

• Reject misuse

• Be accountable when it’s wrong

Then the metric will always be fragile, no matter how clean the pipeline looks.

A simple model to fix decision authority for data

This situation doesn’t require a new tool, a maturity program, or a framework rollout. It requires making ownership explicit and enforceable.

Here’s a proven practical model to improve or establish decision authority.

1. Name a single accountable owner

Not a team, a forum, or “the business”. One named person who is accountable for:

• The meaning of the metric

• The calculation logic

• When it is allowed to change

This person may delegate work, but cannot delegate accountability.

2. Separate ownership from contribution

Many functions will contribute to a metric. That’s fine. But contribution is not ownership. We need to make it explicit:

• Who provides inputs

• Who builds pipelines

• Who validates data

• Who approves changes

Only one person owns the outcome.

3. Define change rules upfront

Most metric chaos comes from silent change. Every metric needs clear answers to:

• What counts as a breaking change

• Who approves changes

• How changes are communicated

• When historical values are recalculated

If this isn’t defined, change will happen anyway. Just informally.

4. Tie data quality rules to ownership

Data quality rules should never exist in isolation. For each critical rule:

• Name an owner

• Define acceptable thresholds

• Agree on consequences when it breaks

Quality improves when breaking the rule creates friction instead of tickets.

5. Enforce, don’t escalate

If every issue escalates to a steering group, ownership is fake. Owners need:

• The mandate to say no

• The ability to block changes

• Support from leadership when decisions upset someone

Without enforcement, governance has no authority.

The Sovereign Data Ownership Canvas

What changes when this model is enforced

When ownership is explicit and enforced, metrics stop drifting, quality issues surface earlier, blame games decrease, fixes stick, and trust increases. Data is almost never perfect, but these changes are kicked off because accountability is clear.

The uncomfortable part

Most organisations avoid implementing such a model because it’s uncomfortable. It forces decisions instead of alignment, accountability instead of consensus, and clear trade-offs instead of vague agreements.

But avoiding discomfort is exactly what creates recurring data quality issues.

Final Note

Adapted from concepts shared by the author | Curated by Modern Data 101

If a metric matters, someone must be accountable for it. Until that happens, data quality initiatives won’t stick, BI tools won’t help, and AI will amplify the problem instead of solving it.

Until we have fixed the root of the problem and assigned an owner, nothing downstream really matters.

I’m collecting practical tools like this in my Substack, starting with a Metric Specification Template I actually use. If that’s useful, it lives here: link to metric specification template

MD101 Support ☎️

If you have any queries about the piece, feel free to connect with the author(s). Or feel free to connect with the MD101 team directly at community@moderndata101.com 🧡

Author Connect

Connect with John Wernfeldt on LinkedIn 💬 | Or dive into his work on Substack:

Data Governance Field Library

Practical tools, templates, and resources for data governance practitioners and leaders

By John Wernfeldt

From MD101 team 🧡

🌎 Global Modern Data Report 2026

The Modern Data Report 2026 is a first-principles examination of why AI adoption stalls inside otherwise data-rich enterprises. Grounded in direct signals from practitioners and leaders, it exposes the structural gaps between data availability and decision activation.

With hundreds of datapoints from 500+ data leaders and experts from across 64 countries, this report reframes AI readiness away from models and tooling, and toward the conditions required and/or desired for reliable action.

Get your copy

Olawale Oyeneye | Senior Data Governance Advisor

Olawale Oyeneye is an accomplished Data Governance leader with extensive experience designing and implementing enterprise-wide data governance frameworks across complex, regulated organisations. As Senior Data Governance Adviser at The Pensions Regulator, the UK regulator of work-based pension schemes, he plays a pivotal role in strengthening governance controls across cloud platforms, applications, and systems.

A Certified Data Management Professional (CDMP) and ISO 27001:2022 ISMS-certified practitioner, Olawale is widely recognised for his expertise and has successfully delivered governance programmes spanning multiple business units and regions, defining critical data elements, building business glossaries, establishing data ownership, and championing the adoption of governance tooling. We’re thrilled to feature his unique insights on Modern Data 101!

We actively collaborate with data experts to bring the best resources to a 15,000+ strong community of data leaders and practitioners. If you have something to share, reach out!
🫴🏻 Share your ideas and work: community@moderndata101.com

*Note: Opinions expressed in contributions are not our own and are only curated by us for broader access and discussion. All submissions are vetted for quality & relevance. We keep it information-first and do not support any promotions, paid or otherwise.

Let’s dive in

Most organisations believe they have a data governance problem. But what they actually have is a practice problem. Governance, in its formal sense, is rarely missing.

Policies, frameworks, maturity models, all exist. What is missing is their presence in the ordinary moments where data is created, interpreted, and acted upon.

It would shape which metric is trusted in a meeting, which dataset is reused instead of recreated, and which issue is escalated before it becomes visible in an audit.

Instead, governance tends to live outside the work itself: captured in documents and decks, consulted only after something has already gone wrong.

But governance cannot be defined by its existence on paper. It can only be defined by its effect on behaviour.

Anything that does not change how people act, decide, or take responsibility in their daily work does not govern anything at all. It may inform, it may advise, but it does not govern.

This reframes how governance artefacts should be understood. Policies and frameworks are not truths to be adopted by default. They are just hypotheses. Each one makes an implicit claim: if this is followed, outcomes will improve.

That claim is only validated when the policy survives contact with real workflows, real incentives, and real time pressure. Until then, governance is not something an organisation has, but something it assumes.

Reduce Governance to Irreducible Components

Governance fails when it is treated as something large, centralised, and abstract. When it becomes an artefact to be owned by a function, a compliance exercise to be satisfied periodically, or a retrospective control applied after damage has already been done, it loses contact with reality. In those forms, governance is always too far away, too slow, and too detached from the moments that actually matter.

To understand why, the problem has to be reduced to its smallest truths. Governance does not begin with committees or tools.

• It begins with behaviour: what people actually do with data when no one is watching.

• It continues with language: whether people can explain what the data means, where it comes from, and when it should or should not be trusted.

• It depends on authority: who is allowed to fix issues, make decisions, and say no.

• And it is sustained by feedback: what gets measured, noticed, and reinforced over time.

Strip away everything else, and these are the irreducible components. Change behaviour without changing language, and confusion follows. Define roles without authority, and governance stalls. Measure artefacts instead of actions, and adoption becomes performative.

Governance is not a system that can be installed or rolled out. Systems are static. Governance is dynamic: a pattern of repeated, local decisions, made under pressure, with imperfect information. When those decisions converge toward trust, governance exists. When they don’t, no framework, however elegant, can compensate.

🔖 Related Read

Rebuild Governance in the Correct Order

From simple, observable actions to complex outcomes.

Start with Behaviour over Policy

Governance begins at the moment data is created, touched, reused, or questioned. If governance is not present there, it is already too late. The aim is not to ask people to remember rules, but to shape environments where the right behaviour is the default.

When tagging, validation, and checks happen naturally in the flow of work, governance stops feeling like an extra task. It becomes muscle memory.

Translate Policy into Human Language

Most governance policies crumble because they are written to satisfy auditors instead of guiding users. If the people using data cannot explain what a policy means for their work in a short conversation, the policy has no operational value.

Governance only becomes real when it is expressed in everyday language. Language that connects abstract principles to concrete actions. Understanding must come before compliance, and comprehension must come before trust. Without that sequence, policies remain technically correct and practically irrelevant.

Empower the Roles Closest to the Data

Frameworks do not fix issues. People do. Governance collapses when authority is centralised far away from the data, or when responsibility is assigned without the power to act.

Trust only scales when the people closest to the data are trusted to make decisions, resolve issues, and escalate when needed. Governance moves from theory to practice when decision rights move downward, while visibility and accountability remain upward.

🔖 Related Read

Measure Behaviour Over Artefacts

What gets measured defines what gets taken seriously. Counting how many assets are catalogued or how many policies are signed measures effort instead of impact. Real governance shows up in behaviour: which datasets are reused, how quickly issues are resolved, and whether people recommend governed data to others.

Voluntary trust is the strongest signal of success. When people choose governed data without being told to, governance is no longer enforced since it is already working.

Leadership as Living Proof

Governance becomes culture only when it is modelled. Culture is not shaped by what leaders approve in principle, but by what they default to in practice. When leaders ask for data quality in reviews, use governed datasets in decisions, and reward teams for doing the right thing with data, governance gains legitimacy. When they don’t, no amount of policy can compensate.

🔖 Related Read

Review Until Nothing Is Missing

Most conversations about governance end with structures like new councils, new checkpoints, or new artefacts. That instinct is understandable, but it misses the point.

Governance succeeds when it no longer needs to announce itself, when it no longer requires a meeting, a reminder, or a mandate.

When governance vanishes as an agenda item and reappears as instinct, something fundamental has shifted. People no longer ask whether they should follow governance, they assume it as part of how work gets done.

Decisions are made with an implicit awareness of quality, lineage, ownership, and impact, without anyone having to invoke the framework by name.

Governance that must be enforced has already failed.

The future of data governance will not be written into longer documents or stricter controls. It will be enacted subtly, repeatedly, and everywhere data moves. The goal is to get to a day when governance becomes reflex rather than a requirement.

MD101 Support ☎️

If you have any queries about the piece, feel free to connect with the author(s). Or feel free to connect with the MD101 team directly at community@moderndata101.com 🧡

Author Connect

Connect with Olawale Oyeneye on LinkedIn 💬

From MD101 team 🧡

🌎 Global Modern Data Report 2026

The Modern Data Report 2026 is a first-principles examination of why AI adoption stalls inside otherwise data-rich enterprises. Grounded in direct signals from practitioners and leaders, it exposes the structural gaps between data availability and decision activation.

With hundreds of datapoints, this report reframes AI readiness away from models and tooling, and toward the conditions required and/or desired for reliable action.

Get your copy

🧡 The Data Product Playbook

Here’s your own copy of the Actionable Data Product Playbook. With 3500+ downloads so far and quality feedback, we are thrilled with the response to this 6-week guide we’ve built with industry experts and practitioners. Stay tuned to moderndata101.com for more actionable resources from us!

DOWNLOAD!

The Modern Data Report reveals how enterprise data is nowhere near AI-ready. AI cannot reason over data that humans themselves cannot rely on.

Through this survey, we are able to observe the precise reasons behind the current state of the data stack, and why so many data leaders, subject matter experts, and practitioners across industries report the same experience: despite years of investment in modern platforms, data still fails at the moment it is needed most.

What emerges is not a shortage of tools, ambition, or technical sophistication. It is a breakdown in activation. AI initiatives struggle to move beyond experimentation because the underlying data is difficult to find, hard to interpret, and only conditionally trusted. Humans have learned to compensate for these gaps through meetings, intuition, and manual checks. AI cannot.

About this Survey

The Modern Data Report 2026: The Data Activation Gap is our second annual survey in the Modern Data 101 community. This year, more than 540 members participated, spanning 64 countries and 29 industries, including and not limited to financial services, manufacturing, healthcare, retail, technology, and government sectors, with an average of over a decade of experience.

Get your copy

Key Findings

• Enterprises have modern data stacks, but their data still can’t be trusted or activated: nearly half can’t fully rely on their data for decisions, and most say it isn’t ready for AI.

• The real work of data teams is no longer analysis. It’s search, reconciliation, and validation: finding and confirming data consumes more time than actually using it.

• Context is fragmented across tools, breaking meaning, lineage, and shared definitions: data exists, but the knowledge required to interpret and act on it does not travel with it.

• AI adoption is blocked by weak data foundations, not model capability: without semantic layers, standardised metrics, and trust signals, AI cannot reliably trigger decisions or close feedback loops.

• Practitioners are aligned on the fix: convergence over complexity: unified access, embedded governance, shared semantics, and self-service are the foundations required to close the data activation gap.

AI: The Breakthrough that Became the Stress Test

Enterprises did not adopt AI to build better dashboards or generate clever insights. They adopted AI to compress time, to collapse the distance between data and action, between detection and response, between knowing and doing. AI was meant to turn data into motion, not interpretation.

What surprised many organisations is that AI didn’t accelerate decision-making but exposed how fragile the path to decisions actually was.

The report makes this visible in uncomfortable ways.

Almost 70% of respondents say their data isn’t clean or trustworthy enough for AI. 65% say their data lacks the clarity and business context AI needs to be useful.

Only 30% use AI for automated data preparation, even though data quality is universally acknowledged as a bottleneck. More than a third cannot trigger operational decisions using AI, and a similar share reports that their AI systems don’t meaningfully connect to their data systems at all.

For years, humans acted as the compensating layer. The system appeared to function because people absorbed its inconsistencies. AI removes that human buffer. AI doesn’t tolerate weak foundations and increasingly surfaces them.

AI Changes What “Data Readiness” Actually Means

For years, data readiness was defined by proximity to storage and presentation. If data was stored, modelled, and visualised, it was considered usable. Readiness meant that someone, usually an analyst, could find it, explain it, and make sense of it when needed.

In an AI-driven system, data doesn’t get the benefit of tribal knowledge. Models don’t browse teams channels, and agents don’t remember why Metric A is “more correct” than Metric B, as discussed on a call. AI doesn’t intuit which dashboard is trusted or which table is unofficial. It only knows what the system itself makes explicit.

That changes the bar for readiness entirely, with respect to how we end up using that data, or who ends up consuming it.

Desired State

Data must be discoverable without insider context. It must be interpretable without explanation and must be trustworthy without manual validation. And most importantly, it must be actionable at machine speed instead of human pace. After all, that’s why we do AI for.

The report signals this shift clearly.

80% of respondents rank a semantic layer with standardized definitions as the most important enabler of AI: above AI tools themselves and above faster processing.

AI doesn’t reward more data or better models. It rewards systems that encode meaning, context, and trust directly into the data layer. Data readiness for AI, or AI-readiness, is about whether machines can reason without asking for help.

AI is Failing Exactly Where Humans Are Unable to Cope

Automation depends on certainty. If humans can’t reliably discover the right dataset, the right table, or the right metric, agents and models have no chance. AI doesn’t explore, infer intent, or guess which version is “the real one.” Discovery delays don’t merely slow analysis but prevent automation from starting at all.

Most of the respondents suggest that discovery is where this gap first becomes visible. 89% of respondents say finding the right data is among their top three time drains.

Trusting the data and quality available is right behind, with 74% citing it as an aspect that consumes most of their time. Understanding the data follows closely, with 71% ranking it in their top three, underscoring how missing context, definitions, and ownership slow progress even further.

For human users, this manifests as delay, while AI goes into failure modes or worse, hallucinations on misdirected discovery.

Context Is Non-Negotiable

Data without context is fundamentally unusable for AI. Over half of practitioners, 57%, report struggling to interpret data simply because the necessary context is missing.

Lineage, the map of where data comes from and how it evolves, is absent for 53%, leaving both humans and AI blind. More than a third cite missing business context as the largest platform gap, and 35% cannot even close AI feedback loops because semantic layers are absent.

The findings are unavoidable: AI does not infer meaning the way humans do. It does not “understand.” It computes, executes, and propagates logic based strictly on the definitions provided. When those definitions are incomplete or ambiguous, AI does not hesitate, stumble, or pause. It makes decisions with full confidence and hallucinations. Context has always been a prerequisite for humans, more so in the age of AI.

Conditional Trust Prevents Autonomous Action

Trust is not binary in modern data systems. Trust in data and business execution is conditional, negotiated, and fragile. And respondents say there’s enough reason for it, given how they cannot trust metrics even through human-led channels of analysis.

93% of respondents say they encounter conflicting metrics. Nearly half do not fully trust their own data, and 68% explicitly state that it is not trustworthy enough for AI (refer: figure 1)

These are not edge cases, but the majority’s vote. These are the dominant operating conditions under which AI is being asked to act.

Humans cope by improvising. They cross-check numbers across dashboards, ask colleagues which metric is “the real one,” and delay decisions until consensus, intuition, or authority fills the gap left by unreliable data. This is inefficient, but it works because humans can suspend action when trust is low.

AI cannot. For an autonomous system, trust must be explicit and machine-readable. No trust signal means no action at all, or worse, action taken on the wrong signal with absolute confidence. This is why AI adoption stalls precisely where the stakes rise, like enterprise autonomy. While that’s the goal most data leaders will to establish, the reality is that the moment decisions matter, conditional trust becomes insufficient, and autonomy collapses.

The Data Stack Was Built for Humans, We Need a Rebuild for Machine Users

The current state of the data stack was designed around a consequential assumption: the primary consumer of data is a trained human analyst.

Meaning was never expected to exist concretely and definitively in the system itself. It mostly existed in people’s heads, scattered documents, and meetings where ambiguity could be resolved socially. Validation was not formalized but negotiated. When numbers conflicted, humans stepped in to arbitrate, which is why the state of data systems never grew beyond dependency. To date, as we observed above in the article, 84% of respondents say they encounter conflicting metrics.

AI breaks this model completely. Machines cannot compensate for fragmentation with intuition or context-sharing. They cannot infer which definition is “correct” this week, or which metric the business actually cares about right now. For machines, meaning must be embedded directly into the system: explicit, durable, and machine-readable. Anything implied might as well not exist.

The consequences show up clearly in the data. Almost 70% experience rework within a single quarter. 66% report productivity loss. More than half point to strategic missteps, and nearly half link revenue loss directly to data issues.

This signal does not yet suggest failed investments or goals, and is not typically a surprise either. It’s a predictable outcome of a stack optimized for human coping mechanisms.

The Industry Signals Convergence of the Data Stack as the Emerging Requirement

The report makes something clear that is easy to misread if framed incorrectly. 77% say a converged platform would be valuable, and nearly 40% go further, calling it extremely valuable. This is not a soft signal of interest or curiosity, but we detect real strain behind such strong signals.

The correct reframing is not vendor fatigue. Organizations are not tired of too many tools, they are exhausted by fragmented architecture. Each additional system introduces another boundary where meaning, trust, and context can leak or are unable to interoperate.

AI demands coherence through convergence of the data stack, and these are the industry’s expectations from convergence (ranked among the top three):

• Unified access to all data, cited by 84%, is not about convenience, but about eliminating blind spots.

• Strong and consistent governance, demanded by 81%, is what makes trust machine-readable.

• Self-service without IT dependency, at 72%, is what allows action to scale.

These are often mislabeled as “AI features.” They are the minimum conditions required for data activation. Without convergence, AI fails in business environments where the stakes are high, because the system on which it depends was never made whole.

Activation Is the New Benchmark for AI Maturity

For years, AI maturity was measured using technical proxies. Model accuracy, latency, and cost dominated evaluation frameworks, as if intelligence could be inferred from performance characteristics alone. These benchmarks made sense when AI was experimental and when the goal was to prove that models could work at all.

That phase is over. The only benchmark that now matters is activation: can AI reliably trigger action without human hesitation? Especially in high-stakes business environments where the cost of inaccurate or hallucinated data is unacceptable.

We asked practitioners what would improve most with faster, more reliable access to high-quality data. The responses show that speed improves first, cited by 87%, followed closely by confidence at 78%.

Only after this hesitation disappears do business KPIs begin to move. Not because models suddenly become smarter, but because decisions finally become executable.

AI maturity is no longer a question of intelligence. It is a question of decision readiness. When systems are trusted, contextualized, and converged, action becomes the default. Until then, even the most accurate models remain spectators.

From Tool Stacks to Data Systems Built for AI-Native Consumption

AI collapses the distance between data and decisions. What once tolerated delay, interpretation, and human mediation is now compressed into moments where action must be immediate and defensible.

In this compressed environment, fragmentation is no longer a nuisance, but an architectural failure. Stacks built as collections of tools cannot survive when the gap between insight and execution disappears.

Convergence emerges as the only viable response. It is not about consolidation for efficiency, but about continuity. Meaning must flow without translation, trust must persist without negotiation, and access must be unified rather than stitched together.

Only then does data become consumable for both humans (without the overheads) and machines (without the hallucinations and costs of lacking precision).

Access the Complete Report

The Modern Data Report 2026 is a first-principles examination of why AI adoption stalls inside otherwise data-rich enterprises. Grounded in direct signals from practitioners and leaders, it exposes the structural gaps between data availability and decision activation.

With hundreds of datapoints, this report reframes AI readiness away from models and tooling, and toward the conditions required and/or desired for reliable action.

Get your copy

About Research and Organisers

This research is part of a yearly survey series and is a collaboration between The Modern Data Company and Modern Data 101.

About The Modern Data Company

The Modern Data Company is redefining data management for the Al era. The company’s flagship platform, DataOS, serves as the foundational analytics and AI-ready data layer for any data stack. This unified platform gives enterprises the ability to build and deploy data products, simplify data management, and optimize data costs. DataOS frees teams to focus on driving real value from data, accelerating the journey to becoming a truly data-driven and AI-enabled organization.

About Modern Data 101

Modern Data 101 is a publication and community for data leaders, practitioners, and visionaries building data platforms, designing data teams, and architecting the invisible. In a world of countless tools, trends, and templated thought leadership, Modern Data 101 slows things down to ask deeper questions: Why was this actually built? And what business problem can it solve?We explore architecture, semantics, and organisational design, the invisible foundations that determine whether data works.

Originally published on moderndata101.com

(*click images in gallery for full view)

Sourced from the community, refer to the full summary here.

If this piques your curiosity, below is the original essay.

Part 2

The Engineering Problem Nobody Talks About

In my previous essay, I argued that AI adoption resembles the adoption of the wheel. Not in its conceptual difficulty, but in the gap between having the technology and deploying it usefully. The wheel sat around for centuries before the enabling infrastructure made it transformative.

But I glossed over something important: what made the wheel work in the first place.

The answer is the axle. And understanding why the axle was hard helps explain why connecting AI to enterprise operations is hard, and what to do about it.

Why the Axle became the Breakthrough

A 2024 computational mechanics study published in Royal Society Open Science examined the evolution of the wheel-and-axle system. The researchers concluded that it evolved over roughly 500 years, progressing from simple rollers to grooved rollers to wheelsets to independent rotating wheels. Their key finding was striking:

The complex number of factors that had to be overcome meant it could not have been developed in phases. It had to come all at once.

Think about what a functional axle requires. The hole in the wheel and the axle shaft must be nearly perfectly round and smooth. The tolerances must be tight enough to prevent wobble but loose enough to allow rotation. The axle must be strong enough to bear loads but thin enough to minimize friction. Lubrication must be consistent. Materials must withstand repeated stress.

None of these problems is visible when you watch a log roll across the ground. All of them become apparent when you try to build something that actually works.

The Same Problem, Different Materials

AI models are the wheel. They spin impressively. Everyone can see them work. Executives watch demos and understand, at least superficially, what is happening.

But connecting these models to enterprise data and workflows is the axle problem. It requires solving multiple precision challenges simultaneously, just as the ancient wheelwright had to solve roundness, tolerance, strength, and lubrication all at once.

The enterprise axle problem has four tolerance challenges.

Format Tolerance

Your enterprise data exists in dozens of forms, including relational databases, JSON APIs, Excel spreadsheets, PDF documents, legacy mainframe exports, and real-time event streams. AI models expect consistent, well-structured input. The gap between these realities is where most integration projects die.

Quality Tolerance

AI models are sensitive to data quality in ways that traditional tools don’t catch. A customer record missing a middle initial might be fine for a mailing list but catastrophic for an identity verification model. Quality requirements are use-case specific, not generic.

Temporal Tolerance

Enterprise data exists across multiple time horizons. Some updates in real time. Some refresh daily. Some is quarterly. Combining data from different time periods without understanding their temporal relationships creates subtle but serious errors.

Semantic Tolerance

The same word means different things in different systems. “Customer” in your CRM differs from “customer” in your billing system. “Revenue” in sales dashboards uses different definitions than “revenue” in financial statements.

Just as the ancient axle required solving roundness, strength, tolerance, and lubrication simultaneously, the enterprise AI connection requires solving format, quality, temporal, and semantic challenges together.

The Enabling Ecosystem

Of course, even a perfect axle doesn’t create value alone. The wheel-and-axle system became transformative only when the broader ecosystem developed. This included roads for efficient travel, draft animals for pulling loads, social structures for trade, and maintenance craftspeople in every settlement.

Historical sources discuss wheeled transport as an integrated system, not as separate components. Rome’s 400,000 kilometers of roads, the Han Dynasty’s Silk Road trade routes, and the Incas’ 40,000 kilometers of highways all represent civilizational achievements that required multiple enabling factors working together. No single element was sufficient.

(*click images in gallery for full view)

Sustainable Infrastructure

Roads are perhaps the most visible enabler. They represent the prepared environment that makes wheeled transport efficient. For AI, the analogous infrastructure includes data governance frameworks, API architectures, security controls, and monitoring systems. These are not exciting, just as roads are not exciting compared to chariots, but they determine whether the technology creates value.

Operational Power

Draft animals represent sustained operational power. For AI, this maps to compute infrastructure, model serving systems, and the engineering systems that keep everything running. You can have the best wheel and the smoothest road, but without something to pull the cart, nothing moves.

Directed and Informed Investments

Trade networks represent the economic context that justifies investment. For AI, this means clear use cases with measurable returns. Organizations that invest in infrastructure without a line of sight to value creation often abandon the effort before it pays off.

Human Intuition & Skill

Maintenance craftspeople represent ongoing operational capability. The wheelwright was essential in every settlement. Similarly, AI operations require continuous attention through monitoring for drift, updating for new data, and adjusting for changing requirements.

Line of Sight

Societies that successfully adopted wheeled transport shared a critical feature: clear visibility from investment to outcome.

The Sumerian farmer could see that a cart tripled his grain transport. The Roman commander could see that supply wagons enabled sustained campaigns. The connection between infrastructure spending and practical benefit was visible and measurable.

When the connection is abstract, when you’re told that data infrastructure will eventually enable AI value without being shown the specific path, investment stalls. This is rational. Executives have seen too many multi-year transformation programs that delivered reports instead of results.

Enterprise AI investments often lack this line of sight. Budgets get approved for vague promises. Months pass. Consultants produce frameworks. Platforms get deployed. And the connection between spending and business value remains theoretical.

The Axle Problem Compounds This

Even organizations that have clear use cases find that integrating AI with their specific data and workflows takes longer than expected. The tolerances are harder to achieve than they appeared in the demo. By the time the system works, organizational patience has often expired.

Approach to “Precision Engineering”

What ancient wheelwrights were really inventing was not a component, but a way of engineering precision into a system that had to work under uneven loads, changing terrain, and repeated stress.

The axle succeeded because it embedded guarantees about fit, strength, and behaviour under motion.

The enterprise equivalent of the axle is not “data infrastructure” in the abstract. It is the data product. A discipline and an approach instead of hard-baked technology.

• A data product is the smallest unit where line of sight is preserved.

• It is designed backwards from an outcome, not forwards from a source.

• It carries its context with it: what the data represents, how fresh it is, what quality threshold it meets, and what decisions it is safe to support.

This is what makes the axle manifest.

Without data products, the connection between effort and outcome remains indirect.

Crucially, data products collapse what are usually treated as separate concerns.
Context does not live in documentation, provenance does not live in a governance portal, quality rules are not enforced after the fact but are embedded inline with analytics and AI workflows. Because they are all properties of the product itself.

This is the difference between assembling parts and engineering a system.

“Solve three of four, and the wheel still doesn’t turn.”

Most enterprise data engineering still operates in batches.

Sources are onboarded, pipelines are built, schemas are standardised, and reports are refreshed. When requirements change, the system stalls while changes propagate end-to-end.

Manufacturing learned long ago that this mode of operation does not scale in dynamic environments. The breakthrough came with Single-Minute Exchange of Die (SMED): a discipline for reducing changeover time so systems could adapt rapidly without stopping production.

Similarly, enterprise data ecosystems are uneven and constantly changing. Source systems evolve, definitions shift, and new AI use cases appear mid-stream.

Treating every change as a mini-transformation program guarantees delay.

Data products apply the same logic as SMED. They remove unnecessary coupling, make constraints visible, and localise change. Instead of re-engineering the entire system when requirements shift, only the affected product adapts.

The Axle: A Precision-engineered Connection Layer

Not another data platform or another point solution. But a precision-engineered connection layer that solves the tolerance problems simultaneously, because they have to be solved simultaneously.

Approach to Format Tolerance

Format tolerance is enforced through explicit product interfaces. Each data product exposes data in forms suited to its consumers (human or machine0, regardless of how that data is stored internally. Translation happens at the boundary, on demand, without forcing global standardisation that breaks under scale.

Approach to Quality Tolerance

Quality is defined relative to outcome, not as a universal hygiene rule. Each data product declares the quality guarantees it provides and the decisions it supports. A field that is optional in one context may be mandatory in another. Quality is enforced where it matters.

Approach to Temporal Tolerance

Freshness is treated as a first-class contract. Data products specify the time horizon they represent and the validity of combining them with others. When you query data from multiple sources, the system understands when each piece was current and whether combining them is valid for the analysis at hand.

Approach to Semantic Tolerance

Semantic alignment is maintained through contextual metadata. The system knows that “customer” means different things in different sources and translates appropriately. Data products encode domain definitions and translate between them when necessary. “Customer” is not reconciled globally, but interpreted contextually with traceability back to source definitions.

These tolerances are not solved independently. They are properties of a single engineered artifact. The result is that AI applications can connect to enterprise data without the months of custom integration work that typically kills projects.

Breaking the Historical Pattern

Here is where the historical metaphor reaches its limit, and where the opportunity becomes most interesting.

Ancient civilizations were prisoners of their environmental conditions. The Inca built 40,000 kilometers of roads through some of the world’s most challenging terrain, but never used wheeled transport. This was a rational calculation given their mountainous geography and lack of draft animals.

Sub-Saharan societies faced biological barriers: the tsetse fly made draft animals impossible across vast regions. Medieval Europe had the technology but lacked political coordination.

These constraints were fixed. You could not will horses into existence or eliminate endemic disease.

Enterprise AI Adoption Does Not Work This Way

The environmental conditions that block AI deployment, including fragmented data, legacy systems, governance gaps, and skill shortages, are not fixed constraints. They are variables. The question is how quickly you can identify which constraints actually bind and address them specifically.

This is what changes when you have a working axle. The feedback loop compresses dramatically.

Deploy an AI Use Case

See immediately where the tolerances fail, whether that’s where format breaks, where quality issues emerge, where temporal misalignment corrupts results, or where semantic confusion produces errors. Fix the specific gaps. Redeploy. Measure.

Progress is visible at each iteration. You are not waiting for environmental conditions to align. You are actively shaping conditions based on empirical feedback.

Line of sight is restored. Executives don’t take on faith that data infrastructure will eventually enable AI. They see specific use cases moving from concept to production, specific blockers resolved in specific timeframes.

Industrialising the Axle

Once the axle is understood as a data product discipline and tolerances are identified, the remaining challenge is scale. Ancient societies solved this by standardizing tools, techniques, and training so that wheelwrights could reproduce reliable axles across settlements.

Modern enterprises face a similar challenge: how do you enable many engineering teams to build their own precision-engineered connections without each team reinventing the fundamentals?

A data operating system construct acts as a standardized workshop that makes building axles repeatable at enterprise scale. It is not the axle itself, but the means through which precision-engineering is first established, and then reproduced.

Operating systems have been at the heart of enabling masses to access and leverage countless benefits of digital applications without the need to be skilled in infrastructure and dirty plumbing. Even app developers who are into heavy engineering need not bootstrap the whole system to build their applications. Instead, they put together different unique capabilities available on the OS to spin up these applications and deploy at scale.
~ Data Operating System, The Philosphy

DataOS, as the platform implementation, provides the essential building blocks that data engineers need to create, operate, and evolve data products: standardized interfaces for format tolerance, embedded quality contracts, temporal awareness for time-based alignment, and semantic translation capabilities.

Every enterprise needs different axles. A manufacturing company’s supply chain and AI for inventory requires different precision engineering than a bank’s fraud detection system.

But both need to solve the same fundamental tolerance problems. DataOS, by virtue, reduces the cost of that precision engineering to make it possible for domain teams to build outcome-first, context-rich data products without each team having to invent their own approach to format conversion, quality validation, temporal alignment, and semantic translation.

In the analogy, DataOS is the workshop that gives every wheelwright the standardized tools, jigs, and techniques to build reliable axles, while the wheelwright still does the precision engineering work for their specific cart and terrain.

The Leapfrog Opportunity

In the historical pattern, late adopters were permanently disadvantaged. Societies that missed the wheeled transport revolution couldn’t catch up until underlying conditions changed, which often took centuries.

The AI adoption pattern is different.

Organizations that start behind, with messier data, more legacy systems, and more governance gaps, are not necessarily condemned to stay behind. If they can identify their specific blocking factors and address them rapidly, they can leapfrog competitors who are still trying to boil the ocean with multi-year transformation programs.

The advantage goes not to whoever has the cleanest starting position, but to whoever can iterate fastest. And iteration speed depends on having infrastructure that makes the feedback loop tight: deploy, observe, fix, redeploy.

This is the opportunity that most enterprises don’t yet see. They assume they must spend years preparing before AI can deliver value. They budget for eighteen-month transformation programs. They hire consultants to build roadmaps.

Meanwhile, organizations that solve the axle problem first are already iterating. Each iteration makes them better. Each deployment generates data about what works. Each failure teaches them something specific. The gap widens not because of better technology, but because of faster learning.

Results Need Action

The wheel took millennia to spread globally. The institutions that accelerate technology transfer today, including venture capital, research universities, global supply chains, and digital communication, simply did not exist.

AI is spreading in months, not centuries. What took generations now happens in budget cycles.

This compression changes the nature of the challenge.
Organizations do not have the luxury of waiting for conditions to be right. By the time conditions are right, competitors who shaped their own conditions will have moved on.

But the compression also creates opportunity.
The same speed that threatens laggards empowers fast movers. If you can identify what’s blocking you, not in a theoretical multi-year roadmap but in concrete, measurable terms, you can address it now. If you can solve the axle problem, you can start turning the wheel while others are still building roads they may never need.

The historical metaphor is useful for understanding patterns. Enabling infrastructure matters. Adoption is uneven. Resistance is rational. Context shapes deployment.

But history is not destiny.
The Sumerian farmer could see that a cart worked because grain moved faster. He could not, however, conjure roads or draft animals into existence. His adoption was constrained by environmental conditions beyond his control.

We are not similarly constrained.
The environmental conditions that determine today’s AI adoption, including data infrastructure, integration architecture, governance frameworks, and workforce capability, are within our power to change. Not easily or instantly. But measurably, iteratively, with clear line of sight from investment to outcome.

Author Connect 🤝🏻

Discuss shared technologies and ideas by connecting with our Authors or directly dropping us queries on community@moderndata101.com

Connect with Sagar on LinkedIn 🤝🏻

MD101 Support ☎️

If you have any queries about the piece, feel free to connect with the author(s). Or feel free to connect with the MD101 team directly at community@moderndata101.com 🧡

From MD101 team 🧡

🧡 Join 1500+ on The State of Data Products

Introducing The Catalyst, a special edition of The State of Data Products that compiles insights around the ups and downs in the data and AI arena that fast-tracked change, innovation, and impact.

📘 The Catalyst, first release is our annual wrap-up that compiles the most interesting moments of 2025 in terms of developments in AI & data products, with expert insights from across the industry.

Some focal topics

• The gap in AI readiness for enterprises

• The shift towards context engineering

• Addressing governance debt with a focus on lineage gaps

• Revisiting fundamentals for better alignment b/w business vision and AI.

Get your copy!

Heather Hedden | Taxonomy Consultant

Heather Hedden is a globally respected taxonomy consultant and educator whose work has shaped how organisations structure, govern, and retrieve knowledge at scale. With more than two decades of hands-on experience, she brings rare depth across taxonomy design, information architecture, metadata, and ontologies, translating conceptual clarity into practical systems that materially improve search, discoverability, and decision-making. As the author of the seminal book The Accidental Taxonomist, now in its third edition, Heather has educated and influenced generations of practitioners, establishing herself as one of the most authoritative voices in the field.

Heather advises enterprises, platforms, and institutions on building sustainable, standards-aligned taxonomy programs, while also training professionals through courses, workshops, and conferences worldwide. Her contributions extend beyond consulting into stewardship of the discipline itself, including active involvement in ISO standards for thesauri and long-standing editorial leadership with BARTOC. Known for her intellectual rigour, generosity in teaching, and unwavering commitment to quality, Heather continues to elevate the practice of knowledge organisation for the modern information economy. We’re thrilled to feature her unique insights on Modern Data 101!

We actively collaborate with data experts to bring the best resources to a 15,000+ strong community of data leaders and practitioners. If you have something to share, reach out!
🫴🏻 Share your ideas and work: community@moderndata101.com

*Note: Opinions expressed in contributions are not our own and are only curated by us for broader access and discussion. All submissions are vetted for quality & relevance. We keep it information-first and do not support any promotions, paid or otherwise.

Let’s Dive In

There is a growing interest in “semantics” among data experts, such as in developing semantic data models and semantic layers. Information professionals, meanwhile, have always been involved with semantics (whether or not they called it that) in developing classification systems, indexes, taxonomies, tagging systems, and metadata schemas.

Semantics is about meaning, rather than just strings of text. The same text string can have different meanings in different contexts, and the same concept can be represented by different text strings for different users, use cases, and systems.

Thus, control, governance, policies, and standards are needed around semantics, especially when it comes to data and information sharing and interoperability across systems, repositories, and users.

Semantic Web Standards

The ultimate information-sharing platform is the World Wide Web, whose core standards have been developed primarily by the World Wide Web Consortium (W3C) since its founding in 1994.

This has included standards for basic hypertext technologies, including HTML (HTML5 in its final version), CSS, XML, and Internationalized Resource Identifiers (IRI). As web standards became adopted both behind the firewall within enterprises and for web-bases (SaaS), Semantic Web standards became important for the technology and data administration of organizations, and not just the World Wide Web.

The extension of the web technologies to support greater semantics in the hypertext links is known as the Semantic Web, or Web 3.0, which is also relevant to organizations. To this end, the W3C developed additional recommendations, with the foundational standard or data model being RDF (Resource Description Framework).

The W3C defines RDF as a “standard model for data interchange on the Web.” The principles of RDF are that the web of data is based on interlinked statements of subject-predicate-object “triples” and all resources have URIs (or IRIs).

In addition to the web and web applications, there is now a class of graph database management systems based on RDF, which are commonly known as “triple store” or RDF graph databases.

Other standards based on RDF are, in order of publication, from 2004 through 2017: RDFS, GRDDL, SKOS, POWDER, PROV, RIF, SAWSDL, RDB2RDF, OWL, SPARQL, RDFa, JSON-LD, and SHACL.

This may seem like an alphabet soup, but some of these standards are more prevalent and important to understand than others. RDF, RDFa (embedded structured data in HTML), SPARQL (the standard query language for RDF), and JSON-LD (JSON-based serialization for Linked Data, used by developers) are widely used in systems today.

OWL, RDFS, SKOS, and SHACL are especially used by semantic experts, knowledge graph experts, knowledge and data engineers, ontologists, and taxonomists. The other standards have more limited adoption and use.

Leveraging the Combined Power of Standards

Controlled vocabularies modeled in SKOS can be integrated with ontologies modeled in RFDS and described more fully in the OWL language, and all can be expressed in a computer-readable form of JSON-LD (among other serializations) and then queried using SPARQL.

Knowledge Organization Systems

The Semantic Web standard SKOS stands for Simple Knowledge Organization System. The concept of a “knowledge organization system” comes from the field of library and information science.

According to the International Society for Knowledge Organization (ISKO)’s Knowledge Organization Encyclopedia, knowledge organization systems are

…functional items designed for organizing knowledge and information, and making their management and retrieval easier…they are basically made of terms/concepts. | (Source)

Broadly defined, knowledge organization systems include term lists with definitions and/or other data (including dictionaries, glossaries, gazetteers, and terminologies) and structured arrangements of terms or concepts (including taxonomies, subject heading schemes, classification schemes, thesauri, and ontologies).

Although ontologies are considered knowledge organization systems, they are better known as knowledge representation systems. SKOS supports these various kinds of knowledge organization systems with the exception of ontologies, since there are other standards from the W3C for ontologies to support their greater complexity.

SKOS does not support all knowledge organization systems, and we usually refer to what is supported by SKOS as “vocabularies.”

Most software today for creating and managing knowledge organization systems, especially ontologies, thesauri, and taxonomies, is based on semantic web standards.

SKOS (Simple Knowledge Organization System)

SKOS was developed by the W3C starting in 2006 and published in 2009, with the objective of supporting the publishing, sharing, and linking of knowledge organization systems on the Web and thus contributing to the Semantic Web.

SKOS was adopted relatively quickly by both open source and commercial vendors of software for managing thesauri and taxonomies.

Of course, most software is also Web-based (SaaS), so following Web protocols, such as using URIs, is practical even for managing proprietary data.

Although SKOS is often referred to as a “standard,” more precisely, it is a common data model recommended by the W3C for making knowledge organization systems both human-readable and machine-readable, in addition to being interoperable.

An example of machine-readable presentation for the concept with the preferred label of Animals in English, the alternative label of Creatures in English, and the preferred label of Animaux in French would be:

ex:animals rdf:type skos:Concept;
skos:prefLabel “animals”@en;
skos:prefLabel “animaux”@fr;
skos:altLabel “creatures”@en.

The designations before the colons are the namespaces, which for our taxonomy example taxonomy is ex. The namespace of skos is used to prefix all SKOS elements.

An example of expressing a hierarchical relationship between two concepts as a subject-predicate-object statement is:

ex:animals skos:narrower ex:mammals.

SKOS takes a concept-approach to modeling vocabularies, and the concepts have various types of properties, including both types of relationships to other concepts and attribute data, such as various types of labels and notes.

If you are familiar with ontologies, you will recognize the features of classes, properties, and attributes. SKOS actually is an upper-level ontology, which is then used for modeling domain-specific vocabularies. However, if you are not familiar with ontologies, understanding that the SKOS model is an ontology is not necessary, though, to use SKOS.

It’s also not necessary to be an expert on SKOS when creating and managing taxonomies or thesauri, now that nearly all taxonomy/thesaurus management software is based on SKOS, and what is presented in the user interface is more user-friendly, such as “Preferred Label” instead of prefLabel.

SKOS Elements

The following table presents all of the elements of SKOS, grouped by type:

Concept Scheme

The “concept scheme” is the highest level of organization for SKOS vocabularies. A concept scheme is a collection of concepts. Concepts are arranged in hierarchical relationships of broader/narrower to each other. The relationship between the highest level of a taxonomy hierarchy and the concept scheme is instead that of “top concept”.

Collection

There is an additional, optional way to organize concepts, and that is by designating a “collection” to which selected concepts belong based on some other designated criteria that is not simply based on hierarchy. Use of collections allows you to create one master taxonomy, in which subsets of it are used in different implementations.

Concepts & Labels

Concepts are things or ideas that have labels. Each concept has one designated “preferred label” (per natural language, if it is in a multilingual vocabulary) and any number of “alternative labels,” to designate synonyms or variants.

A “hidden label” is an optional kind of alternative label that is designated not for display in certain circumstances or user interfaces. “Notation” is a field designated for use in classification schemes that have a numeric or alphanumeric code corresponding to the concept’s classification.

Notes

Concepts may have various notes, such as a “scope note” or a “change note”, definitions, and even examples. A scope note is a standard field in traditional thesauri, and it is used to indicate any restrictions on the meaning and use of a concept for the vocabulary’s context. Notes are used as needed, and their inclusion is optional. The use of definitions tends to be determined by an organization’s editorial policy.

Semantic Relationships

The relationships available between concepts are (1) hierarchical, involving the inverse pair of “broader” and “narrower,” and (2) associative, involving the reciprocal relationship of “related.”

The related concept relationship is standard in thesauri and optional in taxonomies, although most taxonomies make little, if any, use of “related.” The hierarchical relationships of broader and narrower are inherently transitive through multiple levels of hierarchy, thus supporting the inferencing of hierarchy.

There also exists the inverse pair of “broader transitive” and “narrower transitive,” which in practice are not used directly in the construction of a taxonomy, but rather are derived via reasoning from the direct relationships and thus may be used in statements of SPARQL queries to query the taxonomy.

Mapping Relations

Finally, SKOS includes a set of mapping relations, which did not exist previously in thesauri. These relationships are for use across different concept schemes, or different taxonomies or vocabularies, as they may be called.

Use of SKOS mapping relations allows you to link, map, or connect taxonomies together on a concept-by-concept level. These different taxonomies may be on your local server, or they may be external but linked on the web with URLs.

Controlled Vocabularies

Most knowledge organization systems managed in SKOS are controlled vocabularies. Controlled vocabularies are knowledge organization systems that have a particular focus on managing concepts or terms and their labels and that are used especially for tagging or indexing content to support findability and retrieval.

Users select a concept or category from a controlled vocabulary to retrieve the content that has been tagged with it. Controlled vocabularies exclude certain knowledge organization system types, such as dictionaries or glossaries, where the focus is on defining words.

Controlled vocabularies also exclude classification systems, which focus on mutually exclusive class hierarchies, whose labels are of lesser importance, since often a system of alphanumeric codes is used.

The following are common types of controlled vocabularies, starting with the most semantically rich:

Thesauri

An information thesaurus (in contrast to a type of dictionary of synonyms) is a structured, semantically controlled vocabulary comprising terms and both hierarchical (broader-narrower) and associative (related).

Notes, especially scope notes, tend to be used in thesauri. Another basic feature of thesauri is the use of synonyms or variants to point to the preferred term. Traditional, pre-SKOS thesauri have been based on “terms” rather than concepts and labels, but a thesaurus built on the SKOS model, with concepts and labels, functions just the same for users.

Among controlled vocabularies, thesauri are distinguished by having formal standards that provide recommendations for style, format, and construction of thesauri. These include:

• The international standard ISO 25964. Thesauri and interoperability with other vocabularies

• The American standard ANSI/NISO Z39.19 Guidelines for the Construction, Format, and Management of Monolingual Controlled Vocabularies

Taxonomies

Taxonomies are more informal and varied than thesauri. They are characterized by a hierarchical or category-based organization of terms/concepts. Unlike most thesauri, taxonomies have a limited number of hierarchies to which all concepts belong.

The top-level grouping of taxonomy concepts may be by “facet” or aspect type (e.g., topic, location, organization, person type), which may be implemented in a faceted search/browse user interface. Facets may or may not have further hierarchy within them. When designing a faceted taxonomy, each facet usually corresponds to a SKOS concept scheme.

The name “taxonomy” may suggest classification, but a taxonomy is not a classification system that has mutually exclusive classes. The hierarchy of a taxonomy does not imply a classification or hierarchy of things but rather provides a logical way to organize and find the concepts that are used in tagging and retrieval.

The ISO and ANSI/NISO thesaurus standards are also relevant to taxonomies to the extent that they are applicable. Taxonomies are widely used in business and industry to support information access, both for internal/enterprise and customer/public-facing use.

Name Authorities

Name authorities are lists of named entities or proper nouns, such as names of people, companies, organizations, brand-name products, laws, publications, etc. The lists are not hierarchical, unlike a taxonomy, but alternative labels or variants (synonyms) are typical, and are supported by SKOS.

The designation of “name authority” comes from library science, and the best-known example is the Library of Congress Name Authorities. With SKOS, each named authority would be a concept scheme.

Concepts in a name authority typically have additional descriptive attributes, such as birthdate for people and headquarters location for companies. SKOS does not support the creation of such additional customized attributes, although RDFS does, so combining SKOS with RDFS (for ontologies) fully supports the details of named entities.

Term Lists

Term lists are any controlled lists of terms, typically to pick from values in various metadata properties. Sometimes the designation of “controlled vocabulary” is used synonymously with this simplest kind of vocabulary.

In SKOS, each term list is a Concept Scheme. As with name authorities, there is no hierarchy in simple term lists, so all terms are top concepts of their concept scheme. Term lists also lack alternative labels. Although it may seem as if SKOS is not needed for simple term lists, in practice, term lists often exist alongside other metadata properties, each as a SKOS concept scheme.

For example, controlled vocabularies for tagging and retrieving documents may have facets managed as concept schemes for each of the following: topics (hierarchical taxonomy or thesaurus), named people (authority file), publication language (term list), document type (term list), and source (term list).

Final Note

Controlled vocabularies, such as taxonomies and thesauri, existed before SKOS as a standard was published, but the vocabularies were created and managed in application silos in different applications, and they could not easily be shared, exported, and imported.

With vocabularies based on the SKOS model, they can easily be shared, linked, and reused. This has led to a greater alignment of vocabularies and the information to which they are tagged, with the resulting benefits of improved enterprise-wide information intelligence and knowledge sharing. This, in turn, has further promoted the benefits and uses of controlled vocabularies, which has led to even wider adoption and use.

MD101 Support ☎️

If you have any queries about the piece, feel free to connect with the author(s). Or feel free to connect with the MD101 team directly at community@moderndata101.com 🧡

Author Connect

Connect with Heather Hedden on LinkedIn 💬

Or dive into her work on these platforms:

• Almost 200+ on the Accidental Taxonomist Blog

• The Accidental Taxonomist, 3rd Edition

• Hedden Information Management

From MD101 team 🧡

🧡 The Data Product Playbook

Here’s your own copy of the Actionable Data Product Playbook. With 3500+ downloads so far and quality feedback, we are thrilled with the response to this 6-week guide we’ve built with industry experts and practitioners. Stay tuned to moderndata101.com for more actionable resources from us!

DOWNLOAD!

The PoC Problem: Busy, But Going Nowhere

One of the clearest signs of trouble? The Proof-of-Concept trap. Many AI teams are stuck in a loop of building demos that never scale. The work looks promising on slides, but it rarely sees the light of day in production. It’s not that the tech doesn’t work, but that no one thought beyond the showcase.

Executives greenlight pilots to show movement. Teams build in isolation without a clear path to integration. And before learning from one initiative can compound, the organisation has moved on to the next AI idea. Value stalls, but activity continues, which makes it harder to call out the problem.

Venture capitalists are noticing the pattern too. In recent data infrastructure roundtables and essays, VCs are calling out the fatigue from endless PoCs with no repeatable value. They’re asking harder questions:

A PoC without a path to production is now viewed less as innovation and more as a distraction.

This shift reflects a broader realisation: the AI bottleneck isn’t the model, but the missing foundations of data architecture, the lack of ownership, and the absence of a repeatable delivery approach. Without that, every AI effort becomes a one-off, and the organisation ends up solving the same problems again and again.

Unsurprisingly, many firms’ AI adoption journey starts with technology and data. “AI has been a catalyst for companies to really look at their technology,” said Julie Sweet, Chair and Chief Executive Officer of Accenture, with many companies investing in data too. “The companies that have done this early, like Saudi Aramco – or McDonald's, which created its data foundation very early – are surging ahead in how they're using it.”
~ Part of World Economic Forum Annual Meeting

AI Might Be Still Stuck in the Monolith Phase

The AI Hype symptoms? A trail of expensive prototypes, disillusioned teams, and tech that no one uses. These aren’t edge cases. They’re becoming the industry standard. And once you notice the pattern, it’s everywhere.

The Pattern of Failure: Where AI Efforts Derail

The failures aren’t random. They follow patterns. Once you start looking closely, you’ll notice the same mistakes playing out across different industries, budgets, and use cases.

AI isn’t falling short because the technology is broken. It’s failing because the approach is. These are not edge cases but the mainstream symptoms of how AI is being delivered today.

The "AI Theatre" Phenomenon

Many organisations treat AI as a performance rather than a business capability. A chatbot launched to keep up with peers, a GenAI tool unveiled at a town hall, an AI dashboard built for the board, not the user.

These projects often have high visibility and low utility. They’re not designed to solve a real problem. They’re designed to show that AI is happening.

The bot gets built fast, but poorly. It handles simple FAQs but fails on anything nuanced. Within six months, usage drops to under 5%. Customers bypass it. Support teams work around it. The whole thing fades away and takes along the effort spent.

The problem here isn’t the model. It’s the mindset: tech-first thinking. The organisation led with “We need AI,” instead of asking, “What’s the real problem we’re trying to solve?” The result is an expensive, well-marketed tool with no practical fit.

The "Shiny Object" Syndrome

The pace of AI innovation is relentless and distracting. Every month brings new models, new APIs, and new capabilities. But in trying to chase all of them, organisations end up committing to none.

A manufacturing firm starts with computer vision for defect detection. Three months later, they pivot to generative AI for quality reports. Then it’s predictive maintenance. Each project is staffed, scoped, demoed, and then dropped.

The underlying issue? Lack of a seed approach, focus, and failure to follow through. With no clear success criteria and production pathways, AI becomes a lab experiment loop.

The "Big Bang" Fallacy

This one comes straight from legacy IT thinking: the belief that you can plan your way into a successful AI system. Let’s take the situation of a large healthcare network that invests $5 million in a centralised diagnostic AI platform, planning out the rollout across five hospitals over 18 months.

But by the time the system is ready, the workflows have changed, the data is stale, and the models are no longer clinically relevant.

AI doesn’t work like ERP.

You can’t blueprint it for 18 months and expect it to land cleanly. It needs iteration, real-time feedback, and course correction along the way.

Drowning in Tools, Starving for Standards

In the AI landscape, especially in GenAI space, every few weeks, there’s a new SDK, a “must-have” vector database, an observability tool that claims to solve hallucinations, and a framework that promises orchestration at scale. On paper, this looks like progress. In practice, it’s an invitation to more paralysis.

Most teams are overwhelmed. They don’t know what they’ve adopted, what’s redundant, or what’s adding any real value. Instead of clarity, they’re buried in integration overhead, overlapping capabilities, and fragmented workflows. Internal documentation can’t keep up, technical debt piles up silently, and no one owns the sprawl. Engineers are busy wiring things together instead of solving business problems.

This isn’t just a tooling problem; it’s a symptom of a deeper one: the lack of a focused, value-driven approach to AI adoption.

Cultural Siloes and Low Internal Readiness

Boston Consulting Group reports 74% of organisations still struggle to derive value from AI, with only 26% showing clear, ongoing ROI (Source). Similarly, BCG’s findings highlight that scaling AI without accounting for data management, governance, and cultural alignment leads to widespread cost overruns and delays (Source). Tech investment without AI-readiness is an investment lost.

🔖 Related Read

The Changing Landscape of AI Challenges

The current challenges seen in terms of AI adoption were not seen as often before. As technologies change, so do the challenges associated with them.

But why do we see this happen? Let’s dive into the whys of these challenges.

Why are Leaders Getting AI Delivery Wrong? The Loop of Leadership Missteps

Most leadership teams aren’t blindly chasing AI hype but are actively trying to make sense of a fast-moving landscape while balancing pressure from markets, boards, and internal stakeholders.

The intention is rarely the problem. But even with the best intentions, familiar patterns from past tech transformations often get applied to AI without considering what makes it fundamentally different.

These are not failures of intelligence or effort. They’re the result of reflexes shaped over decades of navigating digital transformation. The challenge is that many of those reflexes, like planning for stability, prioritising ROI visibility, or scaling before validating, don’t translate cleanly into AI. And so, without realising it, even the smartest teams fall into familiar loops.

FOMO and Competitive Mimicry

The AI race is more psychological than technical. Boardrooms feel the pressure to act, not because the business is ready, but because everyone else seems to be doing something. This competitive mimicry drives decisions that are reactive, not strategic. It’s not about whether the AI solves a real business problem. It’s about being able to say “we’re doing AI too.”

That urgency overrides the rigour. Leaders announce initiatives before the use case is clear, pilot solutions without defining the problem, and chase market optics instead of operational fit.

The “Magic Box” Expectation

There’s still a persistent belief that AI is a black box that can absorb messy problems and spit out optimised answers. This mindset deprioritises the operational scaffolding AI actually depends on: precise problem statements, clean and well-governed data flows, integration into real business workflows, and mechanisms for feedback and iteration. AI isn’t plug-and-play. It’s a system that needs to be engineered, embedded, and continuously refined.

Consultant-Driven AI Transformation

External consultants often show up with multi-year AI roadmaps, enterprise-wide blueprints, and transformation frameworks that promise scale before substance. These strategies tend to optimise for scope, not value. They emphasise deliverables over discoveries, and static plans over adaptive learning.

The result? Impressive decks, vague metrics, and a portfolio of projects no one’s really accountable for. Organisations get stuck managing the roadmap instead of building momentum around what’s working.

Media Bias Toward Success Stories

Most AI stories that make headlines are polished successes, not the messy failures that came before. This creates a survivorship bias in how AI is perceived. Leaders come across instances of AI transforming marketing, operations, and customer service, but not about the countless attempts that didn’t make it past the pilot phase.

This selective storytelling skews expectations. It makes failure feel like an outlier when, in reality, it’s often the default outcome of poorly scoped or poorly integrated efforts.

Obsessing Over Tech Without Understanding the Problem

In many engineering-led organisations, there’s a bias toward building sophisticated solutions, even when simpler fixes would work. Teams gravitate toward model development, parameter tuning, and infrastructure design without first validating whether the problem warrants AI at all.

This approach leads to over-engineering and under-delivering. It disconnects AI efforts from the users, use cases, and outcomes that matter. What gets optimised is the model, not the business impact.

🔖 Related Read

Why Even Good AI Gets Undone by the Wrong Partners

The Vendor Ecosystem Problem: Everyone’s Selling AI

Few Are Solving for It.

Do you think internal missteps are always hindering the AI efforts? Not really. More often than not, the surrounding ecosystem is adding fuel to the fire.

Platforms promise ready-to-use intelligence, consultants pitch AI-in-a-box roadmaps, and vendors offer generalised solutions to highly specific problems. The result is a market filled with AI that looks complete on paper but falls apart in practice.

Let’s think of the IBM Watson for Oncology case.
IBM’s $62M Watson for Oncology aimed to revolutionise cancer care, but couldn’t move beyond one institution’s data. Why? No real-world feedback loops. No adaptability. No integration into clinical workflows. A static solution in a dynamic world.

A classic case of “build it, train it, ship it,” with no lean mechanism for validating or evolving the product in motion.

The Cost of Broken AI We No Longer Can Afford

The damage of subpar or failed AI initiatives runs deeper than mere budget overrun or missed KPIs. On the whole, they start depleting the culture, credibility, and the organisation's ability to try again.

These are not isolated project risks but systemic outcomes of treating AI as a tech layer rather than a strategic capability. And over time, they destroy the foundation needed to deliver anything meaningful with AI.

So, what does broken AI impact?

Opportunity Cost

Every resource tied up in a poorly scoped project could have gone into something else, a proven analytics enhancement, a core process improvement, or even a simpler automation that would’ve moved the needle. Time and talent are finite. Burning them on initiatives that have no real path to value doesn’t just slow down AI, but also slows down the business.

The most dangerous part is that these opportunity costs are rarely tracked. On paper, the project “explored a use case.” In reality, it displaced efforts that could’ve delivered impact.

Organisational Fatigue

When teams spend months building something that never launches, or gets sunsetted after one demo, the result is a slow, widespread disengagement. Data scientists feel like model developers for PowerPoint. Product teams stop prioritising AI integration because “nothing ever ships.” Business stakeholders grow sceptical and default to manual workarounds.

Data Quality Degradation

AI surfaces data issues fast, but without a solid foundation, organisations often work around these problems instead of fixing them. Teams patch missing values, retrain on biased data, or overfit models to noise. This doesn’t just hurt one project, but creates a technical debt spiral where data issues compound over time.

Worse, these shortcuts rarely get documented. Future teams inherit flawed pipelines without context, making future models less reliable and harder to audit. Data that looks usable but can’t be trusted when it matters most.

Talent Exodus

Good AI practitioners know the difference between experimentation and futility. When they’re asked to ship models without context, build solutions with vague objectives, or defend performance metrics no one understands, they eventually walk away. No one wants to be the engineer behind a system that’s never used.

And once word spreads that the organisation treats AI like a vanity project, it becomes harder to hire and retain the right talent. Smart people don’t stick around to build things that never go live.

Customer Trust Erosion

When AI touches customer-facing experiences, the stakes are even higher. A chatbot that can't solve real problems, a recommendation engine that gets it wrong, a predictive alert that triggers too late: all of these chip away at user trust. In some industries, that erosion isn’t just reputational, it’s regulatory.

Once customers learn to work around AI, they stop trusting anything that looks like it. That’s a complex perception to undo.

Can Older Frameworks Address Today’s AI Challenges

Actually no! As we navigated the problems and their changing nature, the deduction is that there’s a need of new approach to stabilise AI adoptions.

Going All In With AI: Not the Best Call Now

What the AI world needs now is the equivalent of Agile and DevOps: a mindset built for adaptive delivery. A ‘Lean’ approach.

Think of how the evolution of software development didn’t always look the way it does today. We forget how painful it once was: rigid roadmaps, late-stage testing, siloed teams, and massive releases that broke more than they built.

It took years and a lot of broken systems to move from monoliths to microservices, from waterfall to agile.

We need the same kind of shift that rewired the software world, one that recognises AI as a capability to build in smaller loops, focus on real-world feedback early, prioritise user value, and not just technical success. To treat software not as a project, but as a product.

A Marketing Instance

We don’t need bigger AI, we need smarter delivery and more impact. For instance, not every CMO would greenlight a $ 300K ad campaign without A/B testing messaging, creative, and audience targeting. You don’t flood every channel on Day 1, but test a few. A Facebook ad variant here, a landing page tweak there. You measure and iterate. You scale what converts.

So why should we treat AI any differently?

But here’s what often happens instead:

Companies load up on AI infrastructure, onboard expensive talent, and kick off grand initiatives, without first validating whether the business problem is even worth solving with AI. That’s like launching a global rebrand without customer insight. It burns resources fast and quietly underdelivers.

Let’s borrow from the smarter playbook.

Start with the Customer Problem, Not the Cool Tool

In GTMs, you don’t pick a channel because it’s trendy. You start with the goal: boost lead quality, improve retention, reduce churn, and then find the tactic that fits. AI should work the same way.

Before spinning up a model, zero in on the actual friction: Are sales cycles too long? Is churn high in a specific segment? Frame the challenge in human terms. Then evaluate whether AI is even the right lever. Sometimes, it isn’t.

Test Quickly, Not Perfectly

Marketing teams don’t build a campaign engine in one go. They run sprints. Test email headlines. Launch a microsite. The goal: learn before scaling.

AI teams should adopt the same tempo. Set time-boxed experiments with measurable impact goals. Think: “Can this model reduce ticket resolution time by 20%?” and not “Can we get it to 94% F1 score?”

Track Business Value, Not Just Model Metrics

Just as a CTR doesn’t tell you customer lifetime value, a precision score won’t show how AI impacts revenue. What matters is business lift: higher NPS, faster onboarding, lower acquisition cost.

So swap your dashboards. Replace confusion matrices with outcomes. AI that boosts conversion by 10% is far more valuable than a model that’s 98% accurate on a synthetic dataset.

Learn Fast, Then Let Go

Good marketers kill bad campaigns. Fast. If something isn’t converting, it doesn’t matter how pretty it looks. That same muscle needs to exist in AI efforts.

If a pilot doesn’t show traction in real conditions, don’t waste cycles trying to force it. Let it go. The win isn’t in saving the work, it’s in saving the business from waste.

Invest Where the Signal is Strong

Once a marketing test proves repeatable impact, say, a campaign consistently boosts demos by 25%, you don’t hesitate. You scale. More spend, more channels, more support. It’s not a gamble.

Treat AI the same. When a solution drives results in one region or one workflow, double down selectively.

Thinking Lean

LeanAI isn’t a tech stack. It’s not a new framework to evaluate vendors.
And it’s definitely not about hiring more ML engineers. It’s a way of thinking. It doesn’t downplay the power of advanced models, but redirects focus toward continuous delivery, feedback-oriented learning, and outcome-driven iteration. It forces teams to ask uncomfortable but essential questions early:

What signal are we optimising for?

What behaviour needs to change?

What does success look like in the hands of a user, not just in a benchmark?

LeanAI reframes AI work as a flow, not a feature. A repeatable capability over a one-time win. It borrows from the principles that made DevOps and Agile work in the software world: ship smaller, learn faster, validate earlier, and adapt continuously.

Only now, the stakes are higher, and the systems are probabilistic.

MD101 Support ☎️

If you have any queries about the piece, feel free to connect with the author(s). Or feel free to connect with the MD101 team directly at community@moderndata101.com 🧡

Author Connect

Find me on LinkedIn 💬

Find me on LinkedIn 💬

From MD101 team 🧡

🧡 Join 1500+ on The State of Data Products

Introducing The Catalyst, a special edition of The State of Data Products that compiles insights around the ups and downs in the data and AI arena that fast-tracked change, innovation, and impact.

📘 The Catalyst, first release is our annual wrap-up that compiles the most interesting moments of 2025 in terms of developments in AI & data products, with expert insights from across the industry.

Some focal topics

• The gap in AI readiness for enterprises

• The shift towards context engineering

• Addressing governance debt with a focus on lineage gaps

• Revisiting fundamentals for better alignment b/w business vision and AI.

Get your copy!

Winfried Etzel | Data Governance & Data Management Specialist & Podcast host at MetaDAMA

Winfried is a senior data governance and data management practitioner, educator, and community builder with deep experience across energy, consulting, and public-sector organizations. He currently serves as Senior Specialist in Data Management at Equinor. Formerly, he served as Manager of Data Governance & Architecture at Aker BP and did extensive consulting work with Bouvet ASA.

As a prominent contributor to the data community, he is the host of the MetaDAMA podcast and has been recognised as one of the Nordic 100 in Data, Analytics & AI and nominated multiple times as Data & AI Influencer of the Year.

Winfried’s work is grounded in a strongly socio-technical view of data governance. Especially as organizations embed automation, AI, and increasingly autonomous systems into their core operations. He complements this foundation with deep professional certification, including CDMP Master, Microsoft Azure AI Fundamentals, and Generative AI credentials. We’re thrilled to feature his unique insights on Modern Data 101!

We actively collaborate with data experts to bring the best resources to a 15,000+ strong community of data leaders and practitioners. If you have something to share, reach out!
🫴🏻 Share your ideas and work: community@moderndata101.com

*Note: Opinions expressed in contributions are not our own and are only curated by us for broader access and discussion. All submissions are vetted for quality & relevance. We keep it information-first and do not support any promotions, paid or otherwise.

Let’s dive in

Rethinking Data Governance in Times of AI

Data governance is often experienced as a burden: slow, bureaucratic, and disconnected from how organizations actually work. Yet, at the same time, it is increasingly recognized as indispensable, especially as organizations embed automation, AI, and autonomous agents into their core operations.

This tension stems from a fundamental mismatch between how data governance is conceptualized and the reality of modern socio-technical systems.

Traditional Old School data governance is rooted in an essence-based worldview:

…the assumption that data has inherent meaning and value that can be defined, standardized, and controlled. A worldview that was rational in centralized/aggregated, slow-moving systems.

In distributed, AI-enabled environments, this assumption no longer holds.

Yes, we still need to respect regulatory realities, where data does carry non-negotiable obligations, where safety-critical data does have inherent risk implications, or where some meaning is constrained and not freely enacted. But a realization that is slowly unfolding is that context and purpose are limiting these non-negotiables.

Meaning is not stable, but is enacted with intentionality. Yet, New School data governance has not solved the inherent issues of data governance either.

With a focus on automation and design patterns, we finally found the right methodologies to scale data governance efforts. Yet we are still not doing “the right thing the right way,” because…

…scale without intentionality automates existing assumptions.

This is not because we got it “wrong”, but because data governance has evolved, from compliance-focus to value-focus. Now it needs to realize contextual obsolescence and move towards becoming a motor for AI advancement.

Drawing on the socio-technical foundations of data governance, the challenges introduced by autonomous AI systems, the existential reframing of value and meaning, and the operational shift toward embedded and computational data governance, we are required to propose a different lens:

Data governance is not about controlling data or AI.

It is about intentionally structuring socio-technical systems so that meaning, direction, and accountability can be sustained over time. Don’t think of this as a philosophical indulgence. This existential reframing is a practical necessity for governing AI at scale.

Data Governance as a Socio-Technical Foundation

At its core, data governance is not a technical discipline, but largely a social system.

Data governance exists to ensure that collective action remains purposeful, accountable, and legitimate within complex socio-technical systems.

Its role is to help organizations

• negotiate requirements and meaning,

• set direction for data, and

• maintain accountability in relation to data and data-driven decisions.

Simply put, a socio-technical system is any system where humans and machines work together to achieve goals. And this kind of system, with all its interactions and connectedness between humans and machines, has become the reality in most organizations.

These are not different kinds of governance, they are different contexts in which the same governance logic must operate. The purpose remains constant while only the methods evolve.

This framing is critical, because it exposes the core problem facing data governance today: it has often been treated as an operational or compliance function, rather than as a foundational element of how organizations coordinate action and responsibility.

The Limits of Essence-Based Data Governance

Both Old School and New School data governance are built on an implicit belief: that data has an essence. This belief manifests in familiar practices:

• defining canonical data models,

• enforcing enterprise-wide definitions,

• measuring quality against fixed rules,

• assuming alignment once standards are met.

Such practices aim to stabilize meaning by fixing it in advance. They presume that if data is correct, meaning will follow, and that interpretation can therefore be ignored. While there is value in these practices, they treat data and standards for data as constant, while in reality, they are temporary agreements, not eternal truths.

This approach is still viable in centralized, slow-moving systems, bounded context, or facing regulatory deterministic rules. But it breaks down in distributed data landscapes, where data is created, interpreted, and reused across domains with different goals, constraints, and assumptions.

Here, meaning must be negotiated constantly. This brings us to the introduction of a shift in the way we think: from essence to existence. Data cannot reliably carry meaning within itself. Meaning arises only through use, context, and purpose.

Once this is accepted, the role of data governance changes fundamentally.

AI as an Accelerator of Data Governance Failure

Generative AI systems are probabilistic, adaptive, and increasingly autonomous. They consume and produce data like other systems, and the quality of their outputs can have immediate and profound consequences for people, organizations, and society.

Crucially, AI systems act without understanding intent or taking accountability for interpretation. They operate on statistical proxies of meaning and optimize patterns.

Data governance fails when we project understanding onto systems that operate on unknown or misaligned interpretations of intention. Hallucinations, bias, drift, and opacity are the most well-known failure modes of AI.

But they are not merely technical defects. They are symptoms of data governance that failed to:

• negotiate context,

• provide clear direction,

• or maintain accountability as systems evolved.

AI compresses decision cycles and shifts action away from human presence. This renders data governance models that rely on retrospective control or human-in-the-loop approvals structurally inadequate.

What this reframing makes explicit is that the challenge of AI governance is an acceleration of existing challenges. The same socio-technical questions data governance has always faced now surface faster, at greater scale, and with less room for retrospective correction.

Embedded, computational data governance is the only way the existing ambition survives.

From Control to Accountability

A common reaction to AI-driven incidents is to demand more control: additional approvals, stricter rules, heavier oversight. But control does not create meaning and rather constrains behavior.

The deeper issue is accountability. Existential thinking reminds us that accountability cannot be delegated away. Organizations remain accountable for the systems they design, deploy, and scale. Even when those systems act autonomously.

Data governance exists to make that accountability explicit and actionable. This is why there is a distinction between human-in-the-loop and human-designed feedback loops. The former attempts to preserve control by inserting humans into execution. The latter:

The Stable Core: Data Negotiation, Data Direction, Data Audit

Despite technological change, the core roles of data governance remain remarkably stable. These core roles that data governance has to play in an organization are a defining part of what data governance is.

The definition of data governance to tackle what is to come is different from what you may have encountered before. It is aligned with corporate governance, yet defined for data:

Data governance is a human-based system by which data assets in a socio-technical system are directed, overseen, and by which the organization is held accountable for achieving its defined purpose.

Based on this definition, data governance consistently fulfills three functions:

1. Data Negotiation
   Governance aligns business objectives, regulatory constraints, ethical considerations, and contextual realities. In AI systems, negotiation governs meaning: what data represents, how outputs should be interpreted, and where boundaries lie.

2. Data Direction
   Governance provides strategic direction. It answers not only whether systems perform correctly, but whether they should exist at all. Data Direction ensures that automation serves organizational purpose rather than undermining it.

3. Data Audit
   Data governance maintains accountability. In autonomous systems, audit shifts from periodic review to continuous assurance, real-time signals, observability, and feedback loops that allow responsibility to be exercised before harm occurs.

These roles are existential stabilizers in complex socio-technical systems. They operate on different levels, from strategy to tactics to assurance. This is not about thinking of data governance as a way to monitor and enforce, but as a discipline that designs for accountability.

Governance Is One

A key implication of this framing is that governance cannot be fragmented in purpose, even when execution is context-specific. Corporate governance, IT governance, data governance, and AI governance share the same purpose: directing collective action and ensuring accountability.

Treating them as separate disciplines creates gaps precisely where AI systems operate across boundaries. Data governance must be understood as part of organizational governance, not as a technical add-on, while remaining context-sensitive in execution. AI makes this integration unavoidable. This is the very reason why the wording of the data governance definition above is aligned with the syntax of the ISO definition of corporate governance (ISO 37000:2021).

From Human-Led to Human-Directed Governance

Old School data governance depends on human compliance, documentation, and incentives. This approach does not scale once decisions are automated and systems begin to act autonomously.

New School data governance approaches improve speed and decentralization, but still rely on implicit assumptions about meaning and accountability that break down once decisions are executed autonomously by systems.

Embedded data governance responds by making data governance computable. Enforcement, monitoring, and escalation are delegated to systems. Not as a replacement for human judgment, but as an extension of it.

Intent, constraints, and accountability are encoded in contracts, policies, and control mechanisms that platforms and AI agents can execute consistently at scale. This is the automation of existing data governance patterns, redesigned for environments where behavior emerges at runtime rather than being fully predefined.

As systems become agentic, data governance must observe and steer behavior over time. Humans negotiate meaning, set direction, and remain accountable for outcomes. Machines enforce constraints, monitor behavior, trigger escalation, and provide feedback loops through which data governance can learn and adapt.

MD101 Support ☎️

If you have any queries about the piece, feel free to connect with the author(s). Or feel free to connect with the MD101 team directly at community@moderndata101.com 🧡

Author Connect

Data Governance in the Wild

Distributed data landscapes, the advancement of AI, new requirements, they all require a shift in data governance. But not by loosing the core of data governance out of sight: negotiating data requirements, setting directing, and assuring accountability.

By Winfried Adalbert Etzel

MD101 Support ☎️

If you have any queries about the piece, feel free to connect with the author(s). Or connect with the MD101 team directly at community@moderndata101.com 🧡

From MD101 team 🧡

🧡 Join 1500+ Data Leaders Subscribed to The State of Data Products

Introducing The Catalyst, a special edition of The State of Data Products that compiles insights around the ups and downs in the data and AI arena that fast-tracked change, innovation, and impact.

📘 The Catalyst, first release is our annual wrap-up that compiles the most interesting moments of 2025 in terms of developments in AI & data products, with expert insights from across the industry!

Some focal topics

• The gap in AI readiness for enterprises

• The shift towards context engineering

• Addressing governance debt with a focus on lineage gaps

• Revisiting fundamentals for better alignment b/w business vision and AI.

Get your copy!

For most data leaders, the idea of a data platform arrives pre-packaged. It is spoken of as a stack: tools layered neatly on top of each other, or as a purchase decision, something to be evaluated, budgeted, and procured.

These assumptions feel natural because they mirror how enterprise software has been sold for decades. Yet familiarity is precisely what makes them dangerous. When assumptions go unquestioned, they define the limits of what we believe is possible.

A first-principles approach asks us to suspend inherited beliefs and interrogate them at their roots. Is a platform truly a collection of tools, or is that how it is delivered? Is ownership achieved at the moment of purchase, or only when an organisation can shape, extend, and evolve what it has acquired?

The moment we strip away the language of vendors and stacks, a simpler question emerges: what capability does a platform fundamentally exist to create?

The IKEA Moment

On a recent visit to IKEA, somewhere between all the walking and wandering, it clicked: the subtle but powerful analogies between data platforms and IKEA’s remarkably well-designed model, which we’ll get into soon.

IKEA is not in the business of selling furniture. It is in the business of selling optionality. The freedom to assemble, disassemble, customise, outsource effort, change your mind, and scale your ambitions over time. The furniture is merely the visible artefact. The real product is the system that makes all the choices possible.

Seen this way, a platform is the ability to construct, reconstruct, and extend without starting from scratch each time. It is the difference between using something and building with it. The former optimises for consumption, the latter for agency. And agency is what separates tools from platforms.

A Platform Exists to Build Outcomes

What you are offered are Legos, possibilities, and the freedom to assemble them into something that fits your space, not an abstract showroom ideal. The value is in the fact that you can combine them into an outcome that did not exist before you arrived.

Data Developer Platforms, often referred to as DDP, operate on the same principle.

A Data Developer Platform is a Unified Infrastructure Specification to abstract complex and distributed subsystems and offer a consistent outcome-first experience to non-expert end users… (learn more at source)

In analogy to IDP, a DDP is designed to provide data professionals with a set of building blocks that they can use to build data products, services, and data applications more quickly and effectively.

DDP’s existence is credited to teams that want to:

• assemble pipelines,

• assemble models,

• assemble contracts,

• assemble semantics, and

• Choose when to abstract and when to go low-level

THEMSELVES. Not by filling out tickets or waiting for central teams, but by composing building blocks directly. Consumer-friendly construction. And God knows the countless assets business wants to construct on a daily basis when it comes to data.

This is where tools and platforms diverge. Tools optimise for use and aim to make a specific task easier, faster, and more repeatable.

The highest leverage comes from enabling builders to reconfigure the system as their understanding evolves.

To truly construct, you need

• standardised parts that fit together reliably,

• clear interfaces that define how those parts interact, and

• predictable behaviour when components are composed.

Modularity is Power

Complex systems do not scale by becoming smarter or more sophisticated. They scale by the Descartes’ method: being broken down into units that can stand on their own and still work together. Independence is what allows change to propagate without collapse.

Yet freedom requires strong discipline. When orientation is agreed upon, movement becomes faster. The rule is not there to restrict choice, but to eliminate ambiguity where ambiguity adds no value.

This is why Lego works. Infinite creativity is possible precisely because everyone agrees on the studs, the dimensions, and the orientation. No one debates how pieces connect.

Data Developer Platforms rely on the same foundation. Data Operating System, a working implementation of Data Developer Platforms, implements the idea of modularity through composable building blocks or resources:

To own a platform is to accept that constraints must be learned before leverage is earned. Teams must internalise conventions because those conventions are what make independent work compatible at scale.

This is the uncomfortable part of platform ownership: the realisation that freedom is not the absence of rules, but the presence of the right ones.

Ownership is a Gradient

Ownership is often spoken about as a binary state. You either own the platform, or you don’t. You either build everything yourself or you rely on others. This framing is seductive and wrong.

This board by IKEA is not an apology for their moving parts. It’s an acknowledgement of reality. Some organisations enjoy the act of building structures as their core skill, while others value the outcome more than the process.

A system that respects both is not compromising, but simply expanding reach and wider usability.

A mature Data Developer Platform embodies this same humility.

• It supports full DIY for teams that want maximum control and learning.

• It offers partial abstraction for those who want speed without surrendering agency.

• And it provides full-service assistance when reliability, time, or scale demands it.

Platform maturity is reached when the platform has the ability to meet teams where they are without fracturing the underlying system.

When you decompose an organisation, you inevitably find different personas.

• Builders who want to touch every layer.

• Accelerators who want leverage without friction.

• Delegators who want internals outcomes.

The goal of a platform is not to make everyone a builder, but to make the organisation move forward together.

Platforms Optimise for Adaptability Over Perfection

Consider some of the less aesthetic creatures of the world. While we have evolved to think most of them admirable, some still seem a little off course to human senses. But they exist because they survived time, with features that made sense for the benefit of their surrounding ecosystem.

Change is not an edge case to be handled later. It is the default state of any working system. The mistake many organisations make is designing for correctness under static assumptions, only to discover that correctness decays the moment reality moves.

IKEA captures this with disarming honesty: “Because sometimes you change your mind.” This is not a fail-safe mode, but an expectation. The system is designed to efficiently absorb reversals like returns, reconfiguration, and second thoughts without penalty. The cost of being wrong is intentionally kept low.

Data Developer Platforms operate as a defensive system, which means defensible against such reversals. Data products evolve as business meaning shifts. While AI models routinely invalidate yesterday’s carefully curated features and labels. In such an environment, the most dangerous platforms are the ones that make change expensive.

📝. Related Read

This requires a fundamental reframing of governance.

• Governance ≠ rigidity

• Governance = safe change

Governance does not imply locking things down in the name of control. Governance is the machinery of safe change. It provides guardrails that allow teams to move quickly without tearing the system apart. Cheap reversals → Frequent Experiments/Trials → Progress/Higher Clarity

Scale Requires Services Around Core

No matter how well-designed the core is, scale does not emerge from primitives or building blocks alone. It emerges from the ecosystem that forms around them. This is the uncomfortable truth many platform efforts discover too late: building the core is necessary, but insufficient.

The IKEA information and service point makes way for measurement services, planning help, personal shoppers, delivery, installation… while none of these change the furniture itself. Yet without them, the system would be incomplete and in fact extremely user-averse.

These services reduce uncertainty, compress decision-making, and prevent small obstacles from becoming reasons to stop. Or simply just having the option, even if not taken, brings a sense of peace to users.

Data Developer Platforms require the same surrounding services to function at scale. Ergo, on top of core primitives, DDP enables interfaces: existing system of capabilities to enable quick construction of outcomes instead of mulling over logistics.

• Metadata to explain what exists and why.

• Observability to understand how systems behave in the real world.

• Quality checks to prevent silent decay.

• Cost visibility to keep ambition grounded in reality.

• AI readiness signals to indicate what can safely be automated and what cannot.

• And critically, human support structures that provide context when automation reaches its limits.

Alongside the simpler forms like CLI, GUI, and APIs, as a working implementation of DDPs, Data Operating System unfurls these interfaces on top of its resources:

Platforms shouldn’t attempt to remove humans from the loop. They must be built in a way that human judgment cannot be eliminated, only amplified.

Constraints Enable Business Ambition

Ideas in this world are truly abundant, and business initiatives rarely fail because of a lack of them. They fail because the infrastructure required to carry those ideas exceeds capacity.

Big ideas have significant mass. They need logistics, coordination, and systems that can move them from imagination into reality without collapsing under their own weight.

Business vision often outgrows resource constraints. When teams move:

• from projects to data products,

• from analytics to AI systems,

• from dashboards to chat interfaces,

the nature of the work changes. What once fit inside a single notebook or workflow now demands logistics. This is where platforms earn their relevance.

By absorbing operational burden, enforcing consistency, abstracting through usable interfaces, and providing dependable execution paths, platforms bring velocity to data initiatives. They allow teams to aim higher without paying an exponential tax for reversals, failures, or experiments.

Be it Data Products, Generative AI, Context Graphs, or other shapes and ambitions, a composable data platform surrounded by scalable interfaces offers the stamina to stretch across that last mile without hurting the ones who dared to dream.

Economic First Principle: Pay What You Reap

One of the most misunderstood aspects of platforms is their economics. We are conditioned to think in terms of inventory, like what we buy upfront, what we deploy all at once, and what we must fully commit to before value can be realised.

But that doesn’t work for data platforms that are designed for organisational alignment. Imagine the volume of inventory, including and definitely not limited to resources, interfaces, implementation templates, data products, vertical cross-layer graphs, and so on…

Clearly not the way to go. Pricing shouldn’t resist or obstruct action or curiosity (e.g., experiments), given how a data developer platform is especially a builder’s sanctuary.

The platform must acknowledge that confidence grows with use, that clarity emerges through experience, and that commitment should follow value instead of preceding it.

Data Developer Platforms follow the same economic logic. They allow incremental adoption rather than forced migration. Ownership becomes progressive, deepening as teams internalise the system and expand its use. Value is realised over time, through compounding capability rather than a single, risky bet.

By doing this, platforms reduce commitment anxiety. They make it rational to begin without perfect certainty. And in complex environments, where change is constant and understanding evolves, that may be the most valuable capability of all.

Curtain Call

A Data Developer Platform is not a data stack you deploy.
It is a system that lets you decide how much to build, how much to delegate, how fast to change, and how far to scale, without rewriting the foundation each time.

IKEA taught millions to think in systems. While it might seem easy (as it was meant to be seen), it is an incredible feat. A low-key way of education in systems, modularity, trade-offs, and choice. Over time, millions of people learned how to reason about space, components, constraints, and assembly without ever being taught those words.

This is the shift data leaders now face. The question is no longer which tool to buy or which stack to standardise on. That belongs to an era where problems were smaller, and change was slower. Today, new technology is replacing the old every six months or less. The approach to thought must change too.

The more consequential question is this: what capabilities must the organisation own? Capabilities to build, to adapt, to reverse, to scale, and to decide. Tools may come and go. And platforms endure only when they reshape how an organisation thinks.

*All pictures, unless mentioned otherwise, are taken by the author(s).

Author Connect 🤝🏻

Discuss shared technologies and ideas by connecting with our author(s) or directly dropping us queries and notes on community@moderndata101.com

Modern Data 101

Conversations around Modern Data challenges, solutions, and innovations.

By Animesh Kumar

MD101 Support ☎️

If you have any queries about the piece, feel free to connect with the author(s). Or connect with the MD101 team directly at community@moderndata101.com 🧡

From MD101 team 🧡

🧡 The Catalyst: State of Data Products

Introducing The Catalyst, a special edition of The State of Data Products that compiles insights around the ups and downs in the data and AI arena that fast-tracked change, innovation, and impact.

📘 The Catalyst, first release is our annual wrap-up that compiles the most interesting moments of 2025 in terms of developments in AI & data products, with expert insights from across the industry!

Some focal topics

• The gap in AI readiness for enterprises

• The shift towards context engineering

• Addressing governance debt with a focus on lineage gaps

• Revisiting fundamentals for better alignment b/w business vision and AI.

Get your copy!

AI changes that contract. AI systems don’t just analyse data, but are required to act on it. They recommend, approve, escalate, and commit changes to the world. And the moment a system acts, analysis is no longer enough.

Action demands judgment. Jaya Gupta and Ashu Garg brilliantly summarise this premise in their groundbreaking piece on context graphs, which has, by itself, started a domino effect of clearer understanding in terms of true AI requirements.

Judgment is not a function of more rows or cleaner schemas. It emerges from context: what was known at the time, which rules applied, where exceptions were allowed, and which precedents mattered in this situation.

Without access to decision-time context, intelligence stalls at insight and never translates into autonomy.

Where existing data systems are breaking bad

The current data stack was never designed to carry judgment. Systems of record preserve state (what the world looks like now), but they overwrite the reasoning that led there.

Warehouses preserve history (how metrics changed over time), but not the intent or tradeoffs behind those changes. Knowledge graphs model relationships (how entities connect), but not how conflicts were resolved or why one path was chosen over another.

The result is a familiar gap: we can see what happened, but not why it was allowed to happen. The reasoning that turned data into action dissolves into chats, calls, and memory.

What is a Context Graph?

The anatomy of a decision

Every meaningful decision, whether made by a human or an agent, follows the same structure.

1. Something is evaluated: data points, signals, historical snapshots, versions of truth that existed at that moment.

2. Rules come into play. Policies are checked and constraints surface. Often, those rules collide, leading to conflicts and edge cases.

3. Exceptions are invoked, explicitly or implicitly, and authority is exercised. Someone approves and something is allowed.

Only then does an action commit, and the world changes state.

Most systems record the last step. A context graph is concerned with everything that came before it.

Defining the context graph

A context graph is not a model of entities. It is a record of decisions. Each node represents a decision made at a specific point in time. Not an object, not a metric, but a moment of judgment.

The edges carry the meaning:

• the context that was evaluated

• the constraints that were applied

• the precedents that influenced the outcome

• the authority that sanctioned the decision

• the action that was ultimately committed

Relevance of context graphs in an AI-first world

Agents do not operate inside a single system. They move across boundaries, synthesising signals from many sources before acting. As they do, they must be able to explain themselves: to justify why a path was taken, why an exception was granted, why one precedent outweighed another.

They must learn not just from outcomes, but from prior judgments. And they must be governed (audited, constrained, and trusted) as their autonomy increases. None of this is possible if reasoning disappears at commit time.

Ergo, context graphs form the system of record for AI autonomy. Because they make judgment durable.

Reducing architectural ambiguity: Where context graphs?

The decision boundary

Context is not static. It does not exist before execution, and it does not survive after the fact unless it is deliberately captured. It emerges at the moment a decision is made, when inputs are assembled, policies are evaluated, conflicts surface, and an action is about to be committed.

If context is not recorded before state changes, it is lost. Once the write happens, the world moves on, and the decision-time reality can no longer be faithfully reconstructed.

Why read-path systems fail

Most data systems sit downstream of action. Warehouses receive data after decisions have already been made. By the time information lands, the reasoning that shaped it has evaporated.

Metadata tools observe outcomes (schemas, lineage, usage) but not the judgment that allowed those outcomes to exist. Governance layers define rules, but they do not see how those rules were interpreted, overridden, or reconciled in real situations.

These systems can explain what happened. They cannot explain why it was allowed to happen.

A system that does not authorise writes cannot be the system of record for decisions. And without a system of record for decisions, autonomy will always be fragile, opaque, and capped.

How Data Operating Systems Architecturally Align

A data operating system is an extension of the open data developer platform (DDP) standard that aims to bring the first principles of internal developer platforms (hosting software engineering pillars) to data systems.

While the DDP standardises architectural themes, best practices, and building blocks, the data operating system implements them to the degree of available technologies of today.

A Data Developer Platform is a Unified Infrastructure Specification to abstract complex and distributed subsystems and offer a consistent outcome-first experience to non-expert end users… A DDP provides a set of tools and services to help data professionals manage and analyze data more effectively. (learn more at source)

In analogy to IDP, a DDP is designed to provide data professionals with a set of building blocks that they can use to build data products, services, and data applications more quickly and efficiently.

Decomposition by planes

Control plane = Decision authority

The control plane is where decisions are authorised, instead of just being executed. It centralises orchestration, determines execution order, evaluates policy, and resolves dependencies before any action is taken.

This is also where metadata is naturally emitted. Not as an afterthought, but as a byproduct of deciding what is allowed to happen next. In other words, the control plane already operates at the moment where context is richest and stakes are highest.

As you can observe, the control plane also hosts the heart of the system: the orchestration unit.

The orchestration unit is the only layer that simultaneously sees:

• Inputs as they exist at execution time

• Policies as they are evaluated, not declared

• Dependencies as they are resolved, not documented

• Alternatives as they are rejected

• Exceptions as they are invoked

A context graph can only be constructed where decisions are made, not where results are observed. The orchestration layer already performs this work implicitly. It chooses paths, enforces rules, and authorises transitions.

By emitting these choices as structured events, orchestration becomes the natural producer of a context graph:

Each orchestration step → decision node
Each dependency resolution → an edge
Each policy check → a constraint record
Each override → a precedent

In effect, the control plane does not integrate with a context graph. It is the context graph in motion. Orchestration turns execution into evidence.

Development plane = Intent formalisation

Intent does not exist at runtime unless it is formalised upstream. The development plane captures intent declaratively: what should happen, under which constraints, and toward which goals.

This intent is versioned, explicit, and reviewable. It encodes purpose before execution ever begins. Without this plane, decisions collapse into scripts. With it, decisions become explainable.

Data activation plane = State mutation

This is where decisions touch reality. Actions are executed, writes occur, and systems of record are mutated. Once this plane is crossed, context must already be locked in. Any reasoning that happens after this point is post-rationalisation.

It separates intent, authority, and impact: exactly the conditions required for a context graph to exist as a first-class system, not a derived artefact.

Extending DDP with Decision-trace Awareness

Decision-trace awareness is the ability of a system to capture judgment at the moment it is exercised, not after outcomes are observed.

More thoroughly, decision-trace awareness means the system does not just record what happened, but preserves why a particular path was chosen over all other possible paths.

It is the explicit recording of:

• the inputs that were visible at decision time

• the rules and policies that were evaluated

• the conflicts that were detected

• the exceptions or overrides that were invoked

• the authority under which the decision was made

• the precedent that influenced the choice

Crucially, this trace is captured before state mutation, while context is still intact. Without decision-trace awareness, systems can explain outcomes only retrospectively.
With it, systems can justify actions natively.

What decision-trace awareness adds

Most critically, decisions are linked to prior decisions, establishing precedent, not just provenance. This is how systems begin to remember why something was allowed to happen.

Resulting evolution

This is not an incremental enhancement. It is a categorical shift.

Metadata graph → Context graph
Orchestration → Judgment capture
Automation → Accountable autonomy

When decision traces are explicit, autonomy stops being opaque. It becomes inspectable, governable, and improvable by design.

The Long-term Landscape, If You Plan to Stick

Today’s systems are built to optimise for recording outcomes. They are exceptionally good at telling us what happened, when it happened, and where it landed. But they are structurally blind to the moment that matters most: the instant a choice is made.

Decisions collapse into execution, and reasoning evaporates as soon as state is mutated. This is the current state: efficient, automated, and fundamentally forgetful.

Because judgment is not captured, it cannot accumulate. Each decision is treated as an isolated event rather than part of a continuum. Automation does not learn from itself, just repeats patterns encoded elsewhere.

Introducing decision-trace awareness changes the unit of value from outcome to judgment. When systems capture the context, constraints, and authority behind each action at decision time, reasoning becomes durable.

A single decision no longer disappears after execution, but becomes a reference point. Over time, these references form precedent. This is the transition point from automation as execution to automation as judgment. As precedents accumulate, future decisions improve. The system begins to recognise familiar conditions, validated exceptions, and historically safe paths.

In the long term, this compounding effect produces a qualitatively different system. Not one that runs workflows faster, but one that encodes how the organisation thinks.

MD101 Support ☎️

If you have any queries about the piece, feel free to connect with the author(s). Or feel free to connect with the MD101 team directly at community@moderndata101.com 🧡

Author Connect

Find me on LinkedIn 🤝🏻

From MD101 team 🧡

🧡 The Catalyst: State of Data Products

Introducing The Catalyst, a special edition of The State of Data Products that compiles insights around the ups and downs in the data and AI arena that fast-tracked change, innovation, and impact.

📘 The Catalyst, first release is our annual wrap-up that compiles the most interesting moments of 2025 in terms of developments in AI & data products, with expert insights from across the industry!

Some focal topics

• The gap in AI readiness for enterprises

• The shift towards context engineering

• Addressing governance debt with a focus on lineage gaps

• Revisiting fundamentals for better alignment b/w business vision and AI.

Get your copy!

“The thing that hath been, it is that which shall be; and that which is done is that which shall be done: and there is no new thing under the sun” | Ecclesiastes 1:9

For most of human history, knowledge was scarce. It had to be discovered, recorded, preserved, and transmitted with care. Today, we live in the opposite condition. Whatever we know or talk about has already been published, spoken, or written somewhere. This was true first of literature, then of history, and it became overwhelmingly true with the internet.

“All is said, we have come too late; for more than seven thousand years there have been humans, thinking” | Jean de La Bruyère

Need a recipe? It exists in some forgotten blog post, a forum thread, a video, or a scanned cookbook. Need to learn a programming language? There are countless tutorials, curricula, walkthroughs, and opinionated guides. Need a detailed breakdown of a niche concept that only a handful of people care about? Chances are, someone has already written it, explained it, argued about it, and archived it online.

The Untapped Power of Discovery

B2C AI chatbots are built to thrive in exactly this environment. At their core, they function as extraordinarily sophisticated search engines: systems that ingest the vast expanse of publicly available information and learn how to combine, filter, compress, and summarise it in a way that aligns with the user’s intent.

Their real strength is not recall alone, but intent interpretation: understanding who the user is, what they are likely trying to achieve, and how much depth or precision is required in the response.

These systems are remarkably good at deciding what not to search, narrowing the space of possibilities to save compute/time, and identifying similarities and patterns with almost unsettling accuracy. They do not just retrieve information, but reorganise it on demand.

And yet, we all implicitly accept that these responses come with a margin of error. Hallucinations, approximations, and confident inaccuracies are part of the bargain. In a consumer context, this is largely acceptable. The cost of being wrong is usually low. A flawed explanation, an imperfect recommendation, or a slightly incorrect summary rarely carries material consequences.

Why Untapped

But this acceptance is a product of a B2C operating lens. The moment we move into B2B environments, the tolerance for error collapses.

Business decisions are not exploratory curiosities, they compound. They affect revenue, compliance, customer trust, and operational stability. In these contexts, inaccuracies are liabilities.

While B2C use cases tend to centre around research, learning, navigation, and individual productivity, B2B use cases are fundamentally collective. They inform decisions made by teams, executed by systems, and scaled across organisations.

The question, then, is not whether AI can generate answers, but whether those answers can be trusted when the cost of being wrong is no longer personal, but organisational.

What problem are we actually solving?

Humans do not need more data. They need to know what exists, whether it can be trusted, and whether it can be used safely. Fast enough to act.

Everything else (catalogs, lineage graphs, AI copilots) is implementation detail.

The irreducible questions every data consumer asks, implicitly or explicitly:

1. Does this data exist?

2. Can I access it?

3. Can I trust it?

4. Is it relevant to my goal, right now?

5. What happens if I use it incorrectly (the cost of error)?

These questions are universal, regardless of whether the person asking them is an analyst, a machine learning engineer, a product leader, or an executive.

The data catalog, in its original form, emerged as an attempt to answer these questions in a static world. It documented assets, traced lineage (barely), assigned ownership, and tried to encode trust through description.

In modern systems today, AI is the new interface for answering the very same questions.

The Gap

This shift exposes a fault that most conversations conveniently avoid. An AI that merely retrieves descriptions is still operating at the surface of the problem, like any other integrated catalog.

Additionally, with hallucinations, the AI version seems to be faster, more fluent, and more confident (tools that easily rope in humans’ trust during first few interaction levels). But the AI is not “smarter” yet.

The inflexion point appears only when AI begins to reason over evidence: quality signals, usage patterns, access constraints, lineage, and risk, all grounded in the operational reality of the data platform.

Breaking Discovery Into Smaller Solutions

To make progress, the problem itself has to be reduced. “AI-powered data discovery” is too large, too abstract, and too flattering a phrase to be useful.

It bundles multiple distinct problems into a single label and gives the illusion of completeness without earning it. The only way to reason about it clearly is to decompose it into smaller units of atomic solvable problems.

At the highest level, the problem sounds simple: help me discover the right data for my intent, safely and confidently.

But this apparent simplicity hides a chain of judgments that must all hold true at the same time. Discovery is not a single action, but a sequence of judgements. And every weak link in that sequence disrupts trust or the data consumer experience almost completely.

1. Existence problem: What data assets exist?

2. Access problem: Am I allowed to see or use them?

3. Context problem: What do these assets mean?

4. Quality problem: Should I trust them?

5. Usage problem: How are they actually being used today?

6. Intent alignment problem: Which asset fits my goal, not just any goal?

7. Risk problem: What could go wrong if I use this?

Traditional data catalogs attempt to answer some of these questions, usually in isolation and usually statically. They describe, document, and point. But discovery, in its true sense, is dynamic.

An AI-driven discovery system must be able to evaluate all of these dimensions together, in real time, and in context. And context is what isolated or independent catalogs do not have access to, given the lack of knowledge, context, visibility, and granular logs operating within native data platforms of organisations.

Any catalog that promises AI-led discovery is, in most cases, working on a surface level with AI, given that it is inherently unable to reason over key ecosystem information that is not accessible to it due to tooling/regulatory/lock-in barriers.

AI is NOT the Data Catalog of the Past

Discovery is inference grounded in evidence. When AI is asked to help users navigate data without access to observable signals like quality, usage, access, change logs, it has no choice but to substitute reasoning with approximation. The output may sound coherent, even authoritative, but it is untethered from reality.

Which is most dangerous in business setups, given the AI’s ability to simulate trust is almost too frequent in one-on-one interactions with humans.

Reconfiguring AI as a discovery tool only works when the underlying platform behaves like an instrumented system.

• Quality must be measurable: freshness, completeness, test health, and failure history need to exist as first-class signals.

• Usage must be visible: who relies on the data, how frequently it is queried, and which workflows break if it disappears.

• Access rules must be explicit and machine-readable.

• And metadata itself must move beyond description, serving as something systems can reason over.

In the absence of these foundations, AI does not become smarter, but it definitely appears to be more confident. It starts to resemble a search engine that speaks fluently, stitching together partial truths and plausible narratives without accountability.

A reasoning system, by contrast, is constrained by evidence. It can explain not just what it recommends, but why and, just as importantly, when not to.

Meeting in the Middle: Top-down AI × Bottom-up
Metadata Infrastructure

From the top down, AI brings intent. It understands natural language, infers personas, and adapts to goals that are often loosely articulated. This is the visible side of the system, and it is genuinely powerful. This is the part of the system most people see and often overestimate.

But intent by itself is ambiguous. Two people can ask the same question and mean entirely different things. Without grounding, intent becomes speculation.

A foundational data infrastructure must actively assess and gauge the state of the data estate. It does this by observing pipelines in motion, tracking freshness and failures, running quality checks, recording access patterns, capturing lineage as systems evolve, and enforcing policies at runtime. Metadata, in this sense, is measured instead of being written.

Measured metadata forms the bridge to reasonable AI

It translates raw operational signals into something AI can interpret: which datasets are healthy, which ones are trusted, which are widely relied upon, which are volatile, and which carry risk. It also encodes constraints like who can access what, for which purposes, and under what conditions. Without this bridge, AI has no stable reference frame. It can speak fluently, but it cannot judge.

Once this foundation exists, AI’s role changes fundamentally. It is no longer guessing relevance but evaluating. It is no longer summarising descriptions but reasoning over observed behaviour.

It can weigh intent against evidence, adapt recommendations based on persona and risk tolerance, and explain why one dataset fits a goal better than another. The quality of its answers becomes a function of the quality of the signals it consumes.

This is why meeting in the middle is not optional.

• Top-down intelligence without bottom-up measurement produces hallucination: answers that sound right but are untethered from reality.

• Bottom-up measurement without top-down intelligence produces complexity: truth that exists but cannot be navigated simply.

The bridge between the two is the system that makes discovery at par with current AI systems: a platform that measures reality continuously, and an AI that knows how to reason with it.

Optimising Discovery with a Chain of Agents

The most common assumption is that discovery is a single act. Ask a question, get an answer. In reality, discovery is not atomic. It is a sequence of judgments, each dependent on the previous one being correct.

From first principles, any discovery flow that aspires to be production-grade must be decomposed into a sequence of bounded judgments. Each judgment becomes an agent because it enforces a clean contract: specific inputs, explicit outputs, and known error surfaces.

Access Agent

Enforcing hard boundaries before reasoning begins.

Intent: The Access Agent exists to terminate invalid discovery paths as early as possible. Its job is not to explain policy, but to enforce it deterministically.

Consider a user asking, “What customer revenue data can I use for churn modelling?” Before intent is interpreted or metadata is explored, the system must resolve whether the user can access raw customer-level financial data, only aggregated metrics, or nothing at all. This agent reads RBAC/ABAC rules, purpose constraints, and data contracts, and produces a filtered worldview of permissible assets.

Why is this not implicit?
If access checks are deferred or embedded inside a general reasoning step, the system reasons about assets it should never have seen. That creates both security and hallucination risk. Architecturally, access must be a gate instead of a suggestion.

Long-term system impact: Over time, discovery becomes policy-shaped by default. The system never “forgets” access rules, even when humans do.

Metadata & Evidence Agent

Separating observation from assumption.

Intent: This agent exists to answer a narrow but critical question: what can we prove about this data right now?

Take two datasets labelled “customer_orders.” One is refreshed hourly, heavily used in revenue reporting, covered by tests, and owned by finance. The other is refreshed sporadically, used only by one analyst, and fails freshness checks weekly. A human might not notice this distinction. This agent must.

It consumes schema, lineage graphs, freshness metrics, test results, failure logs, and usage telemetry to produce derived signals such as confidence, stability, and operational risk.

Why an independent agent for evidence?
Evidence evaluation is not intent-dependent. It must be consistent regardless of who asks the question. Architecturally, that makes it a reusable judgment unit.

Long-term system impact: The system stops treating metadata as documentation and starts treating it as evidence.

Intent Interpretation Agent

Resolving ambiguity.

Intent: This agent exists to resolve ambiguity in human input before any asset ranking occurs.

A question like “Show me user engagement data” is underspecified. Is this for executive reporting, product experimentation, or ML feature generation? The same words imply different correctness criteria. This agent infers persona, domain context, urgency, and acceptable risk based on the user’s role, historical behaviour, and session context.

Why can this not be merged with retrieval?
Intent is not a property of data, but a property of the user. Mixing intent inference with data reasoning creates circular logic and unstable outputs.

Long-term system impact: Discovery becomes outcome-aligned rather than query-aligned. The system always optimises for usefulness over literal matches.

Relevance & Fit Agent

Evaluating suitability.

Intent: This agent answers the actual selection question: which asset best fits this intent under current constraints?

For financial forecasting, a slower but audited dataset may be preferred. For exploratory analysis, a fresher but noisier dataset might be superior. This agent compares evidence (quality, usage, stability) against intent-derived requirements.

Why make this an independent agent?
Most systems implicitly equate relevance with description quality or usage frequency. Making this an agent forces relevance criteria to be articulated and inspected instead of being rule-driven.

Long-term system impact: Data selection becomes defensible. When decisions are questioned, the system can explain tradeoffs.

Synthesis Agent

Assembling judgment into explanation

Intent: This agent exists to convert a multi-step reasoning process into a human-comprehensible outcome.

Rather than saying “Use dataset X,” it explains: Dataset X is recommended because it is refreshed daily, widely used in finance workflows, passes quality checks, and aligns with your reporting intent. Dataset Y was excluded due to freshness and access constraints.

Without synthesis, multi-agent systems feel arbitrary. Explanation and reasoning evidence convert human doubt into trust.

Long-term system impact: Discovery decisions become auditable and teachable, reducing reliance on tribal knowledge.

Lineage & Blast Radius Agent

Reasoning about consequences.

Intent: This agent evaluates downstream impact before action is taken.

If a dataset feeds a revenue dashboard, a pricing model, and an external report, using it for experimentation carries risk. This agent traverses lineage to surface those dependencies explicitly.

Long-term system impact: The organisation develops systemic awareness. Changes stop being local accidents and start being intentional actions, which often turn out to be rewarding to humans.

Risk & Compliance Agent

Making constraints first-class

Intent: This agent ensures that legal and contractual realities shape discovery. Reactive governance is way in the past for this agent.

A dataset may be technically perfect but unusable for a given purpose due to consent or contractual limits. This agent reads PII classifications, consent scopes, and historical violations to constrain discovery paths.

Long-term system impact: Risk evaluation shifts from post-hoc governance to pre-decision design.

Change & Drift Agent

Invalidating stale trust.

Intent: This agent asks whether past assumptions still hold.

If schema changes invalidate a downstream model or usage drops sharply, this agent dials down confidence scores/ranks automatically.

Long-term system impact: System resists fossilisation.

Learning & Memory Agent

Institutionalising experience.

Intent: This agent captures outcomes to improve future discovery.

If a dataset is repeatedly selected and later leads to incidents, the system learns to penalise it. If recommendations are ignored, the system adjusts relevance.

Long-term impact: AI becomes organisational memory.

Overview of Our Recommended Discovery Agents

Note that the sequence in the image follows the order in which they’re described. While some of the agents are sequential, some operate in parallel or interchange data both ways.

Author Connect 🤝🏻

Discuss shared technologies and ideas by connecting with our Authors or directly dropping us queries on community@moderndata101.com

Find me on LinkedIn 🙌🏻

MD101 Support ☎️

If you have any queries about the piece, feel free to connect with the author(s). Or feel free to connect with the MD101 team directly at community@moderndata101.com 🧡

Last Thursday, returning home from Washington, D.C., on the late train to Princeton, I found my seat occupied by a gentleman in his seventies. He was traveling with his wife, surrounded by four large suitcases, the kind meant for international journeys. The bags had wheels at the bottom, and he was holding onto them with evident concern.

Anyone who has traveled on Amtrak knows the problem: the trains lack proper restraining mechanisms for large luggage. Without someone holding them, those wheeled bags would roll across the aisle with every curve in the track.

As a good Samaritan, I offered to help secure a couple of them. We struck up a conversation.

When I asked what he did for a living, he said he was retired but had worked in LPI, or Low Probability of Intercept. I had never heard the term. He explained it succinctly:

A class of radar and communication technologies designed to evade detection, used primarily in defense systems but with some commercial applications.

In under a minute, he had compressed decades of expertise into a crisp, accessible explanation. I was impressed.

Then he turned the question back to me.

So what does AI actually do? How does it change lives?

I felt the pressure immediately. Here was a man who had just distilled a complex defense technology into sixty seconds. His wife was gathering their belongings. The train was approaching their stop. As a technology executive, I needed to compress this massive, sprawling concept into something equally elegant.

My mind raced through metaphors. And then it struck me:

As the couple prepared to disembark, rolling their wheeled bags toward the exit, the metaphor seemed almost too perfect. He now had a fresh perspective on the casters beneath his luggage, those simple rotating mechanisms that trace their lineage back five thousand years to the plains of Mesopotamia.

I had another couple of hours in the quiet of the late-night train. I kept thinking about it. I researched the history. And the more I looked, the more I realized that the wheel’s story offers a remarkably precise lens for understanding what we are living through with artificial intelligence. Its delayed invention, its uneven adoption, and its transformative consequences all find echoes in our current moment.

The Wheel’s Mysterious Delay

The wheel appeared around 3500 BCE in Mesopotamia, modern-day Iraq. This seems early until you consider what humans had already accomplished without it. Agriculture had been practiced for millennia. Complex textiles were being woven. Boats navigated rivers and coastlines. Metallurgy had produced bronze tools and weapons. The great pyramids of Egypt were constructed by moving massive stones on wooden sledges and rollers.

The concept of something round that rolls was not a mystery. Humans had been using logs to move heavy objects since prehistory.

Attaching a wheel to a fixed axle, or a rotating axle to a stationary platform, required solving problems that were far from obvious. The tolerances had to be precise: too tight, and friction renders the mechanism useless. Too loose, and the wheel wobbles apart. You needed sophisticated carpentry, an understanding of lubrication, and materials strong enough to bear loads while remaining light enough to be practical.

The most likely evolutionary path ran through the potter’s wheel. Artisans spinning clay had already solved part of the rotational mechanics. Someone, we will never know who, recognized that the same principle could move objects horizontally, not just spin them in place.

But the wheel only became transformative once the surrounding infrastructure caught up. Wheels require relatively flat, cleared paths. They require draft animals capable of pulling carts. They require a density of settlement and trade that justifies the investment in building roads and vehicles. The technology preceded its widespread adoption by centuries because the ecosystem was not ready.

The Axle Problem

This is where the parallel to artificial intelligence becomes striking.

We have built the wheel. Large language models, computer vision systems, and predictive algorithms are the visible, conceptually graspable parts of the revolution. People can see a chatbot responding to questions or an image generator producing artwork and understand, at least superficially, what is happening.

But the real engineering challenge, and the reason adoption lags capability by years, is the axle.

How do you connect an AI model to existing workflows, databases, and decision-making processes?

How do you integrate it with legacy systems that were never designed to interface with probabilistic reasoning?

How do you establish the tolerances: enough autonomy for the AI to be useful, enough oversight that humans remain in control?

Just as the ancient wheel needed the right fit between hub and axle, AI needs the right fit between model and enterprise. Too tight, with too many restrictions and too much human review, and the friction eliminates any efficiency gains. Too loose, with too much autonomy and too little validation, and the system produces errors that undermine trust.

The organizations that will benefit most from AI are not necessarily those with access to the most sophisticated models. They are the ones that solve the axle problem by building the data infrastructure, the integration layers, and the governance frameworks that allow AI capabilities to actually connect to real operations.

Evidence of the Gap

In healthcare, AI systems have demonstrated radiologist-level accuracy in detecting certain cancers and diabetic retinopathy since 2016 and 2017. Yet most diagnostic imaging worldwide still relies primarily on human interpretation. The barriers are not technological. They are liability concerns about who bears responsibility when the AI misses something, reimbursement models that do not account for AI-assisted diagnosis, physician skepticism, regulatory approval timelines, and the difficulty of integrating new tools with hospital IT systems that were designed decades ago.

In legal services, AI-powered contract analysis and document discovery tools have been commercially available for nearly a decade, with demonstrated ability to review materials faster and often more accurately than junior associates. Adoption remains limited. The billable hour model actively discourages efficiency gains. Partners resist changing workflows that have served them well. Clients worry about confidentiality. Malpractice concerns loom.

In manufacturing, predictive maintenance systems have been available since the mid-2010s. These combine sensors and machine learning models to anticipate equipment failures before they occur. Adoption is patchy despite obvious return on investment. Factory equipment often lacks the sensor infrastructure. Operational technology teams are unfamiliar with data science. Organizations default to scheduled maintenance because that is what they know, and it is difficult to quantify the value of a failure that never happened.

In agriculture, computer vision for crop monitoring and AI-driven irrigation optimization have existed for years. Adoption among farmers remains low outside large industrial operations. Rural connectivity is unreliable. Upfront costs are prohibitive for small operations. The demographic of farmers skews older and more skeptical of black-box recommendations.

In government, AI tools for fraud detection, benefit eligibility determination, and citizen service automation exist and work. Government adoption lags the private sector by years. Procurement regulations move slowly. Political sensitivities around automated decision-making are acute. Workforces are unionized and resistant. Institutional cultures are risk-averse.

The Enabling Technologies

Once the basic wheel existed, rapid differentiation followed. Spoked wheels emerged for chariots, offering lighter weight and faster speeds. Solid wheels persisted for heavy carts, providing more durability under load. Potter’s wheels were optimized for rotation speed and control. Each variant was tuned for different tolerances and use cases.

We are witnessing the same pattern with AI.
General-purpose foundation models are spawning specialized variants for code generation, legal documents, medical imaging, conversational interfaces, and financial analysis. The base technology is fragmenting into purpose-built applications, each optimized for its domain.

The biggest enablers in technology today are not the AI algorithms. They are the data platforms that make enterprise information accessible, governed, and ready for machine consumption. They are the integration layers that connect AI capabilities to operational systems. They are the observability tools that let organizations understand what their AI systems are actually doing. They are the security frameworks that protect sensitive data while still allowing AI to extract value from it.

A company can license the most sophisticated language model on the market, but without clean data pipelines, without APIs that expose the right information at the right time, without monitoring that catches when the model drifts off course, the technology sits idle. Like a wheel without roads.

Network Effect and Compounding Returns

Wheels enabled carts, which justified roads, which enabled trade routes, which created demand for better wheels and more roads. Each improvement reinforced the others. The wheel did not just solve a transportation problem.

AI shows the same compounding dynamics, but on a compressed timeline. Better models create more usage, which generates more data, which funds more research, which produces better models.

💡 Organizations that enter this flywheel early accumulate advantages that become increasingly difficult for latecomers to replicate.

The firms that are extracting value from AI today are not necessarily the ones with the best algorithms. They are the ones who started building their data infrastructure five or ten years ago. They are the ones whose workflows were already digitized, whose data was already centralized, whose employees were already comfortable working alongside automated systems.

The AI models became powerful enough to matter only recently, but the organizations that can deploy them were preparing for decades.

This is the uncomfortable truth for enterprises that feel they are falling behind: you cannot simply purchase your way to AI maturity. The underlying infrastructure, including data governance, integration architecture, and cultural readiness, takes years to build. Starting now is essential, but expecting immediate results is unrealistic.

📝 Related Read

Resistance and Redistribution

Historical evidence suggests that some communities resisted wheeled transport because it disrupted existing porter and pack-animal economies. The transition was not purely technological but social and political. Those who made their living carrying goods on their backs or managing mule trains had every reason to oppose a technology that would render their skills obsolete.

Today’s debates about AI in creative industries, knowledge work, and professional services echo this pattern. The technology’s capability is rarely the question. The redistribution of economic value is.

Writers worry about being replaced by generators. Artists resist having their work used to train systems that compete with them. Lawyers and consultants see junior roles being automated before the senior partners have figured out how to bill for the new capabilities.

This resistance is rational. Technological transitions create winners and losers, and the losers have every incentive to slow the transition down. The political economy of adoption matters as much as the technical feasibility.

Organizations that want to accelerate their AI journey must grapple not just with technical integration but with workforce anxiety, union negotiations, retraining requirements, and the simple human reluctance to change.

Uneven Terrain

The wheel thrived in the flat, dry plains of Mesopotamia but was less useful in mountainous terrain or dense forests. The technology fit certain contexts better than others. Civilizations that lacked suitable geography or draft animals, like the pre-Columbian Americas, developed sophisticated societies without ever inventing the wheel for transport.

AI adoption will follow similar contours.
Some industries, regulatory environments, and organizational cultures will prove more hospitable than others, not because of the technology’s inherent limits but because of contextual fit.

Highly regulated industries like healthcare and financial services face compliance burdens that slow experimentation. Organizations with fragmented data landscapes cannot easily train or deploy models that require unified views of their information. Cultures that prize individual judgment and resist standardization will struggle to accept algorithmic recommendations.

This means that competitive advantage from AI will not be uniform. Some enterprises will leap ahead because their context, including their data maturity, regulatory environment, and workforce culture, is favorable to rapid adoption. Others will lag not because they lack access to technology but because their terrain is rougher.

Levers and Limits

While the technology landscape appears flat, with the same models and tools available to everyone, the levers that determine enterprise success are remarkably uneven.

Data Readiness

Data readiness is the primary lever. Organizations with centralized, well-governed, high-quality data can move from experimentation to production in months. Those with fragmented, siloed, poorly documented data spend years just preparing the foundation. This is not a technology problem but an organizational and governance problem. But it determines everything that follows.

Integration Architecture

Integration architecture is the second lever. AI models are only as useful as the systems they connect to. If your customer service AI cannot access order history, inventory status, and shipping information in real time, it cannot actually solve customer problems. The enterprises that built modern API-driven architectures, often for reasons having nothing to do with AI, now find themselves with a structural advantage. Those still running on monolithic legacy systems face years of re-platforming before AI can deliver meaningful value.

Workforce Capability

Workforce capability is the third lever. The wheelwright was an essential craftsperson in every settlement, maintaining and repairing the technology that enabled commerce and agriculture. AI is creating analogous roles: prompt engineers, ML operations specialists, AI ethics reviewers, and fine-tuning experts. Organizations that develop this internal capability, either through hiring or training, will iterate faster and depend less on external vendors. Those who outsource everything will find themselves perpetually behind.

Culture

Culture is the more hidden lever. Some organizations embrace experimentation while others punish failure. Some trust algorithms while others demand human approval for every decision. Some see technology as a strategic asset while others see it as a cost center. These cultural orientations, often set decades ago by founders or industry norms, now determine how quickly an enterprise can move. Changing culture is possible but slow, often slower than changing technology.

Regulatory Environment

Regulatory environment is the external limit. No enterprise operates in isolation. Financial services firms must comply with model risk management requirements. Healthcare organizations must navigate FDA clearances. Government contractors face procurement rules that were written for a different era. These constraints are not insurmountable, but they determine the pace of adoption regardless of internal readiness.

Time stood by, until it didn’t

One divergence from the historical parallel is worth noting: wheels took centuries to spread across civilizations. AI is propagating in years.

When the wheel spread slowly, societies had generations to adapt. New occupations emerged gradually. Road networks expanded incrementally. Social norms evolved to accommodate wheeled transport without sudden disruption.

AI allows no such luxury. Organizations have months instead of generations to develop new capabilities. Workforces must retrain in real time. Business models that were viable a decade ago may be obsolete before current strategies fully execute.

The enterprises that thrive will be those capable of learning and adapting at a pace that would have been unthinkable in prior technological revolutions.

This is not a comfortable message. It creates pressure, anxiety, and a sense that falling behind, even briefly, may be irrecoverable. But discomfort is not a reason to ignore reality. The timeline is what it is.

The View from the Train

As my fellow traveler rolled his bags off the Amtrak and toward his international flight, I wondered what he made of our brief exchange. He had spent a career in low probability of intercept, technologies designed to avoid detection. Perhaps there was a lesson there too: the most consequential technologies are often the ones you don’t see coming until they have already changed everything.

The wheel, after all, seems obvious in retrospect. We look at it now and cannot imagine why it took so long to invent. But for millennia, humans moved enormous stones, built pyramids, and conducted agriculture without it. The conceptual leap was not obvious at all.

Future generations may view our current AI debates with the same puzzlement. They will wonder why it took so long to apply these systems to problems that, in retrospect, were obviously suited to them. They will struggle to understand why some organizations resisted while others leapt ahead.

The challenge is that we cannot yet see which applications will feel inevitable in retrospect. We are living through the transition, not studying it from the safe distance of history.

What we can do is learn from the patterns. The enabling infrastructure matters more than the technology itself. Adoption follows unexpected paths. Late arrival does not mean less importance. Resistance comes from entrenched interests. Geographic and cultural factors shape deployment.

And above all: the wheel works fine. The question is whether you have proper axles and roads.

Part 2 now published 🔖

Author Connect 🤝🏻

Discuss shared technologies and ideas by connecting with our Authors or directly dropping us queries on community@moderndata101.com

Connect with Sagar on LinkedIn 🤝🏻

MD101 Support ☎️

If you have any queries about the piece, feel free to connect with the author(s). Or feel free to connect with the MD101 team directly at community@moderndata101.com 🧡

Contribute Your Voice

Join one of the most diverse and forward-thinking data communities. With 60+ authors spanning data architects, tech founders, strategists, consultants, product managers, SMEs, and industry leaders, our collective voice shapes how the world understands modern data.

Modern Data 101 is read across all 50 U.S. states and reaches readers in 135+ countries, giving every contribution a truly global footprint. We’re excited to learn more about you and explore how your expertise can help push the conversation forward. Let’s get started.

Join Modern Data 101 Authors