Where You Place AI Determines Your Odds
The strategic trade-offs behind AI placement with your data platform: The choice of embedding, sharing, or externalising AI infrastructure.
An AI-Data Burden Paradox
Most organisations today are collecting more data than ever, but what was meant to be a competitive edge has become an operational bottleneck. Enterprises have spent years integrating cloud warehouses, cataloguing assets, and standing up pipelines. Yet for many, the result isn’t fluid access or insight velocity, it’s stack fatigue. The very infrastructure meant to liberate data often ends up making it harder to use.
The promise of AI is to turn this around: to abstract away complexity, accelerate insight delivery, and shift data from a cost centre to a competitive advantage.
But here’s the paradox: AI, too, might become a burden.
Why?
Organisations are already juggling a sprawl of technology blocks, from analytics engines to metadata stores, each adding its own governance layers, API contracts, and cost centres. Now, AI enters the picture with another set of infrastructure demands. One is the cost for LLMs, and the other is around the data, which will provide the context to AI. It’s yet another layer on an already expensive stack with new added infrastructure cost, new governance challenges, and integration overhead.
The promise of abstraction turns into another operational tax.
This is where the conversation needs to shift. AI shouldn't be a last-mile feature bolted onto an already strained architecture. It should be a foundational enabler, one that helps transform complex processes powered by data into adaptive, intelligent systems. Not by layering on more tools, but by collapsing the complexity that stalls value creation.
The Role of AI in Abstracting the Data Stack Complexity
The modern data stack has layered itself into a maze of pipelines, tools, and handoffs. AI doesn’t need to replace these components to add value; rather, it could transform how we interact with them.
Rather than navigating a number of stacks to trace a data issue or generate a report, AI becomes the abstraction layer, simplifying orchestration, translating intent, and surfacing context in real time. It reshapes the user experience across the stack, collapsing layers into intuitive interactions.
Think about how most users engage with data today: they navigate catalogs, construct SQL, interpret dashboards, and still often need help translating it all into business action. AI has the potential to bypass many of these manual touchpoints, but also execute on these insights autonomously for faster business impact. With natural language interfaces, generative explainability, and agentic workflows, AI systems can let users ask questions and receive answers without tracing pipelines or toggling between tools.
But for AI to do this effectively, it needs more than just model access. It requires contextual understanding of the data, the semantics of the domain, and governed access to the right information. That’s where infrastructure matters.
AI, positioned correctly, becomes the interface layer to the stack, surfacing insights, triggering actions, and automating low-value tasks. It turns pipelines into conversations and documentation into guidance. It shifts the role of the data platform from a system of tools to a system of intelligence.
Done right, this is not just a UX enhancement. It’s a fundamental architecture evolution. The goal is to move from bolted-on intelligence to a more AI-first intentional design, where AI is embedded at the core of how products behave, adapt, and govern themselves.
Embed AI Meaningfully: Turning Platforms into AI-Native Builders
Embedding AI meaningfully isn’t about adding chatbots or slapping on dashboards. It’s about enabling AI-native data apps, applications where AI isn’t an afterthought, but a core part of how data is consumed, orchestrated, and translated into action.
The key isn’t to compete with BI tools, but to empower them and move beyond the dashboards, bridging the gap to make a bigger business impact. Tools like Power BI, Tableau, or Looker have their own UIs, semantics, and user bases. They’re not going away. What these tools lack is context-rich, AI-consumable data, and that’s exactly what your platform can provide.
By exposing AI-ready data using protocols like MCP (Model Context Protocol) and A2A (Agent-to-Agent interoperability), your platform becomes the ideal substrate for AI agents and BI tools alike. You’re not replacing visualisation tools, you’re turning your platform into a connective tissue that feeds them with structured semantics, promptable outputs, and context-aware responses.
📝 Related Read
Building this capability can work in four layers:
A consumption layer,
where data becomes promptable. Vector stores, graph stores, and curated data products with embedded context make data legible for LLMs and agents. This is where transformation happens, from raw to ready-to-query.
A memory layer,
which lets your platform retain context across interactions. Instead of treating every prompt like a blank slate, this layer stores factual knowledge, user preferences, and episodic context across sessions so that agents get smarter over time. It complements retrieval by deciding what is worth remembering and surfaces only the most relevant context into the model’s prompt, reducing cost while increasing relevance.
An orchestration layer,
that manages how prompts, queries, and actions travel between user interfaces and model runtimes. It handles agent execution, context chaining, retrieval augmentation, and any logic required to convert user intent into a meaningful AI call.
A synthesis layer,
where insight becomes usable. Natural language outputs, visual summaries, metadata suggestions, all land here. It ensures the AI output is consumable by a human or another system (e.g., a BI tool, CMS, or API consumer).
Now let’s try to decode how we can position the AI infra layer to power business applications, build AI apps and also enrich existing data stacks.
AI-Infrastructure Layer Placement and Why It Matters
How you position AI infrastructure with your data stack isn’t just a technical architecture decision; it’s a product strategy call. It dictates who benefits from the intelligence you're enabling, how scalable your solution is, and whether your AI efforts amplify your platform or become a separate cost centre.
We see three primary models emerging, each with distinct trade-offs:
Type 1: AI Embedded Within the Data Platform (AI-for-You)
Placing an AI layer within the data product/data developer platform is the most tightly integrated model. The AI infrastructure, vector stores, LLM runtimes, and orchestration layers sit inside your data developer platform. Here, AI works natively within your system to enhance internal workflows, power proprietary agents, and automate internal intelligence. The output of AI directly shapes the experience your platform offers.
Think Apple Intelligence: AI here is not exposed as a service. It's deeply built into the user experience across devices. Summarising notifications, generating images, or prioritising alerts, the intelligence doesn’t sit outside the ecosystem. It’s integrated natively into iOS and macOS workflows, running on-device or within Apple’s own private cloud infrastructure.
The AI is dissolving complexity within the platform: surfacing what matters, adapting to context, and operating securely within tight feedback loops. Since it’s embedded, it becomes part of the system: context-aware, low-latency, and privacy-preserving.
When placed inside the platform, AI becomes part of the operational loop, improving system adaptability and enhancing response to user signals.
It helps close the burden-to-asset loop directly within the platform itself.
But there’s a limit: the benefits stay internal. Your clients or partner ecosystem can't build on top of your AI infra, and you also carry the full cost of hosting and scaling it.
Type 2: AI as a Shared Horizontal Layer (AI-for-You-and-Them)
This is the hybrid placement. AI infrastructure is treated as a foundational, horizontal capability: slightly distinct from the data platform, but deeply connected to it via APIs, shared context, and orchestration logic.
This allows your platform to both consume and expose AI services, a setup that enables your internal teams and your customers to build on the same base. It becomes a utility layer that powers embedded experiences within your stack (like natural language query or auto-insight modules), and also supports external development of AI-native data apps.
For instance, when a user triggers a query inside a BI tool, it may route through your AI orchestration layer, leverage your vector DB or metadata store, and generate a contextual response, all without needing the AI stack to be rebuilt externally.
📝 Related Read
Why Vector DB should be an inherent feature of Data Developer Platforms
The real advantage here is extensibility. You retain control and insight into how AI is applied, but you also allow others to plug in. You don’t just accelerate your own roadmap, you enable your ecosystem to build with intelligence too.
As an organisation’s GenAI strategy matures, a shared AI infra layer supports self-serve application developments, where internal agents, client-facing apps, and partner use cases can be driven by the same infra. In several cases, your platform becomes the platform for AI-native data builders alongside a simple data platform with AI features.
This might turn out to be a boon for teams pursuing platform extensibility and differentiated experiences.
Type 3: AI as an External Utility (AI-for-Them)
This is when the AI layer can be used mostly by clients, becoming useful for building on top of your asset, but does not enrich the internal stack.
In this pattern, the AI infrastructure sits entirely outside your data platform, perhaps built on a separate stack (like AWS-hosted LLMs, or third-party orchestration layers). The platform may send prompts or data externally via API calls, and the AI system responds, but the intelligence doesn’t enrich or persist within your system.
Think Amazon Bedrock. It offers access to foundation models, Claude, Mistral, and Llama, via APIs, enabling customers to build generative AI applications on their own terms. But it’s a customer-facing utility, not something deeply embedded into AWS’s own internal workflows. The intelligence lives on the edge, useful, elastic, external.
This is AI as a service instead of AI as a system. Your platform becomes a conduit.
✅ The tradeoff? You retain flexibility, and your clients move faster, but none of the intelligence returns to strengthen your own stack. The learning stays external. You’ve provided the infrastructure, but haven’t built a feedback loop.
While this model is quick to implement and easy to swap out, it’s also the least strategic. The data platform becomes a conduit, not a builder. Intelligence lives elsewhere. The cost of inference, tuning, and context management belongs to someone else, but so does the value.
Making MCPs your Shining Armour
There are ample opportunities to derive the most from your MCPs. MCPs are most valuable when:
1. context is close to the data (e.g., metadata, semantics, lineage, policies),
2. agents or models need that context to reason, generate, or automate, and
3. the platform wants to maintain and evolve intelligence internally over time.
In the first and second placement models, the AI layer has access to this context because it is interfaced directly with the platform's internal governance and orchestration systems. That means when an agent or an external application queries data via the shared AI layer, it doesn’t just get a response; it gets a context-rich, promptable payload governed by MCP.
The proximity allows precise responses suited to domain semantics; governed outputs that respect access rules and classifications; composable intelligence, where agents can chain context between workflows; bi-directional feedback loops, where user interactions inform future recommendations or metadata suggestions.
📝 Related Read
Enabling duality: AI as a service for others, and as a strategic asset for yourself. It lets you expose intelligence via shared protocols like MCP while still retaining ownership of the underlying semantics. That’s what makes it a platform-level play, not just a feature layer.
Choosing the Right Positioning Is a Product Strategy Decision
This isn’t just an architecture question. It’s a product strategy decision masked as infrastructure.
Where you place the AI infrastructure, within, beside, or outside the platform, shapes far more than performance or deployment patterns. It’s a bet on how you plan to distribute intelligence, how you want users to consume it, and how feedback will shape future AI behaviour.
If you embed AI entirely within your data platform, you’re optimising for tight control. The AI is tuned to your internal semantics, your governance boundaries, and your performance constraints. It’s ideal for building smart internal features and boosting in-platform workflows, but that’s where it stops. Value stays internal. There’s no natural path for others to extend or reuse what’s been built.
If the AI layer sits outside, you’re likely chasing speed and external reach. You can spin up new interfaces, deploy use-case-specific apps, and integrate off-platform tooling quickly. But this freedom comes at a cost: disconnected insights, duplicated governance, and little to no enrichment of the core platform. It’s velocity instead of compounding value.
That’s why the shared horizontal model matters. It gives you both embedded context and extensible reach. This model allows AI capabilities to be exposed as infrastructure, not just features.
That means multiple teams, tools, or client-facing apps can tap into the same intelligence foundation, without rebuilding or reinterpreting each time. It creates a flywheel: every interaction, every agent, every query strengthens the system.
The choice isn’t about which model is “best.” It’s about what you’re solving for right now, and what you’re preparing for next. Are you looking to improve internal efficiency? Enable external builders? Or create a shared intelligence layer that scales with both?
Most mature product strategies don’t pick one. They evolve from the inside out. Starting with embedded AI to drive early value, moving toward shared layers that unlock client-facing extensibility, and eventually exposing select components externally for ecosystem growth.
The key is to make placement intentional, not just as a reflection of what’s technically feasible, but as a driver of what’s strategically desirable to your org’s vision.
MD101 Support ☎️
If you have any queries about the piece, feel free to connect with the author(s). Or feel free to connect with the MD101 team directly at community@moderndata101.com 🧡
Author Connect 🖋️
Find me on LinkedIn 🙌🏻
Find me on LinkedIn 💬
From The MD101 Team 🧡
Bonus for Sticking With Us to the End!
🧡 The Data Product Playbook
Here’s your own copy of the Actionable Data Product Playbook. With 1500+ downloads so far and quality feedback, we are thrilled with the response to this 6-week guide we’ve built with industry experts and practitioners.
Stay tuned on moderndata101.com for more actionable resources from us!
 | A guest post by
|