Why Your Data Agents Need a Context Layer

The models are not the problem anymore.

Frontier LLMs are fast, capable, and increasingly cheap. Every enterprise wants agents that answer questions about their data — revenue, churn, pipeline, compliance. The demos look great. The pilots feel promising.

Then the agent hits real data.

"What was revenue growth last quarter?" A question any analyst answers in minutes. The agent hallucinates, contradicts itself, or returns a number no one trusts. Not because the model is weak. Because the agent has no idea what "revenue" means in your organization.

This is the pattern a16z identified in their recent analysis, Your Data Agents Need Context: context is now the bottleneck, not model capability. And the fix isn't better prompting. It's infrastructure.

The Wall Every Team Hits

The failure mode is consistent. An organization connects an LLM to their data warehouse. The agent can query tables. It can generate SQL. It can produce charts. It looks like it works.

Then someone asks a question that requires business context:

• Which revenue definition? Net or gross? Including or excluding the product line that launched in Q4?
• Which data source is the source of truth? The raw tables, the curated views, or the BI tool's cached metrics?
• Which fiscal calendar? The company's fiscal year doesn't match the calendar year.

The agent doesn't know. It can't know. This knowledge lives in people's heads, in Slack threads, in onboarding docs that haven't been updated since the person who wrote them left.

a16z illustrates this with a concrete scenario: an agent tries to answer a revenue question, but the semantic layer YAML files were last updated by a team member who has since left. The definitions are outdated, missing new product lines, and no longer used by the BI tools. The agent has no idea how revenue is actually defined — and it doesn't know that it doesn't know.

MIT's State of AI in Business report put it directly: most AI deployments fail due to "brittle workflows, lack of contextual learning, and misalignment with day-to-day operations." The models work. The context doesn't.

What a Context Layer Actually Is

The concept goes by many names — context OS, context engine, contextual data layer, ontology. The underlying requirement is the same: a structured, maintained, machine-readable layer that captures how an enterprise actually works.

This is more than a semantic layer. A traditional semantic layer defines metrics — revenue is this formula, churn is that formula — using specific syntax like LookML, connected to specific BI tools. It solves metric consistency for dashboards. It doesn't solve context for agents.

A context layer is a superset. It includes:

Canonical entities and identity resolution. "Customer" means different things in different tables. The sales CRM tracks accounts. The billing system tracks contracts. The support platform tracks organizations. The context layer resolves which entities map to which records across systems, creating a unified understanding that no single data source provides on its own.

Metric definitions with temporal validity. Revenue isn't just a formula. It's a formula that changed in Q3 when the new product line launched. The context layer tracks which definition is valid for which time period. An agent querying Q2 data should use the old definition. An agent querying Q4 data should use the new one. Without temporal awareness, the agent applies whichever definition it finds first — and the answer is silently wrong.

Tribal knowledge, made explicit. The fact that "the marketing pipeline numbers exclude partner-sourced leads because that team reports separately" — this is institutional knowledge that lives nowhere in the schema. No column header encodes it. No table description captures it. The context layer captures it as structured rules that agents can consume programmatically.

Governance and access logic. Which data can this agent access? Which decisions require human approval? Which answers need caveats about data freshness? A data agent that returns a confident number from a staging table — because it had access and didn't know the difference — erodes trust faster than one that says "I can't answer this with the data I'm authorized to access."

Source-of-truth mapping. For any given question, which system is authoritative? When sources conflict, which one wins? Enterprise data is duplicated, replicated, and transformed across dozens of systems. Without explicit source-of-truth mapping, the agent picks whichever source it hits first.

Think of it as .cursorrules for your data stack. Just as developers configure rules to guide coding agents, data teams need to configure rules that guide data agents — what to use, what to ignore, how to interpret ambiguity.

The Semantic Layer Isn't Enough

This distinction matters. Teams that assume their existing semantic layer solves the agent context problem will keep failing.

Semantic layers are hand-constructed. They use specific syntax. They connect to specific tools. They define metrics for human analysts who already understand the business context. A human analyst knows that "pipeline" means something different in the sales standup than in the board deck. They carry that context in their heads. They don't need the semantic layer to tell them.

Agents don't have that understanding. An agent doesn't know that "the Q2 numbers in the finance warehouse are preliminary until the third week of the following quarter." An agent doesn't know that "when sales says 'pipeline,' they mean weighted pipeline, but when the board deck says 'pipeline,' it means unweighted." An agent doesn't know that the YAML files defining your semantic model were last updated by someone who left six months ago and no longer reflect how the business actually operates.

A context layer has to be dynamic, self-updating, and comprehensive in ways that traditional semantic layers never needed to be. It needs to evolve as the business evolves — not wait for a data engineer to manually update a config file.

Delivering Context to Agents in Real Time

Building a context layer is necessary but not sufficient. The context also needs to reach the agent at the right moment, in the right format.

This is where protocols like Model Context Protocol (MCP) become critical infrastructure. MCP provides a standardized way to expose structured context to AI agents via APIs — turning the context layer from a static repository into a live service that agents can query on demand.

The pattern is straightforward: the agent receives a question, queries the context layer for relevant business rules, metric definitions, source-of-truth mappings, and governance constraints, then generates its answer within those guardrails. The context layer doesn't replace the model's reasoning — it constrains and informs it.

But the context layer can't be a one-time build. Enterprise data systems are never static. New tables appear. Definitions change. Teams reorganize. The context layer needs self-updating flows — automated processes that detect schema changes, flag outdated definitions, and surface gaps where tribal knowledge hasn't been codified yet. Human input remains essential for the final links, but the system should do the heavy lifting of keeping context current.

Three Emerging Categories in Context Infrastructure

a16z identifies three verticals emerging in this space, each approaching the problem from a different starting point:

Data gravity platforms. These are existing data infrastructure companies — warehouses, lakehouses, catalogs — that don't yet have sophisticated context layer capabilities but hold the data itself. They allow for lightweight semantic modeling and have a feasible path forward through acquisition or in-house development. Their advantage is that the data already lives with them. Their challenge is that context engineering is a fundamentally different discipline than data storage.

AI data analyst companies. The wave of "chat with your data" products that launched in 2024-2025. Many started with a simple architecture: connect LLM to warehouse, let users ask questions in natural language. The best of them realized quickly that the bottleneck wasn't the model or the SQL generation — it was the context. They've evolved to build context layers as their core differentiator, investing heavily in business logic capture, metric governance, and source-of-truth resolution.

Dedicated context layer companies. A new category building context infrastructure from the ground up. They're not constrained by legacy architectures or existing product lines. But they face a cold-start problem: every new customer requires ingesting data, collecting tribal knowledge, and building the context graph from scratch. The depth of deployment required makes this a high-touch, high-value play.

The competitive dynamic across all three categories is the same: whoever owns the structured understanding of how the business works owns the ability to deploy reliable agents.

Why This Is a Context Graph Problem

This is the pattern we keep returning to at The Context Graph: the gap between what an AI system can do and what it reliably does in production is almost always a context problem.

RAG doesn't solve it — retrieval finds similar text, not applicable business logic. A RAG system will happily retrieve an outdated revenue definition because the embeddings match. It has no concept of "this definition was superseded." Fine-tuning doesn't solve it — you can't fine-tune institutional knowledge that changes quarterly. By the time the fine-tuned model is deployed, the business rules it learned may already be obsolete. Prompt engineering doesn't solve it — no prompt template captures the complexity of enterprise data relationships. You can't prompt your way out of not knowing which of three conflicting data sources is authoritative.

What solves it is structured context that is:

• Maintained — not a one-time setup, but a living system that evolves with the business
• Machine-readable — exposed via APIs or MCP so agents can consume it programmatically
• Temporally aware — knows which rules apply now versus which applied last quarter
• Scoped — different agents, different questions, different contexts
• Auditable — when an agent answers a question, you can trace which context rules informed the answer

This is decision infrastructure. Not a smarter model. Not a better prompt. Infrastructure that makes the model's intelligence actually useful against real enterprise complexity.

The Shift Happening Now

The first generation of enterprise AI optimized for model access — getting an LLM connected to your systems. The second generation optimized for retrieval — RAG pipelines, vector databases, embedding strategies. The third generation is optimizing for context — structured, governed, temporally aware business logic that turns raw data into trustworthy answers.

The industry is converging on this realization. The context layer becomes the strategic asset — not the model, not the data warehouse, not the BI tool. Models are commoditizing. Data warehouses are table stakes. But the structured understanding of how your business actually works — which definitions apply, which sources are authoritative, which rules override which — that knowledge is unique to every organization. It cannot be downloaded, fine-tuned, or retrieved from a vector store. It must be built, maintained, and governed as infrastructure.

For teams building production data agents today, the implication is direct: stop investing in better models for bad context. Start investing in better context for good models.

The models are ready. Your context isn't.

---

Patrick Joubert. (2026). "Why Your Data Agents Need a Context Layer." The Context Graph. https://thecontextgraph.co/memos/why-your-data-agents-need-a-context-layer