The data team's harness and context matter more than the model

An AI agent that answers questions about your numbers is a language model with layers around it. The data team builds the harness, and the context is managed by those who own the data.

In short: An AI agent that answers questions about an organisation’s numbers is a language model with layers around it: a harness that runs the work, and a context that tells the model what it needs to know, from what the words mean to how the numbers are calculated. The model is the same for everyone, while the harness and the context are what the data team builds. At Data Culture the same model went from 0 to 92 per cent correct answers as the context was put in place. Anthropic’s internal analytics agent fell from around 95 to around 65 per cent in a month, before maintenance became part of the development work. The context is data and has several owners: the domain owns the concepts, the data product owns the definitions, and the use case owns the questions and the answer key the agent is measured against.

An AI agent that answers questions about an organisation’s own numbers consists of more than a language model. Around the model sits a program that runs the work, and a context that tells the model what it needs to know. The context includes two layers of knowledge: one that explains what the words mean, and one that knows how the numbers are calculated. The article first follows one question through all the layers. Then it looks at who owns the context, and how the data team keeps it up to date.

What does an AI agent for data consist of?

The simplest picture is a capable new hire on their first day. The new hire already knows a lot, but nothing about your organisation in particular. The difference is that the agent has no colleague to ask, so everything it needs to know must be written down somewhere it can read.

What happens when someone asks the agent for a number?

The question is “How many active customers did we have in September?”. It looks simple, but “active customer” can mean different things in the same organisation. Winfried Adalbert Etzel works on data governance and data management at Equinor and hosts the MetaDAMA podcast. At Helsinki Data Week in October, he told of a bank where one team counted a customer as active after a purchase in the last 90 days, while another team counted an opened email. Both numbers were calculated correctly, but they answered different questions.

1. The question comes in: The user types the question in a chat window or in Slack.
2. The harness fetches context: It gathers the instructions from the user, the team and the company, and looks up what “active customer” means in your organisation. Both end up in the context window, which is what the model sees.
3. The model interprets and plans: It understands that the question is about the metric active customers for September, and chooses to ask the semantic layer rather than write a query itself.
4. The gate checks access: Before the tool runs, the harness checks that the user and the agent are allowed to see the number.
5. The semantic layer calculates: The tool asks for the metric, and the layer knows which tables, joins and filters give the right number.
6. The verification checks: The harness compares the answer with something it knows, for example a question with a known answer or the number for the previous month. If something looks wrong, the result goes back into the loop as new context.
7. The answer goes out: The user gets the number together with the definition it is based on, and can see what has been counted.

What does each layer do?

Each layer has its own job, and only the model is the same for everyone. The other layers are the ones you build yourselves, or get from a vendor and have to fill with knowledge about the business.

The model

The language model is the same for everyone who uses it. The model has read enormous amounts of text and knows a lot about customers, revenue and orders in general, but it does not know how you define them. A better model writes better SQL, but it cannot know what is not written down anywhere.

The harness

The harness is the program around the model. It runs the work in a loop, gives the model tools, decides what it is allowed to do, remembers between sessions, checks the result and assembles the context before each call. Claude Code and other coding agents are examples of a harness around a language model.

A rule can live in two places. If it is text in the context, for example “never show national identity numbers”, it is advice the model usually follows, but can overlook. If it is code in the harness, as a gate that stops the query, it applies every time. Rules you cannot afford to see broken therefore belong in code.

The context

The context is everything the model gets to see in one call: instructions, documents, definitions, results from tools and the conversation so far. The harness assembles it afresh for each task, so the context itself does not keep. The sources it is drawn from, on the other hand, must be managed over time, and they are of two kinds.

The first kind is the instructions on how the agent should work. The instructions sit on three levels:

• Personal: your own notes, preferences and memory.
• Team: shared conventions, templates and work descriptions.
• Company: policy, access rules and shared guidelines.

The second kind is the knowledge about the data: what the concepts mean, how the metrics are calculated, what the tables and columns contain, and why a number has changed. This knowledge belongs to those who own the data, and it gets its own section further down.

For both kinds, the level decides how strictly the source must be governed. Juha-Pekka Joutsenlahti is Data Catalog Lead and a data advisor at Solita. In a post on LinkedIn, he distinguishes between four levels of knowledge, from personal notes to the company’s shared concepts. He points out that personal notes are stifled by bureaucracy if they are governed as shared knowledge, and that AI acts on things nobody has agreed on if the company’s knowledge is run like personal notes. The more people depend on a source, the stricter the governance it needs.

The ontology and the concept model

An ontology describes the concepts in the business, what they mean and how they relate. The ontology can say, for example, that business customers and private customers are both customers, that only business customers can have sales contracts, and that an order always belongs to a customer. A concept model is the same idea drawn for people, and often the starting point for the ontology. Both describe the business and not the systems, so you can create them before you have looked at a single table.

The ontology helps most where the business uses words differently from everyone else. The model already knows the usual meaning of words such as customer and revenue, so the gain lies in the concepts you use in your own way.

The ontology must also fit the data. In a masterclass at Helsinki Data Week, Mark Freeman II, who wrote the O’Reilly book “Data Contracts” with Chad Sanderson, told of an evaluation where he gave the agent Linked Art, a ready-made ontology for museums, and asked it questions about the database of one particular museum. He expected the answers to improve, but they fell sharply, because the ontology lacked the material wood, which the museum uses. If the agent is to calculate numbers, the ontology must also be linked to the tables. In a study from data.world in 2023, the researchers asked GPT-4 questions about an insurance database without giving the model any examples. It answered 16 per cent of the questions correctly directly against the database, and 54 per cent with an ontology and mappings to the tables.

The semantic layer

The semantic layer says how each metric is calculated: which tables are used, how they are joined, and which filters apply. The layer is bound to the data, and the agent uses it as a tool. Instead of writing SQL from scratch, the agent asks for the metric “active customers” for September, and the layer calculates it the same way every time.

Without descriptions and such a layer the agent has to guess. At the analytics firm Data Culture, an agent filtered on the status value “Available”, which did not exist in the data. No description said that the value “Open” meant available. The query returned zero rows without an error message, and nothing in the answer signalled that anything was wrong. In 2026 dbt Labs asked eleven questions. Answers through dbt’s semantic layer were correct 98.2 per cent of the time, against 90.0 per cent when the model wrote SQL itself against the same tables with the entire schema as context. The layer can only answer what has been modelled, so the accuracy comes at the cost of coverage.

A layer built for reports says how the number is calculated. An agent that is to explain the number also needs to know when the definition changed, and what the number must not be confused with.

Specialised agents

A specialised analytics agent uses the same type of model as all other agents. The difference lies in the harness and the context: tools that run queries against the semantic layer, build and test dbt models and create dashboards as code, and a context with the schema, the metrics and the business rules. Without a glossary, a semantic layer and governance, an analytics agent only moves the guessing from the user to itself.

What do the measurements of the layers show?

Data Culture built an agent for a medical clinic and asked it 13 business questions with known answers. The team added one layer of context at a time and kept the same model throughout. With raw tables and with one modelled table, none of the answers were correct. Column descriptions gave 15 per cent, the business rules raised the share to 77 per cent, and with metrics, example queries and a fixed evaluation the agent ended at 92 per cent. In the last step the team also fixed the evaluation. Around 23 percentage points of the final result came from answers that had been correct all along, but which the old evaluation rejected because of small differences in formatting. The biggest improvement came when the business rules were written down, in other words knowledge that until then had only existed in the analysts’ heads. We have written more about the experiment in the article on metadata and ontology.

Three questions show how far you have come with the layers:

1. Have you written down what the most important concepts mean, and do the people who use them agree?
2. Is each metric defined in one place, with an owner who can correct it?
3. Are the rules you care most about written as code in the harness, or only as text?

The layers around the model decide the answers, and they are built by people who know the business.

Why do the harness and the context matter more than the model?

When Anthropic launched its internal analytics agent, it answered around 95 per cent of the questions in the test set correctly. A month later it answered around 65 per cent correctly. The skills the agent read described a data model that changes every day, and maintaining them was not yet part of the development work.

Anthropic tells this story itself in an article on how its data team lets Claude answer analytics questions from the rest of the company. Without what they call skills, instructions in markdown that the agent fetches when needed, the agent never got above 21 per cent correct answers. With skills it was above 95 per cent, and around 99 in some areas. The agent fell to 65 per cent before the team began to treat maintenance as part of the development work.

The numbers apply to one agent in one company, but they sum up the main point. The team took the agent from 21 to over 95 per cent with context it wrote itself, and without maintenance it fell to 65.

Xavier Gumara Rigol heads the data department at Manychat and writes the newsletter Data as a Product Driver. In his essay on harnesses for data teams he divides the harness into six parts: context injection, skills, MCP servers, hooks, observability and subagents.

Robert Sahlin is on leave from Nordnet while he builds his new company, Supercargo. He has written about how the data platform team at Nordnet tests agents against real business questions on a production warehouse, with two prompts. One is minimal. The other tells the agent where the business terms and column descriptions are, and how to use them. Sahlin calls it his most surprising finding so far: a mid-sized model with the guiding prompt often does better than a larger model with the minimal one. dbt Labs shows that both count. Newer models raised the share of correct answers from text-to-SQL from 32.7 to 64.5 per cent. Once a language model had built three new tables in dbt, the same newer model reached 90 per cent.

For coding agents the picture is mixed. LangChain moved its coding agent from 52.8 to 66.5 per cent of tasks solved on the Terminal Bench 2.0 benchmark without changing the model, and in the HarnessTax study from UC Berkeley the choice of coding tool mattered little for how many tasks were solved, but a lot for what the job cost. Both measurements concern code and not analytics.

For a data team, the difference lies in what the team itself puts around the model, not in which tool it has chosen.

Why is data harder than code for an agent?

Code can be tested in isolation, and an error usually produces an error message. In data, a query can run without errors and still give the wrong number, as the query for “Available” did. Anthropic writes that an analytics question often has one correct answer from one correct source, and that there is no reliable way to prove that the answer is correct. It therefore calls accuracy in analytics a problem of context and control, not of code generation.

Gumara Rigol describes four problems that data has and software does not:

• State: data passes through layers where it is joined, filtered and aggregated. An agent can write code that runs without errors and still stores completely wrong data, because it has misunderstood how the data is collected and transformed.
• Environment: few data teams have a development warehouse with realistic test data. Without a safe place to test, the agent tests as blindly as the team.
• Metadata: what a table or a column means is often only in people’s heads. A field can have the same name as what the user is asking about, and still mean something else. The agent then writes a query that is technically correct and answers the wrong question.
• Quality: a well-written join can still multiply rows and break a primary key further down the flow.

None of the four is solved by a better model today. All four are work the data team already knows.

Why does it not help to give the agent access to everything?

Because the bottleneck is structure, not access. Anthropic tried pointing the agent at all the SQL the team had written, as they describe in the article on the analytics agent. The team gave it access to search thousands of earlier queries from dashboards, transformations and the analysts’ notebooks, and checked in the traces from the runs that it read them. The share of correct answers moved by less than one percentage point.

The answer was in the material for around 80 per cent of the questions the agent got wrong. The agent saw it, but did not use it. According to Anthropic, the agent lacked structure, the link from the question to the right entity in the data. In our assessment, that link is exactly what the ontology and the semantic layer provide. Anthropic also tried letting a language model generate the metric definitions from the tables and the query logs. The result was worse than a smaller set of definitions that people had made. That the machine-made definitions were worse does not mean the agent should be kept away from the documentation. The team’s advice is that Claude writes the documentation, while a person owns the definition of what the metric means.

Manychat builds on the same principle. A skill generates metadata for each table from the SQL, but people own the text, and new knowledge comes in as a pull request that is reviewed like code. A question about marketing campaigns loads only the marketing area. The agent first queries a small set of approved datasets that someone stands behind. Every week it gets a test with questions it has never seen, and the semantic layer is shared with the BI tool. The answer the agent gives includes the sources, how certain the answer is, the SQL and the name of the analyst you can ask.

The agent needs less data, better described, not more of everything.

Who owns which part of the context?

The context has several owners. The instructions on how the agent should work are owned by those who own the harness. The knowledge about the data is owned where the data is owned: in the domain, in the data product and in the use case where the data product is used.

Gumara Rigol drew the harness as rings around the model in a webinar in the Data & AI Product Management Community on 19 October. At the centre sits the model and the tool the vendor has built around it, for example Claude Code. Outside it are three rings that the company, the team and the individual own, and they correspond to the three levels the instructions are drawn from. The outer rings can add, but never take away, so an access rule from the company applies whatever the individual has set up for themselves.

The company’s ring holds the documentation in the repos, the connections to the systems and the access rules. The team’s ring holds the shared skills, a shared knowledge base, metadata for each table and the evaluations. Outermost, the individual has their own CLAUDE.md, their memory and their own skills. A skill can move inwards, and that is how something one keen employee made for themselves becomes team practice.

The rings show who owns the set-up. The knowledge about the data needs one more division, and it follows the domains and data products the data team manages:

The domain and the data product describe the data the same way for everyone who asks, while a use case is about one particular use. Yali Sassoon of Snowplow writes in an essay on the difference between a semantic layer and a context layer that the semantic layer describes the past in the same way for everyone who asks, while the context layer is written for one agent that is to act well in one situation. He declares that Snowplow is building a context layer for customers. Neither the domain nor the data product can own the context for one agent in one situation, because they do not know which decision the answer will be used for.

Joutsenlahti’s rule says how strictly each level must be governed. The concepts in the domain are used by all the data products and all the use cases, so a change there must be approved by the data owner. The descriptions in the data product follow the data model and change in the same pull request. The instructions in a use case can change fastest, by those who use the answers. If several use cases need the same explanation, it moves into the data product or the domain, the way a skill moves from the individual to the team.

Etzel’s bank shows what happens when the boundaries are not settled: two teams counted active customers in their own way, and both numbers were correct. Our advice is that the domain owns the definition. If a use case needs a different definition, it gets its own name in the domain, and no agent gets its own variant in its instructions.

Many data teams get the agent from the platform they already have, with its own semantic layer and its own AI features. Then more of the innermost ring sits with the vendor, but the owners of the context are the same. The platform can draft descriptions and definitions, but the domain owns what the concepts mean, the data product owns how the metrics are calculated, and the use case owns the evaluation set and the answer key the agent is measured against.

If the context only lives in one person’s set-up, the team does not have a harness. It has one person with a good set-up.

How is the context kept up to date?

In the same flow as the data it describes, or it is wrong within weeks. That is how Anthropic’s agent fell from 95 to 65 per cent. The team’s fix was to put the skill files in the same repo as the transformation models. Whoever changes a model updates the documentation in the same pull request, and a check in code review flags when a reporting model changes without a skill file being touched. Around 90 per cent of the pull requests that change their data model now include a change to a skill.

The explanation of why a number changed is the hardest context to keep up to date, because it rarely becomes data. The analytics firm Tasman uses an imagined coffee subscription company as its example in a walkthrough of how to build a context layer. In the example, subscriber churn was 9.4 per cent in March, against an average of 3.1 per cent. The agent got the number right from the semantic layer, but the explanation wrong. The cause was a migration of the payment system that had recorded around 1,900 paused subscriptions as cancelled, and only the analyst knew.

Tasman gives the explanations three fixed places. Events such as migrations, campaigns and outages are kept in a table with the date, the metrics they affect and an owner, so the agent can link them to the numbers. Each version of a definition is kept in a change log with the date it applies from. The reasons are kept in short decision notes in markdown in the same repo as the dbt models, and a note is never edited, but replaced by a new one. The explanations are written in the pull request or after an event, while whoever made the change still remembers why. In the example, it took about an hour spread over four months to write down what the agent needed to explain the churn correctly. The change log belongs to the data product that owns the metric, while the owner of an event is in the table.

Maintenance goes both ways. Anthropic also removes parts of the skills when newer models no longer make the mistakes they were written to prevent. When a new model arrives, the evaluation set should be run again before it is taken into use, because it may answer better or worse on exactly the questions the team gets. If the business changes what an active customer is, the definition in the domain, the calculation in the data product and the evaluation in each use case must change together.

How much time the maintenance takes has not been measured anywhere. Gumara Rigol estimates that he himself spends perhaps 20 per cent of his time on the harness, and that the share varies. His analysts spend a lot of time on context, metadata and evaluation.

In his essay, Gumara Rigol names two techniques that make the job easier. Hooks are checks placed in the path of an action, which the agent cannot skip, in other words rules as code and not as text. If the team has data contracts, it is simple to let hooks enforce them. And a separate subagent should assess whether the SQL answers the question that was asked. Letting the agent that wrote the query assess it itself, he calls cheating. In its latest advice on agent architecture, Anthropic recommends merging routers, planners and specialist agents into one agent that fetches skills when needed, but counts checking the work among the tasks subagents still do well.

There is one thing the harness does not solve. None of the six parts gives the agent a safe warehouse to test in, with fresh and realistic data. Such a warehouse must be built as a separate investment.

An agent project that only prices in the build prices in the first 95 per cent and not the months after.

Where should a data team start?

With one domain, one data product and one use case, and one named owner for each. Write the definitions for the most used tables in the data product, give them an owner, and put the skill file in the same repo as the models, with a check in code review. Write down the questions the use case is to answer, with known answers for a few dozen of them. Anthropic recommends a similar start: a handful of approved datasets, a few dozen evaluations and a thin skill that points onwards give most of the gain. Run the evaluations every week, so you can see when the documentation starts to lag behind the data model.

How far the team has to go depends on how it works today. Teams that already run data as a product, with owners, documentation and data contracts, have come a long way. What they need most is to surround the agent with practice they already have. Teams that still work as a support function delivering reports on request must build the product discipline and the harness at the same time. In our assessment, the fastest move for them is to write down what today only exists in people’s heads, the way the business rules made the difference at Data Culture. We have written about that work in the value of metadata and in the guide to data products.

Five questions from Gumara Rigol show whether a data team is ready:

1. Does the agent find the right data?
2. Do metadata and definitions sit next to each dataset?
3. Are there access rules the agent follows?
4. Are there tests and evaluations?
5. Is there a safe place to try things out, with a sample of the data?

We would add two, which the Anthropic numbers and the ownership of the context make necessary. Who updates the skill file the next time someone changes a model in dbt? And who owns the answer key the agent is measured against in each use case? If the answer is no one, you have a demo, not an analytics agent.

Frequently asked questions about harness and context for data teams

What is a harness for an AI agent? A harness is the program around the model: the context it gets, the skills it can fetch, the tools it connects to, the checks that always run, the logs and the subagents that check the work. For a data team it consists mostly of metadata, definitions, evaluations and the ownership of them.

What is the difference between the harness and the context? The context is what the model is told before it answers: instructions, definitions, descriptions of the data and results from tools. The harness is the whole set-up around the model: it assembles the context, gives the model skills and tools, runs checks and lets subagents check the work.

Who owns the context of an AI agent for data? The context is owned where the data is owned. The domain owns the concepts and the business rules, the data product owns the descriptions, the metrics and the change log, and the use case owns the questions the agent is to answer, the answer key it is measured against and the instructions for that particular use. The access rules are owned by the company and apply regardless.

What is the difference between an ontology and a semantic layer? The ontology describes what the concepts in the business mean and how they relate, independently of the systems. The semantic layer says how each metric is calculated in the tables. The agent reads the ontology as context, and calls the semantic layer as a tool when it needs the number.

Does it help to switch to a better language model? Yes, but the model alone only gets you halfway. dbt Labs measured that newer models raised the share of correct answers from text-to-SQL from 32.7 to 64.5 per cent. The same model only reached 90 per cent once the data had been modelled in three new tables in dbt. Robert Sahlin writes that a mid-sized model with good guidance often does better than a larger model without it.

How much time does it take to maintain the harness and the context? There is no measured share. Xavier Gumara Rigol, who heads the data department at Manychat, estimates that he spends perhaps 20 per cent of his own time on it. Anthropic saw the share of correct answers fall from around 95 to around 65 per cent in a month, before the team put the documentation in the same repo as the data model.

Sources

• Anthropic: How Anthropic enables self-service data analytics with Claude, 3 June 2026. Figure 3 and the percentage in figure 5 are based on the figures from this source.
• Anthropic: Building agents: When to fold subagents into your main agent loop, 22 October 2026.
• Opeyemi Fabiyi: The Boring Work That Makes AI Analytics Actually Work, Data Culture, 28 April 2026. Figure 2 is based on the figures from this source.
• Jason Ganz and Benoit Perigaud: Semantic Layer vs. Text-to-SQL: 2026 Benchmark Update, dbt Labs, 7 April 2026.
• Juan Sequeda, Dean Allemang and Bryon Jacob: A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model’s Accuracy for Question Answering on Enterprise SQL Databases, arXiv, November 2023.
• Winfried Adalbert Etzel: From Handover to Hangover, talk at Helsinki Data Week, 27 October 2026.
• Mark Freeman II: masterclass on data contracts and semantics at Helsinki Data Week, 26 October 2026, with the workshop repo.
• Juha-Pekka Joutsenlahti: post on four levels of knowledge for AI, LinkedIn, 8 October 2026.
• Xavier Gumara Rigol: Harness Engineering for Data Teams, Data as a Product Driver, 22 September 2026.
• Xavier Gumara Rigol: Building an AI Harness for Your Data Team, webinar in the Data & AI Product Management Community, 19 October 2026, members only. Figure 4 is based on the model from this source.
• Robert Sahlin: How do you know if the analytics agent is right?, Data Platform Engineering, 29 August 2026.
• Yali Sassoon: A semantic layer is not a context layer, Data Creation, July 2026.
• Tasman Analytics: How to Build a Context Layer (Because You Cannot Buy One), July 2026.
• Vivek Trivedy: Improving Deep Agents with harness engineering, LangChain, 17 February 2026.
• Pan, Yang, Arabzadeh, Chiang, Stoica and Zaharia: HarnessTax, UC Berkeley, September 2026.