Your team trusts the AI. But are its answers right?

The people who ask do not know the answer, so nobody sees the mistakes. We measure how often your AI assistant is right, find the step that breaks, and check again every month.

For AI assistants in pilot or in production. Fixed price, agreed before we start.

Your team asksHow long was the new drug tested on patients?

Your AI assistant18 days. Source: assessment report, page 58.

The page says Day 182. That is about six months, not less than three weeks. When the PDF was read in, “Day 182” was cut into “Day 18” and “2”. The text still looks normal, so nobody sees the error.
Illustration. The broken page is real: an EMA document.

Your AI works in 5 phases before it answers. One of them breaks.

Each phase can break on its own. So we test each one on its own: with vector search, a graph or an agent.

  1. 1

    Collect

    Get the documents
    • Crawl
    • SharePoint
    • Upload

    Pages are missing, or old versions sit next to new ones.

    How we test
    We compare what fits
    Sources, crawl rules and ways to fetch a page
    We measure
    Which pages in scope arrive, and which fetch gives the cleanest text

    No public number yet. We measure it on your data.

  2. 2

    Read

    Turn files into textthis phase broke in the example aboveBreaks in the example
    • PDF
    • Scans
    • Office

    Tables fall apart: “Day 182” becomes “Day 18”.

    How we test
    We compare what fits
    The parsers that fit your documents and your rules: free tools, cloud services or AI models
    We measure
    Blind AI judges: can an AI use this text, and is it a true copy of the page?
    79% vs 19%Usable table pages in our reference test: best parser vs a free oneThe test with 13 parsers →
  3. 3

    Prepare

    Make it searchable
    • Clean
    • Cut
    • Embed
    • or
    • Knowledge graph
    • Ontology

    Page headers become pieces of their own, or a fact is cut off from its heading.

    How we test
    We compare what fits
    Cleaning rules, piece size and where to cut, embedding models, or how the graph is built
    We measure
    Does each known answer sit whole in one piece or node?
    149 → 178Headings found in ten documents, after one cleaning ruleThree examples →
  4. 4

    Search

    Find the right passage
    • Keyword
    • Vector
    • Reranker
    • or
    • Graph search
    • Agent

    The right passage is in the index, but not in the top 10.

    How we test
    We compare what fits
    Keyword, vector, graph or agent; embedding models; rerankers
    We measure
    Recall, MRR and nDCG on your test questions. How they work →
    55% → 70%Right source in the top 10: both searches vs keyword onlyThree search results →
  5. 5

    Answer

    Write the reply
    • Model
    • Prompt
    • Sources

    The right passage was found, but the answer leaves part out or cites the wrong source.

    How we test
    We compare what fits
    AI models, prompts, and how the found text is given to the model
    We measure
    Does every cited source exist and say what the answer claims? Which of two answers is better, judged blind?

    No public number yet. We measure it on your data.

In the example above, phase 2 broke. A better AI model would not help there.

How we measure search1 Found at all?2 Near the top?3 Really useful?How it works →

Client project · energy services company, Germany

AI assistant for employees

About 70,000 pages from 8 sources: process handbooks, laws and industry rules. Answers were stuck at 51%. We measured every step on its own and found the cause in the search. Two fixes lifted the answers.

39%78%Right source in the top 10 search results
51%90%Answers rated helpful by employees
~90 daysWorking time saved in 8 months, estimated. Over 4 months of one person's work.
Search: about 90 test questions with known answers. Ratings: 1,133 from 97 employees. Time saved: 4,380 questions in 8 months × 10 minutes = 730 h, an estimate; 8 h per working day.

We test your AI. Not once, but every month.

Documents change. New models come out. We set up a test bench for your system, and it checks every month that the answers stay good.

How we work: 3 steps
  1. Step 1 · 2 weeks

    Parsing check

    Do your documents reach the AI in one piece?

    You getCorpus report, error rate per document type, the parser that fits
    • The corpus report: what your documents look like, by type
    • The error rate for each document type, with real examples from your files
    • Which parser fits your documents and your rules
    • A clear answer: is reading in your problem, or not?

    Always the first step. Can be booked on its own. Fixed price. We only need read access.

  2. Step 2 · 6–8 weeks

    Setup

    How good is your AI, step by step?

    You getGold set, every step measured, fix plan and audit report
    • A gold set: test questions with proven answers. It is yours.
    • Every step measured on its own: reading, cutting, search, ranking, answer
    • Your system measured against the alternatives you are allowed to use
    • Advice on database, search, reranker and embedding model
    • A fix plan with costs, and an audit report for QA and notified bodies

    Fixed price. The price of step 1 is fully credited.

  3. Optional · after step 2

    Implementation

    We build the fixes from the plan into your system.

    You getFixes built in, hit rate before and after, test bench as a CI gate
    • The fixes from the plan, built into your system
    • The hit rate before and after, against a target agreed up front
    • The test bench, re-run on every change, also as a CI gate

    Fixed price after the audit. You keep running the system.

  4. Step 3 · every month

    Ongoing check

    Does your AI stay good when documents and models change?

    You getTest bench with alerts, new models tested, quarterly report
    • The test bench runs every month and on every change. You get an alert when quality drops.
    • New models tested on your data, with a clear answer: switch or stay.
    • A short quality report every quarter, for QA and auditors.
    • Up to one day a month for questions and small fixes.

    3 months, renewed 3 months at a time. You run the system, we measure and advise.

Your choice
Yours to keep
  • Corpus reporterror rate per document type
  • Gold settest questions with proven answers
  • Test benchre-run on every change, also as a CI gate
  • Audit reportfor QA, notified bodies and the EU AI Act

Runs in your infrastructure, in the EU. No lock-in.

Ask for a 30-minute callThe price depends on the scope. We agree a fixed price before we start.

✓ Fits: system in pilot or in production · pharma, medical devices, energy, finance, insurance

✗ Does not fit: a first demo · you want us to run the system

Benchmarks give pointers. Your data makes the choice.

A public benchmark tests somebody else's documents. New models are often not in it yet. Only a test on yours shows whether a model works on yours.

  1. 40models and parsers on the market
  2. Public benchmarkother documents, mostly English
  3. 4on the shortlist, new models with no benchmark too
  4. Windtunnelyour documents, your questions, your language
  5. 1that fits your documents

The result is not one score but, per stage: how often the right passage is found, what it costs, and how fast it is.

The best tool you are allowed to use

A benchmark winner does not help if your data may not leave the EU, or if only Azure is approved. We start with your rules, then measure what is left, on your documents.

Vector database

turbopuffer, Qdrant, pgvector or Azure AI Search. Chosen by size, cost and where it may run.

Search

Keyword, vector or both, and how to merge the results.

Reranker

Which one, and if it pays for its cost and delay.

Embedding model

Hosted or on your servers, and in which region.

Common questions

What does it cost?

Each step has a fixed price. It depends on the scope: documents, sources and questions. We agree it before we start. The price of step 1 is credited against step 2.

Does our data leave the EU? Do you need write access?

No. The test bench runs in your infrastructure, in the EU. We only use services you approve. For step 1 we only need read access to the documents.

How much time does our team need?

Mostly your subject experts: they check test questions and answers for the gold set. We do the rest. We name the effort for your team in the proposal, before we start.

We already use Ragas, Langfuse or another eval tool. Do we still need you?

Those tools mostly measure the end result. We measure each step on its own and compare alternatives on your documents. So you see which step breaks and what fixes it. We are happy to reuse the test questions you already have.

Does it work with our cloud and our vector database?

Yes. We work in your cloud (Azure, AWS, Google, STACKIT or IONOS) or on-prem, on your own servers. For example with turbopuffer, Qdrant, pgvector or Azure AI Search. We start with your rules and only test what you are allowed to use.

Can we stop after step 2?

Yes. The test bench, the gold set and the reports are yours, and you can keep using them yourself. The ongoing check makes sure someone runs them every month, tests new models, and tells you when quality drops.

What happens after 3 months? Can we cancel?

The ongoing check renews 3 months at a time. You can cancel at the end of any term. The gold set, the test bench and the reports stay with you.

Does it help with the EU AI Act?

It gives you evidence: test questions with proven answers, measurements per step, and an audit report for QA and notified bodies. You need that for documentation and proof. It is not legal advice.

Tim Treskatis

Tim Treskatis

Windtunnel, Berlin. I build and evaluate AI assistants and agents on company knowledge, also under MDR requirements. With hard documents: tables, scans and regulatory filings.

About 8 years in data and ML, 5 of them in enterprise consulting, for clients such as Airbus, Lufthansa, Deutsche Bahn and Hapag-Lloyd.

The name comes from my aerospace degree at KTH Stockholm: there, you test in a wind tunnel before you fly.

Ask for a 30-minute call

No slides. No demo. Five questions on your system, and an honest answer on fit:

  • A score per step?
  • How often is the right passage found?
  • Where is the answer lost?
  • What do your PDFs look like after reading?
  • Can you prove an answer to an auditor?
Ask for a 30-minute call
Is your system in production?

Used only to reply. No newsletter. No tracking. Privacy

Ask for a call