RAG

As of v4, KERN has a first-class RAG node family. Your corpus, embedding contract, retriever, grounding policy, and evaluation cases are part of the program — typed, validated, and testable — instead of configuration scattered across notebooks. Citations, grounding, and eval thresholds become enforceable contracts.

What first-class RAG means in KERN

KERN models RAG as declarative contracts plus a deterministic in-memory reference runtime. You declare the pipeline shape and its grounding and eval requirements as nodes; the core validates and evaluates them without binding to a provider. The retrieve node, for example, declaratively binds a tool to a retriever without executing provider retrieval in core. KERN does not ship vector-database integrations or live embedding-provider execution — provider execution is adapter-owned. What it gives you is a typed, reviewable, testable description of the retrieval and grounding contract your program depends on.

The node family

The family nests from a corpus down to per-case assertions. Here is the full shape in one example.

corpus name=Docs title="Support docs"
  source name=manuals kind=local uri="./docs/**/*.md" media=markdown
  chunking source=manuals strategy=semantic maxTokens=600 overlap=80 unit=tokens

embed name=DocsEmbedding corpus=Docs model=text-embedding-3-small dims=1536 metric=cosine

retriever name=DocsSearch corpus=Docs embed=DocsEmbedding mode=hybrid topK=8 minScore=0.72

rag name=AnswerDocs retriever=DocsSearch prompt="./answer.md" citations=true
  grounding requireCitations=true policy=strict maxContext=6000
  ragEval name=Faithfulness metric=faithfulness threshold=0.85 mode=contract
    ragCase name=refunds query="How do refunds work?" tags="smoke,policy" topK=4 minScore=0.72 chunkCount=1 sources="docs/refunds.md"
      ragAssert kind=sourceGlob value="docs/refunds.md" required=true
      ragAssert kind=citesRequired
  ragAnswerContract name=RefundAnswer query="How do refunds work?" answer="Refunds follow the refund policy." requireCitations=true minGroundingCoverage=0.8
    answerSpan start=0 end=33 chunks=refunds required=true

corpus, source, chunking

A corpus names a document collection and its source and chunking contract without binding to a provider runtime. Its source children name raw document locations (local files, S3, HTTP, or an MCP resource), and chunking describes segmentation — strategy, max tokens, and overlap. (chunking is named to avoid colliding with the collection chunk primitive.)

embed

An embed node names the embedding model and dimension contract for a corpus — model, dims, and similarity metric. Provider execution is adapter-owned; the node declares the contract, not the call.

retriever

A retriever binds a corpus and an optional embedding contract to a search policy: mode, topK, and minScore.

rag and grounding

A rag pipeline connects a query/answer flow to a retriever and to grounding and evaluation requirements. Its grounding child declares citation and context constraints — for example requireCitations, policy, and maxContext — as part of the program.

ragEval, ragCase, ragAssert

A ragEval declares an evaluation metric and threshold and holds ragCase children. Each ragCase is a query plus expectations and holds ragAssert children — closed, static checks over the retrieved chunks or grounding. The assertion kinds are a fixed, closed set (see the table below).

ragAnswerContract and answerSpan

A ragAnswerContract declares a provider-free answer grounding shape evaluated by the core RAG runtime — query, answer, citation requirement, and a minimum grounding coverage. Its answerSpan children map answer-text character ranges to the retrieved chunk ids that ground them.

Assertion kinds

ragAssert kind= accepts a closed set of 11 static checks evaluated against the retrieved chunks, grounding, or latency:

The deterministic reference runtime

KERN core includes an in-memory reference runtime that is the substrate the eval contracts run against. It is deterministic by design, and that determinism is covered by the core test suite. Guarantees include:

• Lexical ranking — exact lexical matches rank first.
• topK limiting and minScore filtering — results are bounded and thresholded.
• Descending-score ordering with deterministic tie-breaks by chunk id.
• Citation and provenance preservation, returned as defensive copies so corpus reads cannot be mutated by callers.
• Stable upserts — replacing a chunk by id does not change retrieval determinism.
• Unicode tokenization for non-English retrieval.

This runtime is the reference and evaluation substrate — it is how a ragEval contract gets a deterministic, repeatable result. It is not a production retrieval service; provider-backed retrieval is adapter-owned.

retrieve — bind a RAG retriever to an MCP tool

The retrieve node is an MCP retrieval intent: it declaratively binds a tool or prompt to a RAG retriever or pipeline — without executing provider retrieval in core — and maps the citation, source, and score fields so the tool returns provenance-carrying results.

tool name=AnswerDocs
  param name=question type=string
  retrieve rag=AnswerDocs queryParam=question as=context output="RetrievedChunk[]" requireCitations=true provenance=source citationField=citation sourceField=uri scoreField=score

Validation and parity

The RAG node family, the closed assertion kinds, the deterministic runtime, and the eval and answer-contract conformance are covered by the core test suite. Assertion kinds, value shapes, and grounding requirements are validated statically. See kern check for the semantic type checker that validates KERN programs.