Add document context to chunks before indexing

Generate a short contextual description for each chunk and embed it with the original text to improve retrieval of context-poor passages.

Goal

Make indexed chunks retrievable for domain questions even when separated from their full document.

Retain each chunk’s heading, text, and document-level context as in the contextual-retrieval notebook. [contextual]
Ask Claude for a context description per chunk, then embed context + chunk with Voyage into the vector index. [contextual]
Compare retrieval against the chunk-only index on the same gold questions and record context/embedding versions. [contextual]

The contextual index improves retrieval for at least one low-recall case while preserving original-source links and reproducibility.

If a description adds unsupported facts, reject that enrichment and use a short description grounded in the source document.

Python, Anthropic/Voyage API keys, authorized corpus, and an existing baseline RAG index.

Send only authorized chunks to the model; do not reuse sensitive descriptions across documents.

Generated descriptions add indexing cost and may introduce bias; retain original chunks and provenance—summaries do not replace source text.

These values describe the upstream repository, not popularity or likes for an individual skill or playbook.

No metrics snapshot is available yet.

License: MIT