RAG against the machine

RAG against the machine is a milestone M4 Python project about retrieval-augmented generation. You index a real codebase (the source and documentation of vLLM, a pinned version), retrieve the passages most relevant to a question, and use Qwen3-0.6B to answer it while citing exactly where the answer comes from: file path and character range. The grade depends on measured retrieval quality, so this is the core's first project where you iterate on a metric, not on a feature list. Details below come from student repositories (Reported by students); check them against your subject.

What you will learn

• The RAG pipeline: chunking, indexing, retrieval, reranking, generation with context
• Lexical retrieval with BM25 and TF-IDF, and why it struggles on code
• Dense retrieval with sentence embeddings, and hybrid fusion with reciprocal rank fusion
• Chunking source code by syntax (functions and classes with the ast module) and Markdown by headings
• Evaluating retrieval with recall at k on a labelled dataset
• Running a small LLM on CPU within memory and time budgets
• Building a command line tool with subcommands

Before you start

• Previous nodes: Call Me Maybe (working with Qwen3-0.6B and its tokenizer) and the Python Modules.
• Tools: Python 3.10+ (one repo reports 3.13), uv, flake8, mypy, and a BM25 implementation (the bm25s package or your own).
• Expect around 2.5 GB of RAM for the model in full precision on CPU.
• Reported datasets: public question sets on the documentation and on the code, around a hundred questions each, with the expected source ranges.

Plan of attack

1. Day 1: clone the pinned vLLM version, walk the tree, and list the file types you will index. Ignore tests, binaries and generated files unless the questions need them.
2. Day 1: chunking. Python files by function and class with ast, Markdown by heading, and a sliding window with overlap for anything longer than your maximum chunk size. Keep exact character offsets for every chunk.
3. Day 2: BM25 index with tokenisation that splits snake_case and camelCase identifiers. Save the index to disk.
4. Day 2: CLI subcommands (reported: index, search, search_dataset, answer, answer_dataset, evaluate) with a --k option.
5. Day 3: evaluation: recall at 1, 3, 5 and 10 on both public sets. Write the numbers down after every change.
6. Day 3: improve retrieval: tune chunk size, add file path tokens, try dense embeddings and hybrid fusion if the subject allows them (reported as bonus).
7. Day 4: generation: a prompt with the top chunks and the question, thinking mode off, a short answer with its sources.
8. Day 5: performance (indexing time, batch retrieval time), lint, README with the AI usage section and your recall table.

def reciprocal_rank_fusion(rankings: list[list[str]], k: int = 60) -> list[str]:
    scores: dict[str, float] = {}
    for ranking in rankings:
        for rank, chunk_id in enumerate(ranking):
            scores[chunk_id] = scores.get(chunk_id, 0.0) + 1.0 / (k + rank + 1)
    return sorted(scores, key=scores.__getitem__, reverse=True)

Common pitfalls

• Off-by-one or byte versus character offsets in the cited ranges: the grader compares positions.
• Lexical retrieval missing code questions that share no words with the code; identifier splitting helps a lot.
• Re-indexing at every query instead of loading a saved index.
• Out of memory on CPU, or minutes per answer because the context is too long.
• Indexing the wrong vLLM version.
• Untyped numpy or library code failing mypy --strict.

How it is evaluated

• Peer evaluation plus an automatic grader on the JSON output of the dataset commands (Reported by students).
• Reported targets: about 80% recall at 5 on the documentation set and 50% on the code set, indexing in a few minutes, and batch retrieval of a large question set within a time limit.
• The evaluator runs the linters, indexes from scratch, runs the evaluation, and asks you to explain chunking, BM25 scoring and your fusion strategy.
• Dense and hybrid retrieval are reported as bonus.

Resources

• Okapi BM25 on Wikipedia
• bm25s on GitHub
• Python ast module
• vLLM repository
• Qwen3-0.6B model card
• Student repo: Sabofxx rag-against-the-machine