Jess Wang defines an evaluation through a dataset, task, scoring system and experiment configuration. She explains how production examples and individual failure traces can guide changes more reliably than a few prompts that appear to work.
Jess Wang constructs roughly 20 bug-repair tasks from merged fixes in Microsoft's TypeScript Go repository. Both search conditions use the same Claude Code harness, with tool restrictions to stop the vector-only condition from defaulting to agentic repository exploration.
Jess Wang reports similar accuracy but about four times the cost for vector search in this experiment. Her trace review links repeated retrieval to chunks missing imports and surrounding call relationships, while agentic exploration can follow connected code.
Jess Wang explicitly describes the vector-search implementation as relatively weak and the dataset as small. Repeated trials, stronger retrieval and more repositories would be needed for a robust comparison, so these results are an evaluation lesson rather than a universal verdict on vector search.
Watch on YouTube
