Anuj Iravane describes the challenge of evaluating healthcare agents when contracts prevent retaining customer medical records or derivative copies. Long, varied documents contain rare scenarios that small test sets can miss, while asking a language model to generate an entire record at once can produce repetitive or inconsistent examples.
Anuj Iravane reverses the ordinary decision workflow: start with an outcome, sample a compatible reasoning path from a symbolic policy representation, and generate a record conditioned on that path. Decision-tree structure provides a way to cover different scenarios deliberately instead of relying entirely on a language model to invent diversity.
Anuj Iravane builds records from coarse plans to detailed documents. The pipeline establishes stable patient attributes, creates a sequence of encounters, plans documents for each encounter and generates their contents. A refinement stage checks consistency across documents and tests whether the generated record supports the intended outcome. Keeping generation in text and Markdown helps manage context size as patient histories grow.
Anuj Iravane gives clinicians two forms of control: steering individual generation stages and editing the skills that define the workflow. He reports that roughly 90% of the relevant datasets are synthetic and that clinicians distinguished synthetic from real records about 60% of the time in a blind review. These reported results support further development, while the talk does not supply enough detail to establish comprehensive clinical validity.
Anuj Iravane recommends using an appropriate scenario distribution, modeling how source documents are created and letting domain experts own quality decisions. The approach is aimed at creating evaluation cases before deployment, especially when useful data is sensitive, short-lived or expensive to label.
Watch on YouTube
