Paranjape and Chitlur Haridas describe DoorDash's GenAI Platform team as a horizontal group that helps product teams balance accuracy, latency and cost. Its shared primitives include LLM and agent gateways, open-weight model hosting and a common evaluation platform.
Their central lesson is that AI quality depends on domain knowledge held across strategy and operations, product, labeling partners and engineering. The platform therefore began with approachable user interfaces, expanded into reusable APIs and now supports workflow-first access so each group can contribute without waiting on the central team.
The continuous loop begins with production traces and sessionsAn AI execution trace is a structured record of the steps, tool calls, state changes, and outputs produced during an AI workflow., samples them into manageable review sets, adds domain-specific annotationsData annotation adds labels, judgments or structured metadata to examples so they can be used for training, evaluation or analysis. and turns the reviewed examples into golden datasetsA golden dataset is a curated collection of examples with trusted labels or expected outcomes used as a stable reference for evaluation.. Teams then calibrate LLM-as-judge prompts against those datasetsLarge language model judge calibration adjusts and validates a model-based evaluator against trusted human-reviewed examples so its scores reflect the intended quality standard., inspect the changes and monitor the resulting scores over time.
A telemetry surface exposes traces, scores and observationsAI agent observability makes an agent's state, actions, tool use, failures, resource use, and outcomes visible enough to understand and operate it. through APIs, SDKs and MCP, while a workflow surface supports annotation, dataset review and judge calibration. Self-serve annotation tools and transparent prompt comparisons reduced annotation costs, shortened feedback cycles and let different teams choose who owns each quality decision.
Watch on YouTube
