SimulationMaxxing: How we ship agents 20× faster - Aman Gupta (Nubank) + Shreya Rajpal (Snowglobe)

AI Engineer16m 29s
Aman Gupta and Shreya Rajpal in blue shirts around the headline “SIMULATE AGENTS. SHIP FASTER.” on a black background.
0 comments · 0 votesOpen discussionClose discussion
Sign in to join the discussion

    Aman Gupta and Shreya Rajpal present a support-agent evaluationEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria. workflow built around simulated customer conversations. They argue that multi-turn agent tracesAn execution trace records the sequence of operations and results produced while a system carries out a task. with tool state are slow to author manually, while relying only on production traces limits how quickly teams can test changes without involving live customers.

    The simulation method creates customer personasA synthetic persona is a simulated person or audience profile used to generate possible responses, often with a language model. Its answers are hypotheses to test against human evidence., consistent synthetic account detailsSynthetic data is information generated to represent examples or situations rather than collected directly from the events it describes. and multi-turn conversations against a real agent with mocked tools. The resulting traces can be scored with evaluation metrics and used to compare prompts, tools, model choices and agent harness changes before a production experiment.

    Aman Gupta reports that Nubank uses this process in an improvement loopAn AI feedback loop occurs when AI outputs or their consequences become inputs that influence later model or agent behavior.: observe agent behavior, generate evaluation data, optimize, catch regressions, and advance the changes that pass. He reports faster iteration and improved customer-service outcomes, including human review indicating many simulated cases were usable; these company results have not been independently verified here.

    Shreya Rajpal stresses that simulated data needs a measured link to real outcomes. The closing guidance calls for human review and offline-to-online comparisons to test whether the simulations reflect production behavior, alongside metrics that match the service qualities a team actually wants to improve.

    Original YouTube thumbnailWatch on YouTube

    Video topic coverage

    • Agents & workflows100
    • Coding & tools50
    • Models & learning25
    • Infrastructure & compute10
    • Research & evaluation100
    • Safety & security10
    • Business & productivity75
    • Creative & media0
    • Robotics & perception0
    • Society & policy0

    Share this video

    Aman Gupta and Shreya Rajpal in blue shirts around the headline “SIMULATE AGENTS. SHIP FASTER.” on a black background. With WWW.ARTIFICIAL-INTELLIGENCE.VIDEO, 29 July 2026 and duration 16m 29s.

    Aman Gupta and Shreya Rajpal describe using simulated customer conversations and aligned evaluations to test support agents faster, while checking simulation results against real interactions.