Synthetic personasA synthetic persona is a simulated person or audience profile used to generate possible responses, often with a language model. Its answers are hypotheses to test against human evidence. use language models to simulate responses for market research and product concepts. The talk compares them to weather forecasts: useful within a tested range, but unreliable when pushed beyond the conditions where they have been validated.
Three failure modes matter. Missing context can make a model invent hidden assumptionsA hallucination is an AI output that presents false, unsupported, or invented information as though it were reliable., such as treating a higher price as a sign of a better product. Reordered answer choices can change responses sharply. Models can also predict stated attitudes more readily than real-world behavior, so those outputs should not be treated as interchangeable.
The proposed methods include grounding prompts in human dataGrounded generation produces an answer from supplied, retrievable, and attributable evidence rather than relying only on a model's learned patterns., testing wording and order changes, fine-tuningFine-tuning adapts a pretrained model by updating its parameters on examples chosen for a particular task or subject. against known response distributions, and mapping free-text answers to human-calibrated rating scales. EvaluationEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria. needs both correlation and distribution-shape measures, plus an estimate of inconsistency in the human reference data. Repeating the same synthetic sample does not create independent evidence or increase statistical significance.
The intended role is to extend existing human research to later questions and simulations, while keeping real human observations as the validation anchor.
Watch on YouTube




