Synthetic personas use language models to simulate responses for market research and product concepts. The talk compares them to weather forecasts: useful within a tested range, but unreliable when pushed beyond the conditions where they have been validated.
Three failure modes matter. Missing context can make a model invent hidden assumptions, such as treating a higher price as a sign of a better product. Reordered answer choices can change responses sharply. Models can also predict stated attitudes more readily than real-world behavior, so those outputs should not be treated as interchangeable.
The proposed methods include grounding prompts in human data, testing wording and order changes, fine-tuning against known response distributions, and mapping free-text answers to human-calibrated rating scales. Evaluation needs both correlation and distribution-shape measures, plus an estimate of inconsistency in the human reference data. Repeating the same synthetic sample does not create independent evidence or increase statistical significance.
The intended role is to extend existing human research to later questions and simulations, while keeping real human observations as the validation anchor.
Watch on YouTube
