Synthetic Survey Data? It's Not Data

Many vendors claim that LLM "AI" systems can generate human-like survey responses quickly and cheaply using previous or sometimes concurrent training data. In this webinar, Chris discusses several fundamental problems that he believes should disqualify any such consideration. First, he proposes that there is no way to prove in advance that LLM data will be "good enough" for any particular question. Second, there is no way to compare LLM data to human data because the fundamental concepts of statistics do not apply to samples from LLM models. Third, he argues that the premise of synthetic data rests on a faulty assumption that the goal of survey research is to obtain an absolute "true answer" to a question. He concludes that, instead of searching for a nebulous true value, the purpose of survey research is to learn from people in real time.

Chris Chapman PhD is the director of the Quant UX Association, after 24 years conducting research at Google (2012-2022), Microsoft, and Amazon. He is the co-author of several books including Chapman & Rodden, Quantitative User Experience Research (Apress, 2023) and the Quant UX Blog, https://quantuxblog.com.