Overview
Many companies are experimenting with generative AI or large language models (LLMs) or using them to deliver services. For example, Märkiting Marketing uses a specially trained Generative Pre-Training Transformer (GPT) that pre-formulates social media posts for them. Nielsen has launched Nielsen IQ, a product that simulates human evaluations of new products. Amidst the technological excitement, questions about quality often get neglected.
The idea of using Large Language Models (LLMs) in marketing emerged shortly after their development (see Qian et al. 2025). One of their main advantages is the ability to generate in silico samples, i.e., produce synthetic data that mimic human responses to questionnaires and interviews, but at a fraction of the cost (Arora et al. 2024). Previous qualitative analyses of LLM results show mixed findings (Sarstedt et al. 2023). Some studies from the marketing literature or related disciplines reported good agreement between synthetic data and human responses (Brand et al. 2023, Li et al. 2023), while others found discrepancies ranging from minor (Goli & Singh 2023, Arora et al. 2024) to severe (Gao et al. 2024). These studies used simple or limited metrics to assess the quality of synthetic samples such as accuracy, mean and variance, and less frequently AUC or Kullback-Leibler divergence.
Companies considering using LLMs thus often have to rely on anecdotal, qualitative evidence to make decisions. This is confusing not only for the companies themselves, but also for their customers, who must trust the providers without knowing the strengths and weaknesses of in silico data for their specific use cases. At the same time, benchmarks play a central role in the development of AI systems (Sculley et al. 2025). As long as a robust method for evaluating performance does not exist, the further development of LLMs for marketing will remain hampered.
Does the above resonate with you? We are organising two round-tables to exchange our insight and discuss best practices with you. Interested? E-mail us to be notified first and secure yourself a participation spot.
