When designing a MaxDiff survey, you need to decide how many questions to ask each respondent, and how many of your items you show per question. While there isn’t exactly a “right” or “wrong” way to choose these values, we have done a lot of research on the topic and have some suggestions. The calculator below provides a quick recommendation.
Number of items
Number of items per question
Recommended questions
Where does this recommendation come from?
At Sawtooth, we’ve been helping customers collect and analyze choice data for over 40 years. We’re passionate about high quality research, so we’ve conducted numerous research projects over the years. These involve both real respondents and assigning known preferences to robotic respondents to see how well we rediscover their true preferences. You can read more in our free library of white papers.
The general recommendations from this research are
- Ask enough questions to show each item 3 times to a respondent
- Keep the number of items per question below half of your total items
When we construct MaxDiff questions, we don’t take the random approach, but instead very carefully design questions with the following goals:
- Show each item equally
- Show each item against every other item equally (minimize correlation)
- Show each item in each row equally (minimize order bias)
- Distribute items equally across the questions
By creating these types of high-quality question sets, we build questions that ensure all items are efficiently linked together to support the calculation of item scores no matter how many times an item is shown.
What about sample size?
Sample size is an interesting question for MaxDiff surveys. If you are showing each item 3 or more times to respondents and using a high-quality design, the scores you can calculate are quite robust and stable for tiny sample sizes even using a simple counting approach. Try building your own MaxDiff exercise to help you decide on your next vacation or restaurant decision for free with Discover, our easy to use survey platform.
Most MaxDiff surveys will have at least 200 respondents or 200 respondents per sub-group if you want to compare item preferences across groups. If you want to get a little more formal, you can read another article about sample size and power analysis for MaxDiff .
What happens if I go lower than the recommendation?
Like most things in statistics, you don’t go from “correct” to “incorrect” if you lower your sample size or the number of data points you have to work with. Recommendations for MaxDiff questions can be thought of as a spectrum. One on side, you favor robust, individual-level analysis and want to make sure you allow for reinforcement of preferences and differentiation in the scores. This would lead you to hit the target of showing each item 3 times (or more). On the other end of the spectrum, you are less worried about individual results and more concerned with identifying the top few items (or making sure to avoid the worst). You might show each item less than 3 times on average. Research has shown that sparse MaxDiff exercises can still do a great job at helping you understand which items perform the best.
Bandit MaxDiff is also an extremely effective approach when your have a large list of items and are mostly concerned with identifying the best.
Want to learn more?
You can read a quick guide to the most popular variations of MaxDiff, such as Anchored, Bandit, and Sparse, how MaxDiff can help you identify random and low-quality respondents in your surveys, or how Blizzard Entertainment uses MaxDiff for their Diablo franchise .