MaxDiff design settings

Behind the scenes, the designer chooses items to display across multiple tasks based on these goals: 

• Each item should appear an equal number of times. 
• Each item should appear with each other an equal number of times. 
• Each item should appear an equal number of times in each position (top, middle, bottom) of MaxDiff tasks. 

Depending on your list of items and the number of tasks, achieving perfection in all three goals may not be possible. However, MaxDiff exercises that are "almost but not quite perfect" remain highly efficient and produce excellent results in practice. 

For more information on Discover MaxDiff designs, please refer to the white paper titled "MaxDiff in Discover vs. Lighthouse Studio."

For a typical MaxDiff study where a high degree of precision is desired at the individual respondent level, we have found that each respondent should ideally see each list item 3 or more times throughout the MaxDiff exercise.

For studies with modest to small sample sizes (e.g., N of 50 to 400) and when you wish to drill down by segment to compare preferences across respondent groups, it is valuable to have a relatively high degree of precision in the individual-level MaxDiff scores by showing each item 3 or more times per respondent.

In other studies with relatively large sample sizes (e.g., N of 600 or more) and less need to drill down by segments to analyze differences across respondent groups, you may opt for a design that shows each item around one to two times per respondent.

There are three variables that affect the exercise design, and subsequently the number of times each item is shown per respondent:

1. Total number of list items
2. Number of items per task
3. Number of tasks

MaxDiff exercises are based on lists of items for the respondent to compare typically in terms of importance or preference; and the length of your list (total number of items) is a key factor in determining its design.

MaxDiff is a robust research method that can allow hundreds of list items to be evaluated. MaxDiff in Discover requires a minimum of 6 items to create a viable design.

Though you can use a longer list, we recommend keeping list lengths to about 30 items or less for most applications.  Exceeding 30 items can lead to longer surveys that increase respondent fatigue and reduce score precision. Exceptions could involve situations in which you have relatively large sample sizes, where you plan to show each item around one to two times per respondent, and when you do not need to drill down with a high degree of accuracy at the individual level or to compare differences among subgroups.

We recommend showing three to five items per task for most MaxDiff exercises. Displaying more than five items can lead to fatigue and response errors. Additionally, avoid displaying more than half of the total number of items per set to ensure score precision. For instance, if there are only eight items in your MaxDiff study, you should not show more than four items per set.

The Number of tasks input allows you to determine how many total sets (or tasks) are shown to your respondents.  

This setting is a bit more flexible than items per task, though we highly recommend that you keep it within a reasonable number. Excessive tasks often lead to respondent fatigue and response error. 

There is a tradeoff between the frequency with which each item is viewed per respondent and the precision we achieve in estimating individual utility scores. Displaying each item fewer than two times per respondent can lower the precision of individual-level utility scores. However, you may choose to proceed if you have a large sample size or don’t require high precision at the individual level.

We strongly recommend against displaying more than half the total MaxDiff items in a single task. Doing so reduces the ability to distinguish respondents' middle-preference items.

Displaying more than seven items in a MaxDiff task can increase respondent fatigue and lead to higher error rates. We recommend showing five or fewer items per task for optimal data quality.

To ensure respondents can reliably identify the best and worst items in a set, each task must include at least three items.

You cannot display more items in a task than are present in the MaxDiff list.

The number of tasks cannot exceed the total number of unique item combinations. This prevents the software from repeating identical sets for the same respondent.

The software requires a minimum of one MaxDiff task.

To ensure connectivity—where each item is either directly or indirectly compared to all other items—increase the Number of items per task and/or Number of MaxDiff tasks.

Connectivity is particularly important for individual-level analysis, such as HB estimation (Discover's default method). If you plan to use aggregate methods like aggregate logit or latent class MNL, some loss of connectivity may be acceptable, as connectivity is achieved when pooling responses across participants.