MaxDiff design settings

Behind the scenes, the designer chooses items to display across multiple tasks based on these goals: 

• Each item should appear an equal number of times. 
• Each item should appear with each other an equal number of times. 
• Each item should appear an equal number of times in each position (top, middle, bottom) of MaxDiff tasks. 

Depending on your list of items and the number of tasks, achieving perfection in all three goals may not be possible. However, MaxDiff exercises that are "almost but not quite perfect" remain highly efficient and produce excellent results in practice. 

For more information on Discover MaxDiff designs, please refer to the white paper titled "MaxDiff in Discover vs. Lighthouse Studio."

This is among the most important factors in your MaxDiff design. Thoughtfully adjusting this setting along with other design variables will help you achieve improved results.

For a typical study requiring precise individual-level data, we recommend displaying each item three or more times to each respondent.

• For modest to small sample sizes (e.g., N of 50 to 400), showing each item three or more times improves precision when comparing preferences across respondent groups.
• For larger sample sizes (e.g., N of 600 or more) that require fewer segment-level comparisons, showing each item one to two times is typically sufficient.

This setting varies slightly based on the type of MaxDiff you are using:

• In the Traditional MaxDiff type, this setting is not directly editable. However, you can influence it by changing the number of items per task or the overall number of tasks.
• In the Relevant items MaxDiff type, this setting is an adjustable field that determines the maximum number of times an item can appear for each respondent. In cases in which relatively few items are moved into the respondent’s dynamic list, the default setting is 5 so that the number of MaxDiff questions usually remains the same across respondents.

In Discover, MaxDiff exercises require at least six items to create a viable design. While MaxDiff can handle lists with hundreds of items, we recommend limiting your list to about 30 items or fewer to reduce fatigue and improve precision.

Again, an exception can be made if you have a larger sample size. With less need for segment-level comparisons and a high degree of accuracy at the individual level, showing each item one or two times is typically sufficient.

We recommend showing three to five items per task for most MaxDiff exercises. Displaying more than five items can lead to fatigue and response errors. Additionally, avoid displaying more than half of the total number of items per set to ensure score precision. For instance, if there are only eight items in your MaxDiff study, you should not show more than four items per set.

The Number of tasks input allows you to determine how many total sets (or tasks) are shown to your respondents.  

This setting is a bit more flexible than items per task, though we highly recommend that you keep it within a reasonable number. Excessive tasks often lead to respondent fatigue and response error. 

There is a tradeoff between the frequency with which each item is viewed per respondent and the precision we achieve in estimating individual utility scores. Displaying each item fewer than two times per respondent can lower the precision of individual-level utility scores. However, you may choose to proceed if you have a large sample size or don’t require high precision at the individual level.

We strongly recommend against displaying more than half the total MaxDiff items in a single task. Doing so reduces the ability to distinguish respondents' middle-preference items.

Displaying more than seven items in a MaxDiff task can increase respondent fatigue and lead to higher error rates. We recommend showing five or fewer items per task for optimal data quality.

To ensure respondents can reliably identify the best and worst items in a set, each task must include at least three items.

You cannot display more items in a task than are present in the MaxDiff list.

The number of tasks cannot exceed the total number of unique item combinations. This prevents the software from repeating identical sets for the same respondent.

The software requires a minimum of one MaxDiff task.

To ensure connectivity—where each item is either directly or indirectly compared to all other items—increase the Number of items per task and/or Number of MaxDiff tasks.

Connectivity is particularly important for individual-level analysis, such as HB estimation (Discover's default method). If you plan to use aggregate methods like aggregate logit or latent class MNL, some loss of connectivity may be acceptable, as connectivity is achieved when pooling responses across participants.