Relevant Items (Constructed List) MaxDiff
Overview
Sometimes researchers want to show only the MaxDiff items that are relevant to a given respondent. At the 2018 Sawtooth Software Conference, Bahna and Chapman (from Google Cloud) described such an approach and outlined the benefits.
Lighthouse Studio’s constructed (dynamic) list capabilities are very flexible, allowing a customized list of items for each respondent based on a variety of conditions and logic. Using a constructed (dynamic) list, you can customize the MaxDiff items (potentially a different number of items per respondent) to show respondents based on relevance, awareness, importance, preference, etc.
Analysis can involve logit, latent class, or HB and you need to specify in the analysis interface how to treat missing items for the respondent. There are three methods of treating missing items:
1.Missing items are missing at random and inferred from the population information (default method),
2.Missing items are inferior to included items,
3.Missing items are unavailable and should have near-zero probability of choice for a respondent (we insert a large negative utility such as -20 for the missing utility).
All three methods lead to a full set of utility scores for all the items (for the group, each latent class, or the respondent if using HB estimation). If using HB estimation, you can also export the scores with missing items per respondent replaced with blanks, such that the missing items have missing utility scores.
There are two ways to create the combinations of items to show respondents (the design) for MaxDiff questionnaires: Pre-generated and On-the-fly. These are described in the next sections.
On-The-Fly Design Generation
On-the-fly (during survey) MaxDiff designs contain the combinations of items to show each respondent and are generated in the moment that the respondent takes the survey. On-the-fly designs are required when the number of items in the constructed (dynamic) MaxDiff list is different across respondents for the same MaxDiff exercise. If you are curious about the quality and characteristics of on-the-fly (during survey) designs prior to collecting the data, you could use Lighthouse Studio’s data generator (Test + Generate Data...) to create 100s of random-answering robotic respondents and use your own tools to analyze the resulting designs which are stored in each respondent’s record.
You should take special care that the constructed list logic you use ensures that respondents always get a reasonable number of items in their constructed list. If too many items are included on the constructed list, each item might not be shown enough times for stable HB estimation given the maximum MaxDiff questions you set for your questionnaire. If too few items are included on the constructed list, there may be too few items to populate even one MaxDiff question and the entire MaxDiff exercise would be skipped for the respondent. The ListMin() list building function can help you ensure that your constructed list has a minimum number of items per respondent.
Bahna and Chapman recommended that to obtain more stable utility estimation, it can be helpful to add a few items to the constructed list that are not relevant/important to the respondent. However, given the method we use for augmenting each respondent's choice data with full information when missing items are treated as inferior, this may not be necessary. This is an interesting topic open for additional research and perhaps we'll see a research paper at a future Sawtooth Software conference on the matter.
•On-the-fly (during survey) designs are not supported if using Offline Surveys (CAPI).
•On-the-fly (during survey) designs are only supported if using Sawtooth Software's hosting service. Please contact Sawtooth Software if you are hosting on your own.
Advanced Note: On-the-fly designs may also be used for constructed list MaxDiff exercises where the number of MaxDiff items is fixed across respondents. Examples include Express or Bandit MaxDiff--the principal benefit for these being that prohibitions may then be used (whereas prohibitions cannot be used for Express or Bandit if using Pre-Generated designs).
Pre-Generated Designs
When the number of items to show each respondent is the same (the typical case with MaxDiff), Lighthouse Studio users can generate the combinations of items to show respondents prior to data collection. This is called a Pre-Generated design. The default is to generate 300 versions (blocks) of the design prior to fielding and respondents are assigned to different versions (often times with multiple respondents receiving the same versions, which is just fine from a statistical standpoint).
There are at least two MaxDiff approaches that involve customizing a subset of items to show each respondent that could involve pre-generated designs, because the subset of items always has the same constructed list length: Express MaxDiff and Bandit MaxDiff. For example, 30 items could be drawn from a master list of 100 items for each respondent to evaluate, so a pre-generated design accommodating 30 item slots could be employed. We should note, however, that prohibitions may not be specified if using pre-generated designs for Express MaxDiff or Bandit MaxDiff. Rather, on-the-fly design may be used with Bandit and Express MaxDiff for implementing prohibitions between items.
