Boosted Bandit: All Items Shown to Every Respondent
Bandit MaxDiff may be used advantageously even for MaxDiff problems involving very few items. For example, the experiment could involve just twelve items where all respondents see all twelve items in their MaxDiff questionnaires and the prior most preferred items (as judged by previous respondents) are oversampled in the design. We describe how this could be done using Lighthouse Studio below.
Recall that Thompson sampling is used to select the items to show the respondent based on the prior means and variances. For the purposes of studies involving very few items, all items are selected for each respondent, but Bandit MaxDiff ensures that the most preferred items are oversampled. The key to making this happen hinges on the fact that the constructed list command that implements Thompson sampling in Lighthouse Studio sorts the draws for the items from best to worst.
First, decide to what degree you would like to oversample the most preferred items. A reasonable scheme involving 12 items is to show the top three (Thompson-drawn) items for each respondent 2x as many times as the 9th through 12th most preferred (Thompson-drawn) items.
Below, we describe the steps in Lighthouse Studio for implementing a within-respondent Boosted Bandit MaxDiff study involving just 12 items:
1.Create a predefined list that includes a few replicated items to accomplish the oversampling scheme above. Specify a predefined list to use in the MaxDiff exercise that includes 15 total items in the list (that we’ll recode back to the original 12 items in a later step). The 15 elements in that list are as follows (one row per element):
1
1 <replicate>
2
2 <replicate>
3
3 <replicate>
4
5
6
7
8
9
10
11
12
2.Prohibit the replicated items from showing with each other within the same sets (for example, items 1-2, items 3-4, and items 5-6 above). Generate a standard MaxDiff experimental design with multiple versions where the total number of items is 15. Following typical practice, you might decide to use 10 sets per respondent showing 4 items per set such that each item appears on average 2.67x per respondent. The default is 300 versions, but just 10 would work extremely well (hardly any loss of precision due to having 10 versions rather than 300).
3.Export the 15-item design to a .CSV file using the Export… button on the Design tab. Open that .CSV file with Excel and modify it to recode levels 1 and 2 to 1; recode levels 3 and 4 to 2, etc. Now you have recoded all item indices to the original 12 items, but items 1, 2 and 3 are now represented 2x as many times in the design as before. In the new design, the top 3 Thompson-drawn items will now appear on average 5.33 times per respondent across the 10 tasks. The bottom 9 Thompson-drawn items will now appear on average 2.67 times per respondent across the 10 tasks.
4.Modify the pre-defined list specified in the MaxDiff exercise to have only 12 items. (The software will complain that this will invalidate the previous design. This is OK, since you haven’t fielded the study yet, and you can ignore the warning.)
5.Using the Import… button from the Design tab, import the modified .CSV design file. Run Test Design to make sure the level counts are as expected across all versions in your questionnaire (item 1 should appear 2x as often as level 12, etc.)
6.Create a constructed list using the Bandit MaxDiff constructed list instruction (as described in the Lighthouse Studio documentation). Specify that all 12 items should be selected for each respondent.
7.In the MaxDiff exercise, specify that the exercise should use the constructed list created in step 6 rather than the predefined list.
The reason this procedure works is that the Thompson Sampling constructed list instruction selects items to include on the constructed list in priority (best to worst order) according to the draws from the prior preferences. So, the probable best item is assigned to the first list element—and that first list element has been oversampled in your design.
Note: if an item appears in too many of a respondent’s MaxDiff tasks, this could become distracting or annoying. For example, the same item appearing in 2/3 or more of the MaxDiff tasks should probably be avoided.
