Boosted Bandit: Respondents see a Subset of Items
Very similar to the strategy outlined in the previous section, we can oversample most preferred items within a respondent to improve bandit MaxDiff results (achieving even better results than reported by Fairchild et al. 2015). For example, imagine a situation in which you are studying 100 total items. You decide to use Bandit MaxDiff with Thompson sampling to select only 30 items to show any one respondent. And, you decide to use 18 sets, with 5 items per set. With a standard MaxDiff design, each of those 30 items would appear 3 times across the planned 18 sets. However, we can do a power trick within the bandit sampling scheme to further boost and oversample the very top few items.
•1st through 6th best Thompson-sampled items <show 5x per respondent>
•7th through 30th best Thompson-sampled items <show 2.5x per respondent>
Steps for doing this in Lighthouse Studio are shown below:
1.Create a predefined list that includes a few replicated items to accomplish the oversampling scheme above. Specify a predefined list to use in the MaxDiff exercise that includes 36 total items in the list (that we’ll recode back to 30 items in a later step). The 36 elements in that list are as follows (one row per element):
1
1 <replicate>
2
2 <replicate>
3
3 <replicate>
4
4 <replicate>
5
5 <replicate>
6
6 <replicate>
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
2.Prohibit the replicated items from showing with each other within the same sets (for example, items 1-2, items 3-4, and items 5-6, etc. above). Generate a standard MaxDiff experimental design with multiple versions where the total number of items is 36. For this example, we’ve decided to use 18 sets per respondent showing 5 items per set such that each item appears 2.5x per respondent. The default is 300 versions, but just 10 would work extremely well (hardly any loss of precision due to having 10 versions rather than 300).
3.Export the 36-item design to a .CSV file using the Export… button on the Design tab. Open that .CSV file with Excel and modify it to recode levels 1 and 2 to 1; recode levels 3 and 4 to 2, etc. Now you have recoded all item indices to 30 items, but items 1 through 6 are now represented 2x as many times in the design as were before. The original design included each of the items 2.5x, so the Thompson-sampled best item six items are now included 5x per respondent. (Note that due to the random nature of the Thompson sampling draws, this item can be different between respondents.)
4.Modify the pre-defined list specified in the MaxDiff exercise to have only 30 items. (The software will complain that this will invalidate the previous design. This is OK, since you haven’t fielded the study yet, and you can ignore the warning.)
5.Using the Import… button from the Design tab, import the modified .CSV design file. Run Test Design to make sure the level counts are as expected across all versions in your questionnaire (item 1 should appear 2x as often as level 7, etc.)
6.Create a constructed list using the Bandit MaxDiff constructed list instruction (as described in the Lighthouse Studio documentation). Specify that 30 items should be selected for each respondent.
7.In the MaxDiff exercise, specify that the exercise should use the constructed list created in step 6 rather than the predefined list.
The reason this procedure works is that the Thompson Sampling constructed list instruction selects items to include on the constructed list in priority (best to worst order) according to the draws from the prior preferences. So, the probable best item is assigned to the first list element—and that first list element has been oversampled in your design.
Note: if your goal is to identify the top five items for the sample, we recommend a within-respondent oversampling boost on about the top seven items. If the goal is to identify the top ten items for the sample, we recommend an oversampling boost on about the top twelve items, etc.
