Programming and Fielding a Bandit MaxDiff Study
1. First, create a predefined list that contains all the items in your Bandit MaxDiff exercise (say, 80 items if you have 80 total items in your study). To do this, open the Lists manager by clicking the Lists icon on the toolbar. Click Add Predefined List and specify a new list name for this list. Type or paste all the items for your Bandit MaxDiff exercise (or specify graphics for these items if you will be showing graphics).
2. While in the List manager, click Add Constructed List and specify a new list name. In the Parent list drop-down field, select the list of items you previously specified above in Step 1. In the Constructed List Instructions field, specify the following two lines:
BanditMaxDiff (MaxDiffExerciseName, Items)
SETLISTLENGTH (Items)
where instead of MaxDiffExerciseName, you specify the name of a MaxDiff Exercise that you will be creating in Step 3 (if you click the Check for Errors button, it will warn you that no MaxDiff exercise exists yet with that name, which is OK for now).
Instead of Items, you specify the number of items to show each respondent (assuming 30 or more items in your Bandit MaxDiff study, Items should typically be 30). More information about Items.
By default, a moderately aggressive level of adaptivity is employed when selecting which items to show each new respondent completing the questionnaire. You can change how aggressively to favor previously preferred items for Bandit MaxDiff by using the optional NumThompsonItems argument.
BanditMaxDiff (MaxDiffExerciseName, Items, NumThompsonItems)
More information about the NumThompsonItems argument.
3. Next, create your MaxDiff exercise. From the Write Questionnaire dialog, add a MaxDiff exercise to your Lighthouse Studio project with the same name as you specified for MaxDiffExercise in Step 2. On the Items tab, change the Existing List to select the constructed list you specified in Step 2.
Click the Design tab. For the Design Generation approach, choose the Pre-Generated design. Specify the number of Items (Attributes) with the same value you specified in Items in Step 2 (typically 30). Continuing with the Design tab, design the questionnaire just as you would for a typical MaxDiff study: decide how many items to show per set (typically 4 or 5), and how many sets (questions) to show each respondent (typically 8 to 24). Click Generate Design to generate the experimental design (typically with the default number of versions, 300).
(Notes: if you want to use prohibitions between items in Bandit MaxDiff, then you need to select "on-the-fly" design generation. Because one typically uses aggregate logit to analyze large item list bandit MaxDiff problems, designs with individual versions lacking connectivity are usable.)
4. Test your survey to make sure everything looks and functions as you expect. If you answer the MaxDiff questions by always favoring certain items, after 5 or 10 completed respondent records following this same preference strategy you should see that these items tend to appear more often than the other items for later respondents.
Remember that Bandit MaxDiff uses any completed practice records on the server to influence the items drawn for later respondents. So, make sure to RESET your survey on the data collection server prior to launching your study so that it deletes any practice data on the server, including the table that Bandit MaxDiff creates to store the mean preferences and variances for prior respondents. To delete practice/previous data and the associated table of group preferences, you must RESET your survey on the data collection server by logging into the Admin Module. Just deleting respondent data without resetting the survey does not clean out the preferences and variances for these practice records in the Bandit MaxDiff table on the server. (See more advanced notes below about changing Bandit MaxDiff designs mid-stream during data collection.)
5. Field your questionnaire, making sure to limit the rate of flow of respondents into the questionnaire. Respondent answers to the Bandit MaxDiff questions are only updated to the Bandit MaxDiff preferences table after the respondent finishes the entire survey (including any answers to the questions following the MaxDiff section). Therefore, if all respondents begin the survey at nearly the same time (before the first respondent has finished), no adaptive learning can occur and this would essentially negate all the potential benefits of Bandit MaxDiff. Therefore, we recommend that no more than about 10 to 20 respondents take the survey simultaneously for good results.
Because Bandit MaxDiff relies on past respondents' preferences to select the items to show next respondents, it should not be used with offline CAPI data collection (where only data from respondents completed on the same device would be referenced).
-----------------------------------------------------------
Advanced Notes Regarding Changing Pre-Generated Bandit MaxDiff Designs During Data Collection
When you use the BanditMaxDiff constructed list command in a project, Lighthouse Studio creates and updates a data table on the server that keeps track of the preference scores and variances for each item, based on past respondents' MaxDiff answers. If you delete respondent data on the server, this data table won't automatically be deleted or reset. To delete and reset this table you must click RESET survey by logging into the Admin Module.
There may be occasional instances in which you need to make a change to your Bandit MaxDiff survey while at the same time retaining previous respondents with their preference information in the Bandit MaxDiff table on the server.
a) You need to correct a spelling error or make a slight wording change to an item. You can easily make this change by editing the text and uploading the project to the server. You do not need to generate a new design if using pre-generated MaxDiff designs due to editing the wording of an item. The design is unchanged.
b) You need to change the number of sets per respondent or items per set. Perhaps your client asks you to reduce or increase the number of MaxDiff sets per respondent after real data collection has started. You want to make this change, while not losing information about the valid preferences from the previous respondents. If using a pre-generated design, this will end up changing the MaxDiff design ID stored in each respondent's record. The software will warn you not to do this if you have already started collecting data. But, if you need to proceed anyway, you can update the design, realizing that you will need to do some extra work on the back end to export your two groups of respondents separately to a .CSV file and combine the data for analysis afterward. After updating the project with the new design, the previous (still valid) Bandit table with item preferences and variances will continue to be used and updated.
c) You need to add or delete items. If you change the number of items in your MaxDiff study and upload a new pre-generated design and if you still want to preserve the past information from the previous respondents in the MaxDiff data table on the server, this may lead to errors in the proper functioning of the Bandit MaxDiff algorithm for selecting items for new respondents. Therefore, be especially careful if adding or deleting items (without the usual RESET survey procedure that deletes practice data and the Bandit MaxDiff data table), as this can cause problems for proper analysis. If you were to add items to the MaxDiff study and update the pre-generated design, the software assumes that you are adding items to the end of the list. New rows would be added to the bottom of the MaxDiff data table on the server, initialized to have flat preferences with high variance. If you delete items from the list and update the pre-generated design, the software assumes you are deleting items from the end of the list. The previous preference scores and variances are retained on the data table. Thus, deleting an item in the middle of the list and updating the pre-generated design would cause your new list of items to refer to the wrong preferences and variances for the BanditMaxDiff algorithm. In either case, if you choose to proceed anyway in changing the pre-generated design, you can update the design, realizing that you will need to do some extra work on the back end to export your two groups of respondents separately to a .CSV file and combine the data for analysis afterward.
