Robustness of MaxDiff Designs to Prohibitions
This section investigates the robustness of MaxDiff designs to prohibitions using Sawtooth Software's MaxDiff System and simulated (synthetic) respondent data.
The Case for Prohibitions
MaxDiff questionnaires present respondents with typically four to six items at a time and typically ask which item is "best" and which is "worst."
One advanced option in the software is the ability to specify that certain levels should never be paired within the same set. For example, in a fast-food restaurant study, we might wish to study the importance of the length of time to order and receive food. We might use three items in our MaxDiff design related to this:
Order and receive food within 3 minutes
Order and receive food within 5 minutes
Order and receive food within 10 minutes
When people go to fast-food restaurants, we presume they want to receive their food as quickly as possible. Therefore, it wouldn't make sense to show respondents a MaxDiff task in which any two of these items are paired. Respondents would always prefer the faster speed to slower, so this would be a wasted comparison.
Another example that involves prohibitions is the best-worst case 2 (best-worst conjoint) technique (described further below), where conjoint-style prohibitions are used so that MaxDiff items show a product profile (one level each from multiple attributes) and respondents are asked to indicate which two levels makes them most and least likely to want to purchase the product.
Prohibitions and Design Efficiency
Those researchers familiar with conjoint analysis know that prohibiting certain level combinations from occurring in the conjoint questionnaire can often be very damaging to design efficiency (the ability of the questionnaire to estimate preference weights with precision). However, it turns out that prohibitions (unless excessive) are generally not very harmful in MaxDiff questionnaires. We demonstrate that point below using simulated data sets with differing numbers of prohibitions.
Why are prohibitions more damaging in conjoint analysis than for MaxDiff? Prohibitions in conjoint analysis are "within-concept" and lead to correlations between attributes in the independent variable matrix. For MaxDiff, the prohibitions are "between-concept" and are less damaging to orthogonality. The reduction in design efficiency in MaxDiff studies from prohibitions is due in part from deviations from orthogonality. Other contributing factors are the imbalance that is sometimes introduced in the frequency of items and the reduction in the degree of interconnectivity among items. In optimal MaxDiff studies, each item is shown an equal number of times and is also compared an equal number of times with every other item. And, for any one respondent, we want to minimize the total number of logical steps that it takes to directly or indirectly compare all items. For example, if A is compared to B in question one and B is compared to C in question two, then A is "linked" to C through two steps.
"Gold Standard" Design with No Prohibitions
In a MaxDiff study involving 20 items, there are ½t(t-1) possible two-way combinations of items that can be prohibited. The ½(20)(19)=190 possible paired levels are represented by the white region of the table below:
We begin by simulating data for a typical MaxDiff study that involves 20 items shown in sets of 4. Three-hundred computer-generated respondents each received a 20-set questionnaire and "answered" randomly ("bests" and "worsts". Each respondent received a unique version of the questionnaire, as generated by the MaxDiff designer.
To compute design efficiency, we estimate an aggregate logit model and observe the precision of the estimates (the standard errors). We also compute D-efficiency. The efficiency of one design relative to another is found by taking the ratios of the D-efficiencies for the two designs. The base case example (with no prohibitions) is our "gold standard" for comparison to designs with prohibitions.
Modest Prohibitions Case
First we implement a modest number of prohibitions. Let's consider that the last five items in the study are mutually exclusive, leading to the following prohibitions (prohibited levels marked with an "x"):
These ten prohibitions are grouped, with many occurring within the same row or column. This is potentially more damaging than if ten prohibitions were scattered randomly across the table. But, even these prohibitions have very little effect on design efficiency for MaxDiff. We generated a new series of questionnaires involving these prohibitions and again had 300 respondents answer randomly. The standard errors of the estimates were only slightly larger than in the non-prohibited case. This design is 99.3% as efficient as the "gold standard" design with no prohibitions. For all practical purposes, this design is as efficient as the "gold standard."
Extreme Case
Let's consider a much more extreme case. We'll retain the previous prohibitions, but add other prohibitions which at face value might seem to be quite damaging:
With these challenging (insane?) prohibitions, it becomes impossible for the MaxDiff designer to find a very balanced design. It returns an imbalanced solution wherein the most represented item (item #2) occurs twice as often as the least represented (and most prohibited) item (item #18). While this imbalance in the design might seem concerning, these prohibitions wouldn't ruin the MaxDiff project (especially if we had reasonable sample sizes). The overall relative design efficiency (considering all items) for this highly prohibited case is 77% as efficient as "gold standard" design. In other words, we could obtain equal precision with this excessively prohibited design as the non-prohibited design if we increased our sample size by a modest (1.0/0.77-1) = 30%.
"Conjoint-Style" Prohibitions (Best-Worst Conjoint)
As a final example, MaxDiff may be used for "conjoint-style" experiments, also known as "best-worst case 2" or "best-worst conjoint." With best-worst conjoint, respondents are shown a product profile (as in conjoint analysis, with one level per attribute displayed) and are asked which levels make them most and least want to purchase this product. It's typical to sort the items in natural order for display in each best-worst set, such that the attributes always appear in the same order for a respondent (e.g., brand first and price last). (For more information, see our white paper that describes best-worst conjoint).
Let's imagine a conjoint-style attribute list with 4 attributes, each with 5 levels.
The prohibitions table looks like this:
Repeating the simulation process with synthetic data, we find that this design is 98.8% as efficient as the "gold standard" plan with no prohibitions.
In this section, we've only considered statistical efficiency to assess the issue of prohibitions. We find that MaxDiff designs are remarkably robust to prohibitions. However, there may be additional considerations, such as context bias, that researchers may wish to consider when using prohibitions in MaxDiff questionnaires.
