MaxDiff scores

After collecting responses to your MaxDiff exercise, MaxDiff scores (importance scores) are estimated using Hierarchical Bayesian (HB) utility estimation. MaxDiff scores are highly insightful as they provide both a ranked ordered list of items and illustrate the relative importance or preference of items.

MaxDiff exercises (and therefore MaxDiff scores) offer advantages over traditional ranking questions, which only indicate the ranked order of a list. They provide greater discrimination and are free from scale use bias typically associated with traditional rating questions.

MaxDiff operates similarly to a "beauty contest" among items. As respondents select their preferred and least preferred items in the exercise, we compute preferences for each item in the study.

Each item is assigned a preference or importance score, which reflects its impact on choices.

These scores, available in the Summary tab of the exercise analysis area, are positive, sum to 100, and are ratio-scaled. This means that an item with a score of 10 is twice as preferred or important as an item with a score of 5.

The most common way to visualize MaxDiff Scores is to list items from highest to lowest scores for the entire sample or for specific respondent subgroups.

Max Diff Scores Summary Charts in Discover

If desired, you can select the option to display 95% confidence intervals.

Assuming your respondents are representative of the randomly drawn population, you can be 95% confident that the true population score lies within this interval.

You can also use the 95% confidence intervals to determine if one item is preferred over another. If the intervals for two items do not overlap, you can be at least 95% confident that one item is preferred to the other.

Segmentation allows you to analyze exercise results by comparing groups of people. These groups can be defined by different responses or values for questions or variables in your survey.

For instance, if your MaxDiff exercise compares music artists, you can segment the results based on another survey question or variable (like respondents' locations) to compare responses between North America and Europe.

To apply segmentation, click the Segmentation dropdown and choose a question or variable from the menu. Once applied, the Segmentation icon turns green.

To clear segmentation or apply a different one, click the dropdown and select No segmentation or choose another option from the menu.

Segmenting Dropdown in Max Diff Analysis

When segmentation is applied, your results update to display scores for each group across all items in your MaxDiff exercise. Respondents who do not have any response or defined variable value will be absent from the results.

In the upper right-hand corner of the settings panel is a download menu (). Within the menu there are five files that are available for download:

Scores (.xlsx)
Individual Scores-Rescaled (.csv)
Individual Scores-Raw(.csv)
Counts
Charts

The Scores download contains a summary of the MaxDiff Utility scores (identical to the table viewable in Discover). Scores are rescaled.

Individual MaxDiff scores can be downloaded in two formats: Raw and Rescaled.

In both downloads, each respondent's data includes a column labeled MaxDiff_Fit (RLH). This fit statistic utilizes root likelihood (hence "RLH") to indicate the probability that each respondent made their choices based on their preference or utility scores. RLH can be understood as the geometric mean of the probabilities that the raw utilities explain the respondent's choices.

Raw scores in MaxDiff analysis represent the regression weights derived from a multinomial logistic regression model, typically using an HB MNL model. These scores are centered around zero, meaning their average value is 0 unless anchored scaling MaxDiff is used. In anchored scaling, scores are positive for items preferred more than the anchor point and negative for those less preferred.

Raw scores are less intuitive for audiences to understand than the rescaled scores (described below); however, they are useful for advanced researchers who wish to make probabilistic predictions of choice among the items using the logit equation.

These scores are always positive, sum to 100, and follow a ratio scale, meaning a score of 10 indicates an item is twice as preferred or important as an item with a score of 5. Rescaled scores are more intuitive for most audiences to grasp. For more details about how the raw HB scores are converted to probabilities summing to 100, please refer to Appendix K of the  CBC/HB manual.  Within Appendix K, refer to the section entitled: "A Suggested Rescaling Procedure."

The Counts download contains data regarding how many times each MaxDiff list item was shown, was selected as best or worst, and count proportions.

The data includes columns reporting count proportions. The “best” count probability is the likelihood that respondents selected the item as “best” when it was available within a MaxDiff set. The “worst” count probability is the likelihood that respondents selected the item as “worst.” Being probabilities, counts proportions are bounded by 0 and 1. If four items are shown per set, the chance probability of choice is 25%. Thus, an item with a “best” count proportion of 50% is chosen as “best” at twice the chance level (again, assuming four items were shown in each MaxDiff set). A quick-and-dirty summary MaxDiff score that is highly correlated with HB results can be calculated by taking each item’s “best” count proportion minus its “worst” count proportion.

Charts exports a .png image matching the scores chart in Discover.

Discover Help Guide

MaxDiff scores

Introduction

Interpreting scores

Confidence intervals

Segmenting

Downloads

Scores

Individual MaxDiff scores

Raw scores

Rescaled scores

Counts

Charts