Computing Scores On-The-Fly
If you want to compute individual-level MaxDiff scores on-the-fly (realtime during the interview), you can use the following functions:
•MaxDiffRankAttLabel(ExerciseName, Rank#)
•MaxDiffRankAttValue(ExerciseName, Rank#)
•MaxDiffScore(ExerciseName, ItemIndex)
These functions allow you to return (e.g. show to the screen, or use in calculations, or skip patterns) the label and score for each of the items in your MaxDiff exercise. You can also return the label and/or score for the top, second, third, etc. item in terms of importance/preference. You can use the functions above together with list-building instructions (e.g. ADD) to force the top n items onto another list. If the list used in ExerciseName is a constructed list, then the ItemIndex refers to items on the original parent list. If you want ItemIndex to refer to the item position in the constructed list, then you can substitute ListValue(listname, 1) in place of ItemIndex. For example, if you want to reference the score for the first item on your constructed list (named q1list), you would specify MaxDiffScore(ExerciseName, ListValue(q1list, 1)).
Note: If you wish to store the rank order of items, labels, or scores of items presented to respondents (as computed by these functions) within the data file, then you need to do additional steps, such as add Free Format questions to your survey with Hidden Variables, and set the value of these Hidden Variables equal to rank order or scores of MaxDiff Items. Respondents don't see these Free Format questions with hidden variables; they are just used to store information to your data file.
These functions use the individual-level logit method to compute positive scores summing to 100. The scores developed in this way only rely on data from the current respondent (the scores cannot benefit from information of other respondents, as final score computation using HB does). As a result, the scores will have less stability and precision than if computed across a sample of respondents under HB (hierarchical Bayes).
We recommend that each item be shown at least 4 times per respondent for reasonable individual-level score estimation using these functions. For more detail on the individual-logit method and comparisons to HB estimation for MaxDiff, see the white paper entitled "MaxDiff Analysis: Simple Counting, Individual-Level Logit, and HB" available for downloading from our Technical Papers library at www.sawtoothsoftware.com.
You can also return the fit statistics for respondents on-the-fly using the following formulas:
The RLH is the root likelihood (the geometric mean of the task likelihoods), which is an intuitive probability expression of how successful the utility scores are in predicting which items respondents pick. If you are showing 4 items at a time in a MaxDiff study, the null likelihood expected by chance is 1/4 = 0.25. We should hope that the respondent's answers fit the data better than just the chance (at random) level. If showing 4 items at a time, a respondent who achieves an RLH of 0.5 has a fit twice the null level. RLH ranges from 0 to 1.0, where 1.0 represents perfect fit.
The RSQ is the pseudo R-squared. It is a measure of fit directly related to RLH, but conveniently expresses what percent of the way the fit is between the null likelihood and perfect fit. RSQ ranges from 0 to 1, where zero is the fit expected at chance and 1 is perfect fit. The formula for RSQ is (LL-NullLL)/(0-NullRLH), where LL is the natural log of the likelihood.
Recreating Scores On-The-Fly After the Fact
If for some reason you need to recreate the scores on-the-fly during subsequent analysis after the surveys have been collected, click Analysis | Analysis Manager, specify Logit as the Analysis Type, click the settings button and then click the Settings link. Check the box that says Recreate on-the-fly scores and rankings. When you run the logit analysis, the on-the-fly scores and rankings are displayed in the On-the-Fly Scores tab of the report.
Technical Details:
Because individual-level logit includes sparse data that may not converge under logit, we break out if the log-likelihood fails to improve by 0.03 in an iteration. We also break out if the range in the logit scores is greater than 15. For all practical purposes, these break-out rules lead to ample precision in the estimated scores, and keep the scores reasonably scaled. The computation is nearly instantaneous, and should not burden servers nor result in a wait time for respondents.
The scores returned using the functions above are normalized, positive scores that sum to 100. The normalization is done as follows:
1. The raw logit scores for all items are first zero-centered by subtracting off their mean.
2. The scores from step 1 are transformed by e^u/(1+e^u); where e is the exponential constant.
3. The scores from step 2 are normalized to sum to 100.
(Note: For scores on-the-fly we employ a simplified method of logit estimation to compute utility scores that is different from the algorithm (Newton-Raphson) used in the logit estimation routine within Lighthouse's Analysis Manager. Because of the differences in the algorithms and the breakout criteria, the results will be extremely similar but not identical.)
