MaxDiff scores - eliminate respondent first or after output is generated?

Question

Hey Sawtooth Team.
First of, thanks for hosting this great forum with a ton of opportunity for people to learn more about Sawtooth every day. Really helpful.

My question is the following: is the individual raw score for each respondent affected by the scores of the other respondents? Does it take them into account in any way?
I am asking because I was wondering the following: knowing that I will eliminate a few respondents for their answers in other questions, should I eliminate them before running the MaxDiff analysis and generating the scores, or after?

If the individual scores are not affected by the others', I guess I can just eliminate them in the Output file and overwrite the importance rating (combined scores) by the new average of rescaled scores for that item (excl. the eliminated respondents)!?

Thanks for your help.
Tim

Answer 1

HB uses what it knows about the population to help build the scores for each person, so during analysis, it's fair to say that people have some sort of an effect on each other.  So, probably the "purest" thing to do would be to remove respondents you have identifies as "bad" and re-run the scores.  From a practical standpoint, though, if we're talking about a very small proportion of people, I would imagine taking them out and re-running the scores would have very little effect on everyone else.

MaxDiff estimation is usually pretty quick, though, so you could compare things like the average scores and fit statistics before and after cleaning to see if anything really changed.

Comments

If you intend to remove "bad" respondent you are pretty safe, their scores are very close to mean. Removing wont change the prior. But you wont be on safe ground if you remove some special segment of respondents (not just "bad" ones) who have scores far away from mean. It teoreticaly can change the order of utilities for the rest of the group.