Sawtooth Software: The Survey Software of Choice

A Discrete Choice Take on Holiday Movies

Research shows that on Christmas day, holiday-themed programming spikes as households curl up in front of the television for a dose of holiday cheer. Timeless classics like “How the Grinch Stole Christmas” and “It’s a Wonderful Life” come to my mind. And thanks to @RottenTomatoes, we have a list of the 50 Best Christmas Movies of All Time.

Now the last thing you want to do on the holidays is argue with family. So how will you decide which Christmas movie to watch on December 25th? Why, MaxDiff of course!

Picture of Santa Claus looking at Christmas movies

What is MaxDiff? MaxDiff is an approach for measuring consumer preference for a list of items. That list of items could be advertising messages, product benefits, images, names, and so much more. In this case – we have access to Rotten Tomatoes list and will use those 50 movies as our items. Now, we could always test more than 50 (Sawtooth Software allows you to handle up to 2,000 movies!) but, being the Bayesian I am, I decided to just use their list as my prior.

MaxDiff comes in handy when you anticipate low discrimination in your data. For example, if these are the 50 best Christmas movies of all time, then you probably would rate most, if not all of them a 4 or 5 in a ratings question, making it hard to draw conclusions around which ONE movie to watch. By forcing people to make trade-offs through a MaxDiff exercise, we can uncover what they truly value - or in this case, what they want to watch!

Picture of a MaxDiff survey containing Christmas Movie artworkIn a MaxDiff exercise, respondents are typically shown 3-5 items at a time and asked to choose which item is best and which is worst, or, in this case, which movie they’d be most likely to watch and which they’d be least likely to watch. Feel free to take the survey on your own here! I even employed Sawtooth Software's MaxDiff on-the-fly scores so that you can see what your top three movies are based on your survey responses.

Respondents answer multiple screens of various combinations of movies. These combinations are chosen via an experimental design that is frequency-balanced, positional-balanced and orthogonal.

Using hierarchical Bayesian regression, or HB, the resulting output provides individual-level utility scores for each item. These raw utilities are transformed into ratio-scaled data that can be converted into a rank ordered list for the entire sample. Furthermore, MaxDiff data can be turned into simulations to predict how respondents would choose given any scenario!

Read more about MaxDiff and how it compares to ratings data in Bryan Orme’s article, How Good Is Best-Worst Scaling? Or check out this quick video on What Can MaxDiff Do For You?