User Experiences: How to Marry People, Prototypes, and Adaptive Conjoint

Sawtooth Software has an ongoing, open invitation for users to submit stories regarding innovative ways they are using our tools. The following represents one such story, submitted by Thomas Klobucar, Director of Research, Vernon Research Group.

People. Market research would be considerably easier if people were not required to make it work—particularly when testing prototypes with dozens of new characteristics and clients waiting to make critical design and pricing decisions based on the results of consumer testing.

Faced with the problem of designing a study to gather quantitative data from a large sample of respondents for such a test, we mulled over a variety of methods for gathering such data.

One of the most difficult issues we faced, and often face when designing studies for manufacturers of durable goods is in finding a method that allows us to show the actual item to respondents before asking them to express their opinions about the items they have seen. Traditional focus groups of eight to ten people are fine for gathering ideas, but their inherently eclectic and unscientific nature cannot yield the utility and pricing data manufacturers crave. No one wants to base business decisions about entire model lines on focus group outcomes.

Another method under consideration was to use the internet to graphically display the object at hand and then use sophisticated interviewing techniques allowed by Sawtooth Software’s SSI Web ACA Module that are only possible using a computer. The problem, of course, with conventional internet interviewing is that respondents cannot see the actual item being tested. For this recent project, which called for the nuanced understanding only possible using computer-assisted interviewing while exposing participants to the actual prototypes, we hit on a design that incorporated broadly-recruited Voice of the Consumer panels of 30 participants each in a number of cities across the country and combined that design with computer assisted interviewing CAI on the internet

Only the largest market research firms may own sufficient computer resources to maintain a squadron of laptop computers for CAI. Even for those that do own laptops in those numbers, the cost of maintaining each with the software necessary to conduct 30 simultaneous interviews may be prohibitive. Since we needed to conduct a computer-dependent survey design (using Sawtooth Software’s ACA), our solution was to rent laptop computers and arrange for high-speed internet access at each of our interviewing facilities—because the prototypes we needed to demonstrate were so large, our interviews ended up being conducted in hotel ballrooms and convention centers. The interviews were conducted in a single room and the interviews were conducted via the internet, at the moderator’s direction.

The outcome of this project was extraordinary. Not only were we able to gather the information you can only get through ACA, we were also able to capture responses to “normal” survey questions using the same SSI Web interview, all without having to fiddle with a single interview computer beyond setting the browser’s homepage to the study URL. The data were ready for download when we returned to the office the next day without having to enter a single response manually. We had our data, and analysis could begin immediately.

The results? Using Peter Williams’ suggestions for including holdout tasks in the study* to ensure pricing data were correctly estimated (and weighting the data accordingly), we were able to duplicate quite precisely the current market and build a series of models that showed what consumers wanted and how much they were willing to pay for those options. The non-ACA data we also captured using SSI Web integrated well, allowing us to look at market segments very closely and determine the attributes that appealed most to different populations.

Sawtooth Software’s SSI Web and ACA module and a little creative logistics management allowed us to build a research model that got the people, the products, and the computers in the same place at the same time without major headaches. We were left with a high degree of confidence in the answers we gave to our clients, who in turn walked away secure in the knowledge that we had gotten things right.

*Williams, Peter (2000). Calibrating Price in ACA: The ACA Price Effect and How to Manage It. Available in Technical Papers Library at: