SS Fall 2008

Update on Conjoint Usage Survey

During the month of April, we completed our sixth annual customer feedback and conjoint usage survey. Thank you to all that participated! We hope it isn’t becoming an irritation to you to complete this survey on an annual basis. The results truly are helpful to us.

We’re pleased to report that overall satisfaction with Sawtooth Software remains high, with 97% of our customers stating that the interactions they had with us were either good or excellent. This represents two years in a row that we have obtained this high result.

A portion of the survey focuses on tracking the use of conjoint-related methods among our users. Here were some of the main findings:

1. Among the three main flavors of conjoint, CBC continues to be used more often than ACA and traditional, full-profile conjoint (CVA). Among our users, the relative application of the methodologies as a percent of conjoint projects was as follows: CBC (82%), ACA (13%), and CVA (5%). The results for previous years are shown in the chart below:

Though CBC is used most often, one should not conclude that it is best for all types of applications. Researchers continue to employ multiple conjoint methods, depending on the needs of the project. Among our users’ firms conducting preference modeling in the last 12 months, 85% used CBC, 38% used ACA, and 16% used CVA. 42% of respondents relied on one of these three conjoint tools exclusively.

3. Among those who used CBC last year, HB estimation was used in 79% of final models. This degree of utilization of HB estimation shows that it is now, essentially, the standard for most users.

4. MaxDiff was used by 31% of respondents’ firms in the previous year. The adoption of MaxDiff has been rapid among Sawtooth Software users. The chart below shows this increase:

5. We were interested in the advanced options that our CBC users are employing in their CBC projects. Among CBC users, the percent of projects using different advanced approaches were:

Two of these features require the Advanced Design Module (Alternative-Specific Designs, Shelf Display). The others are all available within the base CBC/Web system.

Go back to Index

Cautions Regarding Minimal Overlap Designs and CBC

It has been common practice in CBC questionnaires to use minimal overlap when designing choice tasks. Minimal overlap simply means that we don’t repeat a level within a task unless we have to. For example, consider a CBC study with four brands (A, B, C and D). We might design a choice task like the following:

In terms of statistical efficiency of main effects (the utility of each level considered independently), such tasks are optimal. For this reason, our CBC software has used minimal overlap designs by default. Also, if each attribute has at most four levels, it has seemed natural to show just four products on the screen, as we get full coverage of the attribute list in each task and it limits the amount of information respondents have to evaluate at one time.

But, minimal overlap’s allure of statistical efficiency and the desire to not overwhelm respondents with too many product concepts to consider per task has negative consequences that we recently have begun to appreciate.

It turns out that using these economical, minimal overlap designs encourages more simplification behavior and superficial information processing than the original card-sort conjoint approach. To illustrate this point, consider an extreme case: Imagine a respondent who has a “must-have” requirement that the product must be Brand B. Perhaps she works at the Brand B company, and therefore is intensely loyal. In each choice task, there is only one possible product she can choose. What are the outcomes? The respondent has an easy time answering the questionnaire (she simply scans each task for Brand B). The fit statistic from the individual-level HB model is extremely high since her answers are so predictable. And, we obtain a perfect hit rate for holdout tasks. But, we haven’t learned anything about how she values the remaining attributes beyond brand. Yet, in a real product choice, there are multiple Brand B models for her to choose among that differ on performance and price. Our model might perform poorly in predicting her actual product choice.

Certainly, not all our respondents are so extreme. But, recent evidence suggests that perhaps a majority of respondents’ behavior within CBC questionnaires can be explained assuming they are only reacting to at most two or three attribute levels. To the degree that respondents establish a few must-have or must-avoid features, minimal overlap questionnaires are not very useful for developing much deeper insights at the individual level than those top-most requirements.

Over the past 15 years experience with Sawtooth Software’s CBC module, we’ve reported average time spent with CBC tasks of around 12 to 15 seconds per task (once respondents are warmed up). We’ve seen relatively high hit rates of minimal overlap holdout tasks. These CBC questionnaires have featured minimal overlap and generally few (three to five) concepts per choice task. We’re embarrassed that it has taken us this long to connect the dots and see the weaknesses in minimal overlap CBC designs.

That’s not to say that economical, minimal overlap questionnaires haven’t produced valid insights at the segment and market level. Even if they have missed opportunities to probe very deeply into each respondent’s preference structure, when simplification strategies are heterogeneous across the sample, the population estimates can be relatively accurate, though not optimally so. And, HB estimation has done a great deal to “fill in the missing blanks” for each respondent (such as for attributes of secondary importance), by borrowing information from the population to infer these parameters of lesser, but still significant, importance.

Where next? What to do about it? One simple remedy is to show more product concepts per task. Imagine, for the example we showed at the beginning of this article, we had used six product concepts instead of just four. In approximately half of the tasks, our Brand B loyal respondent would have had two Brand B products to consider. We would learn more about how she traded off performance and price, after her certain choice of Brand B. Another remedy is to use Balanced Overlap designs instead of Complete Enumeration or Shortcut strategies. Balanced Overlap allows a modest degree of level overlap in the study, and only sacrifices a modest degree of traditional design efficiency.

Our recent R&D efforts in Adaptive CBC show promise of reducing the problems of minimal overlap designs. Adaptive surveys can quickly recognize that a respondent requires Brand B, and then all future tradeoffs will involve just Brand B products. This allows the system to learn more about demanding respondents beyond their first few must-have or must-avoid features. Not surprisingly, the adaptive questionnaires are more challenging and take longer to answer, but we believe the results are probably more accurate and realistic. Currently, about 50 beta testers are putting the software through its paces and helping us gain more experience about this new and interesting approach. We plan to start selling Adaptive CBC software in Q2 of 2009.

Go back to Index

Images of Interviewers Available for SSI Web Users

One way to try to raise the level of respondent engagement in online surveys is to include graphics of interviewers that appear to be standing on the page. At the previous Sawtooth Software Conference, we showed an SSI Web project that included a series of photographs of an Asian female throughout the survey. For our work with Adaptive CBC, we have favored displaying an individual on the screen that appears to be having a conversation with the respondent. We think this practice may improve other surveys as well.

This year, we contracted with a graphics firm to hire five models and develop a series of images for us that we could share with licensed SSI Web users. These images feature six poses of each model in either business or casual wear. Here are some examples of the available photos:

We own the rights to these photographs and are licensing SSI Web users to use these (limited to SSI Web surveys). If popularity grows for using images such as these, we may decide to contract with the graphics firm to add to the library.

SSI Web users may download the graphics in various formats (.jpg, .gif, and .png at either 300 or 500 pixels high) from http://www.sawtoothsoftware.com/download/ssiweb/graphics.

Go back to Index

Cluster Ensemble Analysis Software (CCEA) Released

After a successful beta program and free trial period, we are pleased to announce the availability of our CCEA System for Convergent Cluster & Ensemble Analysis. You can download a free demo version of the software (limited to 50 cases and 4 groups) from: www.sawtoothsoftware.com/downloads.php

Cluster analysis consists of finding groups of cases (e.g. respondents) that tend to be similar within those groups on the basis variables (the variables used in clustering), but different on those same variables between the groups. Cluster ensemble analysis leverages a variety of cluster solutions (an ensemble of solutions) to find a single best consensus solution that has stronger characteristics than any one of the solutions within the ensemble.

CCEA may be considered the next generation to our previous CCA (Convergent Cluster Analysis) software system. In addition to the ensemble approach, CCEA includes the capabilities of CCA software for k-means cluster analysis. The literature argues that ensembles perform better than standard cluster analysis. Our work with CCEA v3 also supports that conclusion. Our cluster ensemble approach consistently obtains better solutions for synthetic data sets with known cluster structure than the standard approach in CCA of selecting the highest-reproducibility solution. To read a white paper detailing our findings and our specific method of developing ensemble solutions, download our CCEA White Paper from our Technical Papers Library at www.sawtoothsoftware.com.

Go back to Index

Ci3 Support to Terminate at End of Year

We've tried to contact all Ci3 users via email to notify them of this change. We're encouraging Ci3 users to consider migrating to the newer CiW platform.

We know some of you will be holding on to Ci3 for certain business/personal reasons/functionality. We are supporting "the rebellion" (think Star Wars) by providing a dedicated Ci3 area on the Sawtooth Software Forum where you can post questions and answers. Sawtooth Software employees may occasionally participate on the site (Darth Vader, AKA Chris King may show up unannounced), but we cannot guarantee that we'll often be there.

Go back to Index

© 2013 Sawtooth Software, Inc. All rights reserved.