SS Fall 2009
Covariates in HB Modeling
Hierarchical Bayes (HB) has been an enormously successful method for estimating individual-level preferences for CBC and MaxDiff data. It is quite robust, and the default settings in our CBC/HB and MaxDiff software make it easy for users to obtain excellent preference scores and robust models. Despite the benefits, many have expressed reservation regarding the software’s assumption that all respondents are drawn from a single, normal distribution, and the fact that every respondent is “shrunk” (to some degree or another) toward that global mean.
The earliest academic articles on HB for conjoint analysis actually involved a more sophisticated approach involving the use of classification variables (covariates) such as demographics, attitudes, and usage patterns. With covariates, respondents weren’t shrunk toward a single population center, but were shrunk toward the preferences of other respondents sharing their characteristics. So, for example, low income respondents’ preferences were influenced principally by other low income respondents; and high income respondents’ preferences were influenced principally by other high income respondents. It was a more streamlined (parsimonious) model than simply segmenting first by the covariate and running HB within each separate segment.
The key practical benefit of covariates in HB modeling is that the resulting part-worths (or MaxDiff scores) will typically demonstrate substantially greater differentiation among respondent groups. This is advantageous for any sort of segmentation work, including simple cross-tabulations. A more accurate and compelling story is portrayed when segments of respondents are not smoothed toward a single mean, but gravitate more toward their true, heterogeneous segment preferences. Covariates such as brand preference, budget thresholds, and preference for product characteristics (perhaps even from BYO questions) are the sorts of variables that bring new and useful information to HB modeling. Standard demographics such as gender and age typically are less predictive of preferences and are less valuable.
A recent white paper published on Sawtooth Software’s website in the Technical Paper’s Library (“Application of Covariates within Sawtooth Software’s CBC/HB Program: Theory and Practical Example”) demonstrates how to use covariates in v5 CBC/HB software. We illustrate how covariates increase the differences in importance scores between segments for a typical CBC study. The table below shows importance scores with generic CBC/HB vs. when using covariates. The final column “Spread” shows the absolute difference in importance scores across segments for each attribute.
Hierarchical Bayes (HB) has been an enormously successful method for estimating individual-level preferences for CBC and MaxDiff data. It is quite robust, and the default settings in our CBC/HB and MaxDiff software make it easy for users to obtain excellent preference scores and robust models. Despite the benefits, many have expressed reservation regarding the software’s assumption that all respondents are drawn from a single, normal distribution, and the fact that every respondent is “shrunk” (to some degree or another) toward that global mean.
The earliest academic articles on HB for conjoint analysis actually involved a more sophisticated approach involving the use of classification variables (covariates) such as demographics, attitudes, and usage patterns. With covariates, respondents weren’t shrunk toward a single population center, but were shrunk toward the preferences of other respondents sharing their characteristics. So, for example, low income respondents’ preferences were influenced principally by other low income respondents; and high income respondents’ preferences were influenced principally by other high income respondents. It was a more streamlined (parsimonious) model than simply segmenting first by the covariate and running HB within each separate segment.
The key practical benefit of covariates in HB modeling is that the resulting part-worths (or MaxDiff scores) will typically demonstrate substantially greater differentiation among respondent groups. This is advantageous for any sort of segmentation work, including simple cross-tabulations. A more accurate and compelling story is portrayed when segments of respondents are not smoothed toward a single mean, but gravitate more toward their true, heterogeneous segment preferences. Covariates such as brand preference, budget thresholds, and preference for product characteristics (perhaps even from BYO questions) are the sorts of variables that bring new and useful information to HB modeling. Standard demographics such as gender and age typically are less predictive of preferences and are less valuable.
A recent white paper published on Sawtooth Software’s website in the Technical Paper’s Library (“Application of Covariates within Sawtooth Software’s CBC/HB Program: Theory and Practical Example”) demonstrates how to use covariates in v5 CBC/HB software. We illustrate how covariates increase the differences in importance scores between segments for a typical CBC study. The table below shows importance scores with generic CBC/HB vs. when using covariates. The final column “Spread” shows the absolute difference in importance scores across segments for each attribute.
 Go back to Index
Survey Hosting Business Thriving
Back in 2001, we began assisting a few of our SSI Web users by posting their surveys to our rented servers. At the time, we knew this would give folks with even basic computing skills the confidence that they could invest in SSI Web and execute their projects without a hitch. Little did we know that hosting surveys would turn into a thriving service business, even among customers with strong computing skills and experience with web servers.
Since 2001, we’ve hosted more than 2800 surveys, representing 1.4 million completed survey records.
Our web-hosting service provides:
- A dedicated and helpful Sawtooth Software employee to assist you
- Automatic data backups every 24 hours
- A server that supports “mod_perl” for superior performance
- Choice of dedicated or shared servers
- Ability to choose your own custom domain names
- Optional Secure Sockets Layer (SSL) for added security
A typical web hosting fee runs from about $250 to $500 for a relatively small project.
Why are so many Sawtooth Software customers choosing our hosting services? One big reason is to avoid the bureaucracy and barriers often encountered when dealing with internal IT departments. Another explanation is the superb service provided by our hosting manager, Murray Milroy.
As some examples of Murray’s “extra-mile” service, he not only posts the surveys, but often clicks through them to see if there are any obvious red flags or problems. He has been known to discover broken links and faulty skip-patterns, help users with links from sample providers, or restore project files for studies that had been completed (and lost by the users!) from months before. While we cannot guarantee a perfectly problem-free hosting experience, it’s sure nice to know that an experienced person like Murray is there to safeguard your data and help you along.
To learn more about our hosting services, please visit http://www.sawtoothsoftware.com/services.
Go back to Index
Sawtooth Software Conference 2010
We are pleased to announce that the 2010 Sawtooth Software Conference will be held October 4-8, 2010, at the Newport Beach Marriott, in Newport Beach, California. This conference is held only once every 18 months. Given this location and the success of our last conference, we expect 200+ people for the 2010 conference.
We have chosen the Newport Beach Marriott because of its excellent location just 5 minutes away from Orange County Airport (SNA), and 45 minutes from Los Angeles International (LAX), and also because of its excellent value. Our contract with the hotel will allow us to keep the conference registration fees at the low rates we had in 2009 ($900 registration for the 2.5-day conference, plus $225 per optional tutorial). Also, your costs should be lower than 2009 since the room rates are $179, and due to the proximity to airports.
The Newport Beach Marriott just won the Marriott food and catering competition, so you can expect that you will be treated to the top-notch experience that you are used to at the Sawtooth Software Conference.
Optional workshops and tutorials will be October 4-5, and the main conference sessions will run from October 6-8. So, mark your calendars and put it into your 2010 budget. A call for papers will be issued in January.
To get a feel for the quality of the presentations given at the last conference, you can download a free copy of the written conference proceedings (a 300+ page volume) at www.sawtoothsoftware.com/education/techpap.shtml.
Go back to Index
SSI Web v7 Upgrade Progress
We're making steady progress on a new version of SSI Web v7, and its benefits will affect all components within the platform (CiW, CBC, ACBC, ACA, MaxDiff, CVA).
Key things that are going into the v7 upgrade include:
- No more limitation of 4 character studynames
- Multiple CBC studies, ACA studies, and CVA studies may be included within the same project. This means less linking of surveys!
- Back button (not just the back button on the browser toolbar, but a back button within the survey). The back button allows respondents to back up in the survey, even to sections of the survey that were completed in a previous session (such as after a restart).
- Graphical select and radio buttons
- Randomized Blocks of questions (e.g. the 3 blocks of questions: Q1-Q5; Q6-Q10; Q11-Q15 can be randomized, and Q1-Q5 always stay together as a block, Q6-Q10 always stay together, etc.)
- The standard formatting controls in the edit question dialogs will operate nicely in conjunction with styles. Right now, styles typically override most of the formatting controls. SSI Web surveys will start by default with a more attractive style, so your surveys will look much better even before you apply your favorite style.
- Improving the data export speed
- Better multi-lingual support
- Automatic survey upload and data download. The FTP process will be built into SSI Web, so this feature will make working with the server faster and less complicated.
We are committed to making sure that SSI Web works well for studies that involve a conjoint analysis component as well as studies that don’t include any conjoint analysis. Expect v7 to be released in the first half of 2010.
Go back to Index
Dual-Response MaxDiff for Anchored Scaling
MaxDiff (best-worst scaling) has been successful and popular among Sawtooth Software users. It is a powerful and flexible scaling technique for scoring multiple items. In each question, respondents are shown typically four or five items and are asked to mark which item is best and which is worst (or most important/least important). It takes longer to complete a MaxDiff questionnaire than a ratings grid on the items. But, the results typically are worth the extra effort.
Despite all the benefits of MaxDiff, an issue that sometimes leads to concern is that the scores are placed on a relative (ipsative) scale rather than an absolute scale. For example, for each respondent, we obtain a priority ordering of items on an interval scale. But, each question only allows respondents to say which item is best and which is worst. Respondents cannot tell us that most or all the items are really good, or most or all the items are really bad. This means that the scale is not anchored to any meaningful reference point, such as a point indicating zero preference or a threshold indicated the boundary between good and bad items.
Some pretty sophisticated models that fuse traditional rating data with MaxDiff choices have been proposed and presented at our conferences. These are challenging to implement, and they also rely on the problematic ratings question (such as the 5-point scale).
The inventor of MaxDiff, Jordan Louviere, made a suggestion at the last Sawtooth Software regarding the relative scaling quandary. We followed up with him regarding details, and validated his approach with a methodological study (“Anchored Scaling in MaxDiff Using Dual-Response,” available in our Technical Papers Library at www.sawtoothsoftware.com).
Jordan’s idea is very straightforward. Rather than trying to resolve the relative scaling issue with a sophisticated model fusing rating scale data with choice data, he simply adds another (a “dual”) choice question to each MaxDiff task.
After asking which item is most and least important, Jordan suggests that we ask another question (directly below the standard MaxDiff question):
Considering just these 4 features...
O All 4 are important
O None of these 4 are important
O Some are important, some are not
Our research suggests that this dual-stage add-on question only takes an additional 3 seconds on average per MaxDiff question.
The dual-response gives us the data to establish an anchor point for the scaling of items: the utility threshold between items deemed important vs. not important. For example, a respondent can express that most of the items are unimportant or most of the items are important. The “zero-point” in the resulting scores indicates the boundary between unimportant and important items. Items judged not important carry negative scores, and those that are important carry positive scores.
With existing SSI Web software, it’s very easy to add a dual-response question to MaxDiff tasks. But, analysis requires modifying the data file prior to submitting the data for Latent Class or HB estimation. The details for doing this are described in the white paper we referenced earlier in this article.
We are generally pleased with this approach, and plan to offer it as an option in a future version of MaxDiff software.
Go back to Index
Update on Conjoint Usage Survey
During the month of April, we completed our seventh annual customer feedback and conjoint usage survey. Thank you to all that participated! We hope it isn’t becoming an irritation to you to complete this survey on an annual basis. The results truly are helpful to us.
We’re pleased to report that overall satisfaction with Sawtooth Software remains high, with 99% of our customers stating that the interactions they had with us were either good or excellent.
A portion of the survey focuses on tracking the use of conjoint-related methods among our users. Here were some of the main findings:
1. Among the three main flavors of conjoint, CBC continues to be used more often than ACA and traditional, full-profile conjoint (CVA). Among our users, the relative application of the methodologies as a percent of conjoint projects was as follows: CBC (87%), ACA (8%), and CVA (5%). The results for previous years are shown in the chart below:
Though CBC is used most often, one should not conclude that it is best for all types of applications. Researchers continue to employ multiple conjoint methods, depending on the needs of the project. Among our users’ firms conducting preference modeling in the last 12 months, 85% used CBC, 32% used ACA, and 17% used CVA. 60% of respondents used multiple methods during the past year.
Adaptive CBC (ACBC) was just barely being released at the time we conducted this survey. Therefore, we didn’t include it in the survey. It will be interesting to see what percent of projects are reported to be running under ACBC when we repeat the usage survey in spring of 2010.
3. Among those who used CBC last year, HB estimation was used in 77% of final models. This degree of utilization of HB estimation shows that it is now, essentially, the standard for most users.
4. MaxDiff was used by 37% of respondents’ firms in the previous year. The adoption of MaxDiff has been rapid among Sawtooth Software users. The chart below shows this increase:
5. We were interested in the advanced options that our CBC users are employing in their CBC projects. Among CBC users, the percent of projects using different advanced approaches were:
Two of these features require the Advanced Design Module (Alternative-Specific Designs, Shelf Display). The others are all available within the base CBC/Web system.
Go back to Index
Second Edition of "Getting Started with Conjoint" Book Available
On the heels of the successful first edition, we have completed a second edition of "Getting Started with Conjoint Analysis." We've updated the text throughout, have written two new chapters and new major sections, and have expanded the glossary of terms. Cost is $40, or $35 each if ordered in bulk (20+ copies). Order your copy at: http://www.sawtoothsoftware.com/education/cabook.
Go back to Index
Web Sample Providers: Who We Are Using
Earlier this year, we completed our 2009 Feedback Survey (thanks to all who participated!). We asked our users (who conducted web-based surveys) to list the providers of online sample they had used in the previous 12 months. This was asked as an open-end question, with no prompting regarding potential sample providers. The text of the question and the percent of respondents naming each firm are given below (multiple responses allowed; firms mentioned by fewer than 3% of respondents not displayed).
What providers of Online Web Sample/Online Panels did your firm mainly use in the previous 12 months? (List up to six).
Go back to Index
Scale Factor and Conjoint Results
As researchers gain proficiency in conjoint analysis, they should make sure to pay attention to the issue of Scale Factor. Respondents who are very consistent in their choices have part-worth utilities of larger absolute magnitude than inconsistent respondents. For example, consider two respondents, “Sloppy Sam” and “Consistent Carrie.”
“Sloppy Sam’s” Raw Utilities:
-0.5 Red
0.0 Green
0.5 Blue
0.5 Low Price
0.0 Medium Price
-0.5 High Price
“Consistent Carrie’s” Raw Utilities:
-3.0 Red
0.0 Green
3.0 Blue
3.0 Low Price
0.0 Medium Price
-3.0 High Price
The “raw” utilities above are the naturally-scaled values from utility estimation, as saved to the .HBU or .UTL files. Carrie’s utilities show the same relative pattern of preference as Sam’s; but each value is 6x the size (scale). As respondents are more consistent (respond with less noise), their scale goes up, and sometimes quite dramatically.
Does it make sense to say that Carrie prefers Blue a great deal more than Sam? (Her utility for Blue is 3.0 and his utility is 0.5). She is indeed more consistent in expressing her preferences via the conjoint questionnaire, but it isn’t necessarily the case that she prefers Blue a great deal more.
This highlights the danger of directly comparing raw part-worth utilities across respondents. Since our early versions of ACA in the 1980s, we have recognized this issue and chosen to summarize utilities using a method that normalizes the scale across respondents. We have also recommended that normalized utilities be used in subsequent cross-tab or cluster analysis. Our “zero-centered diffs” normalization gives respondents equal scale (in terms of sums of differences between best and worst levels). The normalized utilities for Carrie and Sam would be identical.
Please note that shares of preference are simulated using raw utilities (and shares are normalized to sum to 100 for each respondent). We do this because we think it probable that Carrie also is more attentive (less haphazard) than Sam in her real world choices. Her simulated share probabilities will be more extreme than Sam (her choice reflects greater certainty).
The average respondent error within the context of the conjoint exercise doesn’t necessarily reflect the amount of error in buyer’s actual choices. In fact, most researchers find that choices in the conjoint laboratory imply less error than purchases in the real world. In other words, the overall scale of conjoint utilities is frequently too high, and the resulting sensitivity of the simulator is magnified relative to actual buyer behavior. Researchers often find that they can improve the validity of simulations by “tuning down” the sensitivity (using the “Exponent” setting) to adjust for these differences. One needs good data and experience to do this, so it shouldn’t be done indiscriminately.
Go back to Index
|
Technical Papers
