SS Summer 2005

SSI Web v5.2 Even Better with Quota Control

We are working hard to make SSI Web your software of choice for both general web interviewing and for projects that include conjoint sections. Many of our conjoint analysis customers have been pleasantly surprised with the capabilities of our general interviewing component, CiW. They’ve been able to save money and time by using CiW.

Given our move to administer more research in-house, the cost savings that we are recognizing from doing this with SSI Web is staggering. We have already saved about $100,000 by doing this ourselves with your software rather than hiring outside consultants, and I only expect the trend to conduct more internet surveys (using your hosting services) to increase. --Tim Glowa, Staff Analyst, Market Research, Reliant Energy

Back in October, we released SSI Web v5, which was a major improvement in our web interviewing platform. We had originally planned for Quota Control to be part of that release, but were unable to include it then. In May, we released v5.2 (a free update for v5 users) with Quota Control functionality.

The desire for Quota Control can be economic: respondents completing the survey are paid a cash incentive, and the researcher does not want to go over-budget. Quota Control is also used to achieve a sample reflecting known characteristics of a population: representative proportions of male, female, low income, high income, etc. Using Quota Control can also help us conserve our relationship with respondents: we limit the amount of time we ask of respondents to a few screener questions if we do not need their completed record.

Consider a B-to-B study, wherein the researcher wants to interview 1000 total respondents, with a certain number of completes within different company sizes (large, small) and types (manufacturing, services):

Quota Cell Definitions:	Quota Cell Limits:
Large Company, Manufacturing	N=400
Large Company, Services	N=300
Small Company, Manufacturing	N=100
Small Company, Services	N=200
Total:	N=1000

A respondent starts the survey and is first asked what size and type company she works for. These are called screener questions. After screener questions are asked, a determination is made (at the Qualification Point) whether this respondent is needed. If this respondent's Quota Cell is already filled with enough completed records, then she is immediately skipped to the end and disqualified. Otherwise, she is allowed to continue the survey. This process is represented by the diagram below:

SSI Web's quota control is flexible and powerful. Quota membership may be based on many variables, resulting in a virtually unlimited number of quota cells. Screener variables can be "passed in" through a URL link or asked at the beginning of the questionnaire. Quota control cannot guarantee that the quotas you establish for groups are not exceeded. There may be some overage. For example, you may wish to interview just 100 males. A simple rule would be to let males continue the survey if you haven't yet achieved 100 completed male records. Imagine that 99 males have already completed the survey and five males are currently in process (have passed the Qualification Point and are in the middle of the survey), but these five haven't yet finished the survey and been counted as complete. A new male respondent reaches the qualification point, and is allowed to take the survey (since only 99 complete records have been recorded). We can see how this could result in 105 completed records, if all the respondents in process eventually complete the survey.

Predictive Quota Control

SSI Web has more sophisticated rules for determining if the Quota Cell Limit has been or is expected to be reached. You can specify that SSI Web should assume that, say, 80% of the respondents who have passed the Qualification Point and are actively taking the survey will eventually finish (i.e. a 20% drop-out rate). You can also specify that respondents who haven't completed the survey within a certain amount of time should be marked as Inactive.

If these options are in place, when a new respondent reaches the Qualification Point, SSI Web considers the number of respondents in this respondent's quota cell already completed and actively in process (discounted by 20%) to decide if the new respondent is needed. We refer to this ability to more intelligently determine if more respondents are needed within a Quota Cell as Predictive Quota Control. Predictive quota control can certainly limit the amount of overage, but it still cannot ensure that quota cell limits are not exceeded.

Download a Free Demo Version Today

Download a functioning demo version at www.sawtoothsoftware.com/ssiweb.shtml. Go back to Index

---

Call for Papers: Sawtooth Software Conference 2006

On March 29-31, we will hold our twelfth Sawtooth Software Conference in Delray Beach, Florida. Our research conference brings together some of the best minds in our industry to talk about practical issues in online interviewing and quantitative market research. It is not a sales-oriented event for our software, but a chance to exchange ideas and receive education from a variety of sources and perspectives. Papers presented at our previous Sawtooth Software Conferences are cited frequently in journal articles.

We're looking for exceptionally strong papers. If you'd like to be on the program, please respond promptly (by September 1, email: bryan@sawtoothsoftware.com) with a one-page abstract describing your proposed paper, with special attention to the findings and what the audience will "take away" from the presentation. You must also include a 50-word description of your paper to include in the conference brochure, should your abstract be accepted.

We are interested in papers on a variety of subjects, including Web interviewing, market segmentation, scale development, customer satisfaction modeling, conjoint/choice analysis, MaxDiff, perceptual mapping, hierarchical Bayes methods, forecasting, pricing research, market simulations, and case studies. These papers need not involve Sawtooth Software's programs or approach.

In an effort to provide more balance to the program, we are encouraging papers that are not about conjoint/choice modeling. For all topics, we are eager to see evidence of managerial relevance, external validity, profit impact, etc.

Presenters receive a complimentary conference registration. To be accepted, a paper must show promise of being sufficiently practical to be of use to the least sophisticated members of the audience, while having enough substance to be of interest to the most sophisticated members. In addition to standard presentation slides, authors are required to submit a journal-quality written paper for publication in the Conference Proceedings.

We strive for the highest quality in our conferences. If your abstract is accepted, a member of the steering committee will review early drafts of your presentation and offer suggestions. Authors are expected to consider these suggestions conscientiously and rework their presentations as needed. Sawtooth Software reserves the right to remove any author from the program or proceedings that fails to meet deadlines or produce high quality work.

Sawtooth Software Conference 2006 Steering Committee Members are:

• Bryan Orme, Sawtooth Software
• Karlan Witt, Customer Metrics Group
• John Wurst, SDR/University of Georgia
• Joel Huber, Duke University

Go back to Index

---

2005 Customer Feedback Results

For the third year running, we fielded a customer feedback survey. A very sincere thanks to all who participated. The quality and quantity of openend responses were tremendous! We are reviewing the comments, and hope to be able to implement many new features and changes based on the suggestions.

“Longer support hours!” was one of the most common themes—especially from our east coast customers in the US. When we moved to Washington State 10 years ago, this put us at a time zone disadvantage with respect to the east coast. Based on a recent hire of another customer support consultant, we will open the office to business and technical support 1.5 hours earlier than before. Starting August 1 our new hours will be 7AM-5PM, Pacific.

Here are a few interesting findings from the 2005 Customer Feedback Survey.

Overall Impression

The overall impression of our company is high, and improving. In 2003, 92% of respondents rated the overall quality of interactions with us as Excellent or Good. In 2004 and 2005, 93% and 94% said the same. We’ve plotted the top box (Excellent—FAR exceeded what I normally expect from a software company) percentage below:

Quality of Advice

A key indicator for success is the quality of the help provided. In 2005, 98% of respondents who had an interaction with us either Strongly or Somewhat Agreed that they were confident the support/advice they received was correct. We’ve plotted top box results below:

Conjoint Methods Used

For three years running, we’ve tracked the relative use of Sawtooth Software conjoint methods. In 2005, the momentum continues for increased use of CBC. CBC accounted for 54% of all conjoint/tradeoff projects conducted by customers responding to the survey over the last 12 months. At Sawtooth Software, we’ve been interested in the relative use of our three conjoint packages. The relative proportions are shown below, weighted by projects conducted.

Best/Worst (MaxDiff) Usage

For the first time, we included Best/Worst (MaxDiff) scaling as a response option in the conjoint/tradeoff method usage question. (Although MaxDiff scaling is not in the formal sense a conjoint methodology.) 12% of respondents that conducted preference modeling employed Best/Worst during the year, and it accounted for just under 4% of the total preference modeling projects conducted.

We currently offer a Best/Worst Experiment Designer, and CBC/HB or Latent Class software for parameter estimation for MaxDiff. However, we haven’t yet integrated a complete MaxDiff solution within SSI Web yet. We hope to offer that soon!

Go back to Index

---

Adaptive CBC: Summary of Three Experiments

This article is excerpted from an article by Rich Johnson, Bryan Orme, Joel Huber, and Jon Pinnell entitled “Testing Adaptive Choice-Based Conjoint Designs” presented at the 2005 Design and Innovations Conference, held in Berlin. The full text may be downloaded from our Technical Papers library at www.sawtoothsoftware.com.

Choice-Based Conjoint analysis (CBC) has achieved a dramatic increase in use during the past decade. One important reason for the growth of CBC is that choices are more like actual marketplace behavior than are the rankings or ratings used by other conjoint methods. Another reason is that hierarchical Bayes (HB) methods now permit estimation of partworths for individuals, where previously it had seldom been possible to collect enough choices from each individual to support individual-level analysis. However, despite the important contribution of HB, there remains considerable incentive to make choice designs more efficient.

Huber & Zwerina (1996) showed that efficient choice designs have four characteristics: orthogonality, level balance, minimal overlap, and utility balance. The first three of these are present in designs provided by Sawtooth Software’s CBC System, but utility balance is not.

Various combinations of authors have been involved in three (2003, 2004, and 2005) recent Sawtooth Software experiments to test an algorithm for adaptive CBC (ACBC) which accounts for individual-level utilities when designing choice tasks.

An Algorithm for Adaptive CBC:

In a good design the estimation error for the parameters is as small as possible. The adaptive algorithm creates a unique design for each respondent, where the questions are chosen to maximize D-efficiency. Because partworths affect D-efficiency (utility balance yields greater statistical efficiency), preliminary estimates of partworths are needed for each respondent.

ACBC begins with an estimate of the respondent’s partworths, obtained differently in the three experiments we conducted. The first choice task is always random, subject to requiring only minimum overlap among the attribute levels represented. After the first choice task has been completed, the information matrix for completed choice tasks is calculated. Alternatives for the next choice set are constructed to maximize D-efficiency, based on the preliminary partworth estimates. There were differences in the three experiments in how the initial estimate of partworths was obtained, whether estimated partworths were updated during the questionnaire, and in how the next task was chosen to improve design efficiency. (Those details are contained in the full written paper, but not in this summary.)

The First Experiment:

This study was done with approximately 1000 allergy sufferers who were members of Knowledge Networks’ web-based panel. The product category was antihistamines, described by 9 attributes having two or three levels, for a total of 23 levels.

The respondents were randomly allocated to receive either standard CBC, or an Adaptive CBC questionnaire. Self-explicated questions similar to ACA’s “prior” section were used to estimate partworth utilities for each respondent for use in the adaptive CBC design algorithm.

Success was measured by hit rates measuring accuracy of prediction of individual choices in holdout choice tasks, and MAEs (mean absolute errors) of share predictions for the holdout tasks. Hit rates for ACBC and CBC were nearly identical, but share predictions showed a directional improvement for ACBC. However, the experiment didn’t allow for testing significant differences between share prediction accuracy.

The Second Experiment:

The second study dealt with laptop computers, and was conducted with approximately 1000 members of AOL’s Opinion Place panel. The data were contributed by SPSS. Respondents were selected using a “river” methodology, recruited from a variety of popular Web portals.

There were two differences from the first experiment that may have affected the results.

• In this study the self-explicated questions included within-attribute desirability ratings but not attribute importance ratings. This decision was made to save interview time since the between-attribute constraints had not been useful in the first study.

• We made a small adjustment to the design algorithm for ACBC.

As with the first experiment, hit rates showed a virtual tie between ACBC and CBC. But in contrast to the first experiment, share prediction accuracy was directionally worse for ACBC, relative to CBC.

This inconsistency led the authors to search for possible causes. They calculated D-Efficiencies based on preliminary partworths, and found the ACBC designs to have average efficiencies nearly twice those of the CBC group. Thus it appeared that the adaptive algorithm behaved as expected. However, when computed using final partworth estimates, the efficiencies for the ACBC group were much smaller. For the Full Profile treatment the ACBC designs were only 9% better than the CBC designs.

For Full Profile respondents, the average correlation between self-explicated estimates and final partworths was only 0.445. The authors concluded that the preliminary partworths had not been sufficiently accurate to be useful in guiding the adaptive algorithm. The authors concluded by observing that the adaptive algorithm appeared to have worked as expected, but that the preliminary partworths available to it were not effective. They suggested further research in which other means might be used to estimate preliminary partworths.

The Third Experiment:

The third experiment was done with approximately 450 respondents who were members of MarketVision Research’s Web panel. The product category was hotels, described by 9 attributes.

An important difference between this experiment and the previous ones was the way preliminary partworths were estimated. Self-explicated questions were asked about desirabilities of levels within attributes so we might investigate the effect of constraining partworths to have desired rank orders, but those answers were not used to construct preliminary partworths. Instead, a hierarchical Bayes procedure was used to estimate partworths and to update them “on-the-fly” after each question.

Although in this third study it appeared that hit rates for ACBC had a slight edge over CBC, that improvement turned out to be illusory. A covariance analysis was done to remove any spurious difference in hit rates due to the difference in test-retest reliabilities, and when the groups were equated on reliability the difference in hit rates disappeared. We are left with results for hit rates very similar to those for the first experiment: no significant difference between treatments.

Results for share predictions showed slightly lower share prediction predictive accuracy for ACBC relative to CBC. ACBC again failed to demonstrate superiority over regular CBC. We believe that the algorithm used in this third study was superior to previous versions, and are disappointed that the results were not better. In the last section we consider possible reasons for its failure.

Discussion:

The ACBC algorithm was tested extensively with simulated respondents before the first study with human respondents. It had worked well in that artificial context. But simulated respondents provide fewer challenges. Accurate partworths for our simulated respondents were available at the beginning of the interview, simulated respondents did not change their preferences during the interview, and they made choices faithfully according to the logit model. We can think of several possible reasons for ACBC’s failure to perform as well with human respondents, some of which are listed below.

Initial estimates of partworths not good enough. The adaptive algorithm requires preliminary estimates of partworths. The first study, where ACBC seemed to work well, collected self-explicated judgments of attribute importances and desirabilities of attribute levels in a way similar to that of ACA. Perhaps this provided better preliminary partworth estimates than either of the later two studies. The second study did not ask for attribute importances, and its preliminary partworths did not correlate well with final partworths. In the third study the “on-the-fly” partworth estimates became quite good for later tasks, but may not have been good enough during the early tasks.

Respondents change their values during the interview. We know from other research (Johnson and Orme, 1996) that brand tends to become less important and price more so as interviews progress. Perhaps there are other changes as well, and partworths obtained from information available early in the interview may not be capable of leading to a design that is efficient for estimating partworths at the end of the interview.

Respondents don't use logit models to make choices. We know respondents don’t make choices by summing partworths. There is ample evidence that they use various schemes to simplify the job of answering choice tasks in market research questionnaires. Despite this, the multinomial logit model has been generally successful in predicting respondent choices. For example, in the three studies reported here, hit rates were usually nearly as large as test-retest reliability percentages, indicating that the hit rates were nearly as good as possible.

D-Efficiency may not be a good criterion to maximize. Although the logit model has many desirable properties and provides a useful approximation to respondent choice behavior, high D-Efficiencies may not translate into favorable hit rates and MAEs. Utility balance is desirable under the logit model, but for respondents who use response strategies different from the logit model, the other three design characteristics may be more critical. Although D-Efficient designs produce good estimation of logit parameters, those parameters may predict choice behavior less well than parameters developed from designs that maximize orthogonality, level balance, and minimal overlap. Designs produced by Sawtooth Software’s CBC System, to which we have been comparing ACBC designs, do precisely that. Perhaps designs which do not take account of partworths, such as those produced by regular CBC, have an advantage for predicting hit rates and choice shares in holdout concepts.

In the meantime, we are impressed that standard CBC designs appear to be surprisingly robust, and regular CBC appears to be hard to beat.

References

Huber, Joel and Klaus Zwerina (1996), “The Importance of Utility Balance in Efficient Choice Designs,” Journal of Marketing Research, 33 (August) 307-317.

Johnson, Richard M. and Bryan Orme (1996), “How Many Questions Should You Ask in Choice-Based Conjoint Studies?”, Available at http://sawtoothsoftware.com/technicaldownloads.shtml#howmany

Johnson, Richard M., Joel Huber, and Lynd Bacon (2003), “Adaptive Choice-Based Conjoint,” Sawtooth Software Conference Proceedings.

Johnson, Richard M., Joel Huber, and Bryan Orme (2004), “A Second Test of Adaptive Choice-Based Conjoint Analysis (The Surprising Robustness of Standard CBC Designs),” Sawtooth Software Conference Proceedings.

Go back to Index

---

CBC/Web Workshop
August 29-30, 2005
Dallas, TX

Bring your laptop for a hands-on experience with CBC/Web in a small classroom setting taught by Tom Pilon, Ph.D. The class emphasizes the use of CBC/Web (within the SSI Web platform), but the principles Tom teaches apply directly to the use of CBC for Windows (SMRT Platform).

If you do not already own CBC/Web (within SSI Web v5), we will provide a free demo version for you, suitable for completing the software exercises covered in the class.

Program Outline:

The CBC/Web Workshop is a 14-hour program (8 hours on day 1; 6 hours on day 2) focusing on:

• Introduction and orientation to the SSI Web platform for Web Interviewing
• How choice-based conjoint (discrete choice) differs from other conjoint methods.
• Setting up a CBC study: formulating attributes, levels, placing alternatives within choice sets, how many questions to ask, paper-and-pencil versus computerized data collection.
• Analyzing CBC data (counts, logit, Latent Class, Hierarchical Bayes, simulations) and the pros and cons of each technique.
• How to use Sawtooth Software's CBC/Web system: a hands-on workshop for a sample study.
• Plus, case studies emphasizing the practical application of CBC in real-world problems, handouts and Excel spreadsheets.

Attendees must provide their own laptop PC. Minimum configuration: Windows 98 (second edition) or later.

The workshop will be held in the Harvey Hotel, right next to the Dallas-Ft. Worth airport. To contact the hotel about reservations, call 972-929-4500 or visit the Harvey Hotel website at www.dfwairport.harveyhotels.com. Be sure to request the "Corporate Gold Rate" of $119.00 per night.

Registration:

The registration fee is just $700 for this event. Class size is limited to just 15! Register for the CBC/Web workshop by visiting www.sawtoothsoftware.com/workshops.shtml

Go back to Index

---

Web-Based Conjoint Analysis: Theory and Workshop
September 29-30, 2005
Ft. Lauderdale, Florida

Would you like to learn more about tradeoff (conjoint) analysis, including the popular Choice-Based Conjoint (CBC) method and Adaptive Conjoint Analysis (ACA)? Join us September 29-30 in Fort Lauderdale, Florida for hands-on training using Sawtooth Software’s SSI Web system. SSI Web can create either web-based or CAPI questionnaires. You must bring your own laptop to the training, running Windows 98 (second edition) or later.

This training is designed for those relatively new to conjoint/tradeoff analysis. Attendees should be comfortable with basic statistics for marketing research methods. Most of the presentation will be conceptual, but some math will be involved.

Hotel Location/Activities:

The venue for this event is the beautiful Bahia Mar Beach Resort in Fort Lauderdale, FL. The room rate is just $99 per night (call 800/327-8154 prior to September 5 to receive this special rate, promotional code “SAW0905”).

The hotel is right on a marina. Nearby, you can experience parasailing, jet skiing, scuba diving (a dive shop is on site), swimming, and tennis. There is skybridge access from the hotel to a pleasant sandy beach. A watertaxi can lift you to a nearby center for shopping, restaurants, and nightlife.

It would be a shame to come to this resort and not have time to enjoy the surroundings. Each day will involve 6.5 hours of classroom time. But, we are taking a 3.5 hour lunch on the first day, and on the second day we’ll end at 3:30PM.

Program Outline:

• What is conjoint analysis?
• Developing lists of attributes and levels
• Interpreting part worth utilities and importance scores
• Orientation to SSI Web system for web interviewing (software exercise)
• Introduction to Adaptive Conjoint Analysis (ACA)
• Developing an ACA survey within SSI Web Software (software exercise)
• Programming general surveys with SSI Web (select-type questions, open-ends, numerics, grid questions, and skip patterns) (software exercise)
• Introduction to Choice-Based Conjoint (CBC)
• Developing a CBC survey within SSI Web Software (software exercise)
• Estimating part worth utilities, and moving data from SSI Web to the market simulation module (SMRT).
• Conducting Market Simulations (sensitivity analysis, competitive scenarios) (software exercise)
• Introduction to product optimization searches
• Pitfalls to avoid in conjoint analysis

Registration:

The registration fee is just $700 for the two-day event. Class size is limited to just 15! The room rate is just $99 per night (call 800/327-8154 prior to September 5 to receive this special rate, promotional code “SAW0905”). Register for the Web-Based Conjoint Analysis by visiting www.sawtoothsoftware.com/workshops.shtml.

Go back to Index

---

Extended Support Hours Starting August 1

In our most recent Customer Feedback Survey, we asked you what we could do to improve our technical support. A predominant theme was to increase our availability hours, especially in the morning. Starting August 1, 2005, technical support and sales will be available from 7AM to 5PM Pacific, rather than starting at 8:30AM. You can contact us by phone 360/681-2300 or by email support@sawtoothsoftware.com.

Don’t worry, we aren’t asking Aaron to work 10 hours a day instead of 8. We have recently hired an additional Customer Support Consultant (John Howell). John recently graduated with his Masters degree in Statistics, and has been working for the last five years at Novell. At Novell, he was involved in testing, quality assurance, and pricing/licensing.

Go back to Index

---

Sawtooth Software Cosponsors Bayes Conference

Sawtooth Software is pleased to be a corporate sponsor for the Bayesian Methods in Marketing – Tutorial and Conference, to be held in Linz, Austria, September, 19-21, 2005.

The conference and tutorial will bring together leading practitioners and scholars in marketing that use Bayesian statistical methods.

Greg Allenby and Peter Rossi will be leading a tutorial on September 19-20. This two-day program will develop hands-on competence in applying Bayesian models and analysis to marketing research problems. The tutorial will provide an introduction to the theory behind modern Bayesian methods, and an introduction into R as a convenient programming environment.

The final two days will feature presentations on HB methods that have been accepted through an open call for papers. For more information on this event, please contact Margarete.Wolfesberger@jku.at or visit http://www.ifas.jku.at/.

© 2013 Sawtooth Software, Inc. All rights reserved.