
	Technical Papers Pricing & Ordering Your Cart

Sawtooth Software » Education » Sawtooth Solutions

Education

Support

SS Fall 2000

Internet Software Available for ACA and General Surveys

We are pleased to announce the completion of two new integrated software systems for collecting market research data over the Internet: ACA/Web (Adaptive Conjoint Analysis for the Web) and CiW (a general survey administration tool for the Web). These two programs are components within a developing suite of Web-based software products that we are calling SSI Web.

These products have undergone one of the most intensive beta test programs ever completed for Sawtooth Software products. A few select firms began testing the software for commercial applications back in November, 1999. Since then, over 50 organizations have used the software. The demand for those desiring to become beta test sites far exceeded our expectations. It is a clear indication of how popular Web-based interviewing has become. We estimate that over 75 projects have been completed for over 100,000 separate interviews using CiW and ACA/Web since last November.

Tom Pilon (TomPilon.com), one of the beta testers, commented to us:

"Very few other web interviewing packages have capabilities such as advanced skip logic and randomization (random number generator, randomize questions within a page, randomize responses within a question) which are critical to serious researchers. Most other web interviewing packages that I have looked at are toys and not appropriate for serious researchers."

The Big Picture

The Internet is a fast-growing environment involving many diverse technologies. Internet servers which store the pages and files accessed over the Web come in different flavors, such as Unix, Apache, Windows NT, and Linux. People access the Web using different browser technologies, the most common being Internet Explorer and Netscape. We decided to develop software that could be run on just about any server and would display surveys that could be viewed consistently by just about any browser.

The process of creating a survey for the Web using our software is as follows:

The researcher uses a point-and-click Windows-based tool operating on his/her local PC to develop a questionnaire. No programming is involved, and the resulting Web pages are viewed using a Web browser running on the researcher's local PC. Standard survey questions (like numerics, open ends, select questions) and conjoint questions (ACA) can be included. To this point, no part of the survey development has involved the Internet.

When the researcher is pleased with the look of the questionnaire, he/she uploads (copies) the questionnaire file and the programs that run the questionnaire (Perl scripts) to an Internet server. This is done using a FTP (File Transfer Protocol) software, which also lets you specify the needed permissions on each file and directory on your server. (Note: In our users' experience, Microsoft FrontPage is not a good choice when doing this process, since it doesn't permit the full control that is needed.) The Internet server can be at the researcher's company, or can be at the ISP (Internet Service Provider).

Once the questionnaire file and programs are copied to the Internet server, the researcher tests the questionnaire by accessing it with his/her browser. If changes need to be made, the researcher makes the changes using the Windows-based interface operating locally on his/her computer, and then uploads the modified questionnaire to the Web server.

After the researcher has finished modifying the survey, respondents are invited to access and take the survey. There are many ways to do this, including email invitations, a clickable link from a corporate site, clickable banner ads, by phone interview, or even snail mail. Probably the most popular method among our beta testers has been email invitations. Email messages are sent to respondents, with a hyperlink (Web address) embedded in the email message. (You should use your own bulk email software to generate these email invitations.)

To: W. E. Coyote
From: Acme Company
Subject: Tell us what you think

Dear Mr. Coyote,

Thank you for your recent purchase of the X-13 rocket powered skates. We hope you are finding hours of enjoyment with this innovative new product. Will you please fill out a five-minute questionnaire regarding your experience with the X-13? Please click the following link: http://www.acme/survey/cgi-bin/ciwweb.pl?resp_name=1455&password=tr8h3 to provide your feedback.

Respondents click on the hyperlink and their browsers access the first page of the survey over the Web. (Note also that we included the respondent number and password information within the hyperlink, so that the respondent does not have to type the passwords manually.)

As respondents complete the survey, their data are stored in ASCII format on the Web server. The researcher can view the results (simple tabulations of select survey questions) in real-time. The data can also be downloaded to the researcher's hard drive for further analysis.

Password Protection

Posting your questionnaire on the Internet gives anyone connected to the Internet potential access to your survey. The imposter/repeater problem is one of the most oft-cited criticisms of on-line survey research. You can control the imposter/repeater problem by assigning passwords and quotas with the Internet Passwords Module.

Respondents must specify a User Name and/or a Password to start your survey. You can assign different password combinations to different sample sources to track the source of sample or limit the number of responses from different sample sources.

Assigning a quota of 1 completed survey per password lets respondents quit a survey part-way through and restart the interview at a later time at the point they left off. You can assign tens of thousands of passwords if needed. When using so many passwords, it is important that the software be able to rapidly search the list of passwords to verify that a respondent is qualified to take the survey. We have designed our software to process respondent passwords rapidly, to reduce the wait time between submitting a password and receiving the next page in the survey.

General Survey Capabilities

The CiW system includes many (but not all) of the capabilities of our popular Ci3 system.

Numeric question types (with response verification to accept values within a range)
Select question types (select one, all that apply, and drop-down combo boxes)
Open-end question type (with response verification to require a certain number of characters)
At user's control, all questions can require responses or be left blank
Ability to show graphics
Skip patterns, question rotation, and randomization of items within questions
Random number generator can be used to randomly assign respondents into groups that receive customized sets of questions.
Questions can be organized onto different pages
Previous responses can be displayed later in the questionnaire
Respondents can resume questionnaires that they prematurely terminate

On-Line Data Management Module

Having your questionnaire on the Internet is not only convenient for respondents, it is convenient for you (or your client) for checking the progress of the survey.

You can calculate marginals, download data, delete incomplete records, or change the quota controls for passwords to your survey. There are two classes of passwords you can assign to control access to the Data Management Module: Read and Read/Write.

You may wish to give your client Read Only capability, which would let the user tabulate marginals and view (but not download) the data file. Read/Write access lets you perform additional functions, such as download data, or adjust password quotas.

Example Internet Survey

We invite you to view an example ACA/Web and CiW survey by visiting www.sawtoothsoftware.com/acanet/login.htm .

Technical Support

We provide free technical support for the software, with the exception of implementing the server-side aspects of your survey. Many Internet servers follow general conventions. But because each server can be configured differently, we require that the support for using that server come from your ISP or from your internal IS support staff.

Given the complexities of Internet servers, bandwidth issues, security, and bulk emailing, SSI Web is a more complicated software system to use than our typical market research software (e.g. CBC, ACA, Ci3). You should make sure that you have the appopriate resources available at your company to manage these challenging issues.

Prices

CiW comes in different sizes, depending on the number of questions in the survey. The prices are as follows:

CiW/50 questions    $3,000
CiW/100 questions   $5,000
CiW/250 questions   $7,000
CiW/500 questions   $9,000

ACA/Web includes CiW capabilities for 10 questions. ACA/Web is sold depending on the number of attributes in your study.

ACA/Web 10 attributes    $3,000
ACA/Web 30 attributes    $6,000

There is a small discount for current ACA users (v4.5).

Future Directions

We expect that Web-based market research will continue to grow in popularity. Sawtooth Software is committed to an aggressive program of enhancing CiW to manage more complex questionnaires, more in-depth realtime reporting, and to offer a greater variety of question types and formatting flexibility than version 1 of CiW supports. Future versions of SSI Web will accommodate not only ACA and CVA conjoint questionnaires, but CBC (Choice-Based Conjoint) and perhaps even CPM (perceptual mapping) surveys as well.

Go back to Index

New Alliance Formed with MarketTools

Sawtooth Software and MarketTools (www.markettools.com) have formed a strategic alliance to integrate specialized versions of Sawtooth Software's analytical tools within Market Tools' on-line ASP (Application Service Provider) offering. MarketTools has developed a product called zTelligence (www.zTelligence.com), a "one-stop" ASP solution for collecting, analyzing and reporting market research and relationship information over the Web. zTelligence is built on a powerful array of multiple high-speed Web, database and application servers offering superior speed, bandwidth and security.
With zTelligence, the software, data and reporting tools reside on MarketTools' servers rather than on the user's PC. Internet surveys can be developed and deployed literally within minutes by users with no programming or Web expertise. Users compose and launch surveys using their own Web browser connected to zTelligence's point-and-click survey designer. Sample can be uploaded or purchased through MarketTools or its partners who specialize in Internet survey sampling.
This alliance provides an additional option for Sawtooth Software customers interested in conducting market research and storing/sharing the resulting data over the Web. We carefully considered the impact that forming this relationship might have upon our users. It is important to us to be a resource to our general user base rather than to compete with our customers. This alliance supports this vision. MarketTools is not a full-service market research provider. Their focus is to provide technology for creating market research surveys, saving those data, and providing enterprise-wide applications for accessing, analyzing and reporting the data. Their approach is very similar to ours: to provide researchers, consultants, and managers with better market research tools to do their work.
Go back to Index

Call for Papers - Ninth Sawtooth Software Conference

Just short of a year from now, we will hold our ninth Sawtooth Software Conference, on September 12-14, 2001 in Victoria, British Columbia, Canada. Our research conference brings together some of the best minds in our industry to talk about practical issues in computer/Web interviewing and quantitative market research. It is not a sales-oriented event, but a chance to exchange ideas and receive education from a variety of sources and perspectives. Papers presented at our previous Sawtooth Software Conferences are cited frequently in journal articles.
We're looking for exceptionally strong papers. If you'd like to be on the program, please respond promptly (email: bryan@sawtoothsoftware.com) with a one-page abstract describing your proposed paper, with special attention to what the audience will "take away" from the presentation. We are interested in papers on a variety of subjects, including Web interviewing, market segmentation, customer satisfaction modeling, conjoint/choice analysis, perceptual mapping, hierarchical Bayes methods, forecasting, pricing research, market simulations and case studies. In an effort to provide more balance to the program, we will favor papers that are NOT about conjoint/choice modeling.
To be accepted, a paper must show promise of being sufficiently practical to be of use to the least sophisticated members of the audience, while having enough substance to be of interest to the most sophisticated members. In addition to standard presentation slides, authors are required to submit a journal-quality written paper for publication in the Conference Proceedings.
Go back to Index

Staying out of Trouble with ACA

About five years ago, we published an earlier version of this article in Sawtooth Solutions. We've updated that article here and have condensed it somewhat. We hope this advice will help ACA users avoid the pitfalls and produce consistently good results.
Though we've been told that ACA is remarkably easy to use, we frequently talk with ACA users who have run into a problem of some kind. We thought it might be helpful to list the problems responsible for the most frequent customer support calls.
Using too many prohibitions: ACA lets you specify that certain combinations of attribute levels shouldn't occur together in the questionnaire. But if you prohibit too many combinations, ACA won't be able to produce a good design, and may fail altogether. You can present combinations of levels that do not exist in the market today, and including unusual combinations can often improve estimation of utilities. During market simulations, you can avoid those combinations that seem unusual. In short, prohibitions should be used sparingly.
Reversing signs of ordered attribute levels: If you already know the order of preference of attribute levels, such as for quality or price, you can inform ACA about which direction is preferred and avoid asking respondents those questions. You inform ACA by means of the a priori settings: Increasing (worst to best) or Decreasing (best to worst). A common mistake is to accidentally specify the wrong order, which can lead to nonsensical data that can be difficult to salvage. To avoid this situation, take the interview yourself, making sure that the questions are reasonable (neither member of a pair dominates the other on all included attributes). Also, answer the pairs section with mid-scale values and then check to make sure the utilities are as you expect them. If you do happen to mis-specify the a priori order of levels, the ACA/HB module can be used quite effectively to recompute the utilities and help salvage the situation.
Using ACA for pricing research when not appropriate: There are three aspects to this point.

All "main effects" conjoint methods, including ACA, assume that every product has the same sensitivity to price. This is a bad assumption for many product categories, and CBC may be a better choice for pricing research, since it can measure unique price sensitivity for each brand.

When price is just one of many attributes, ACA may assign too little importance to it. In a few previously published articles, researchers have reported that it may sometimes be appropriate to increase the weight that ACA attaches to price. This is particularly likely if the author includes several attributes that are similar in the minds of respondents, such as Quality, Durability, and Longevity. If redundant attributes like these are included, they may appear more important in total than they should be, and other attributes, such as price, may appear less important than they really are. This problem is exacerbated if a wide range for price is specified.

It is not a good idea to use the "share of preference with correction for product similarity" with quantitative variables such as price. Suppose there are five price levels, and all products are initially at the middle level. As one product's price is raised, it can receive a "bonus" for being less like other products which more than compensates for its declining utility due to its higher price. The result is that the correction for product similarity can lead to nonsensical price sensitivity curves. This problem also can occur (but typically to a lesser degree) when using the improved method for dealing with corrections for product similarity: Randomized First Choice (RFC). We suggest conducting sensitivity analysis with the Share of Preference method when modeling demand curves for quantitative attributes like price.
Using unequal intervals for continuous variables: If you use the ranking rather than the rating option, ACA's prior estimates of utility for the levels of each attribute have equal increments. That works well if you have chosen your attribute levels to be spaced regularly, for example with constant increments such as prices of .10, .20, .30, or proportional increments such as 1 meg, 4 megs, or 16 megs. But if you use oddly structured intervals, such as prices of $1.00, $1.90, and $2.00, ACA's utilities are likely to be biased in the direction of equal utility intervals. This problem can be avoided if you use the ACA/HB module to compute ACA utilities.
Including too many attributes: ACA lets you study as many as 30 attributes, each with up to 9 levels. But that doesn't mean anyone should ever have a questionnaire that long! Many of the problems with conjoint analysis occur because we ask too much of respondents. Don't include n attributes when n-1 would do!
Including too many levels for an attribute: Some researchers mistakenly use many levels in the hope of achieving more precision. ACA can only study 5 levels in detail, and when there are more than 5 levels, ACA must make assumptions about the others. With quantitative variables such as price or speed, you will have more precision if you measure only 5 levels and use interpolation for intermediate values. If you must measure more than 5 levels, we strongly encourage you to use the ACA/HB module for estimating utility values. It can do a much better job at measuring those levels that are not studied in detail by any one respondent.
Abuse of unacceptables: ACA lets you include an "unacceptables" section, in which respondents are permitted to identify features so unattractive that products with those features would never be considered. Those attribute levels are excluded from the balance of the interview. Unacceptables provide a way to shorten interviews that would otherwise be too long, but respondents are too willing to discard levels as "totally unacceptable." We suggest avoiding use of unacceptables. If you do decide to use unacceptables, the ACA/HB module provides a superior way to deal with their estimation.
Interpreting simulation results as "market share": Conjoint simulation results often look so much like market shares that people sometimes forget they are not. Conjoint simulation results seldom include the effects of distribution, out-of-stock, or point-of-sale marketing activities. Also, they presume every buyer has complete information about every product. Researchers who represent conjoint results as forecasts of market shares are asking for trouble.
Not including adequate attribute ranges: It's usually all right to interpolate, but usually risky to extrapolate. With quantitative attributes, include enough range to describe all the products you will want to simulate. It is a good idea prior to data collection to ask the client to list the product scenarios that should be investigated in market simulations. This exercise can often reveal limitations or oversights in your attribute level definitions.
Imprecise attribute levels: We assume that attribute levels are interpreted similarly by all respondents. That's not possible with "loose" descriptions like "10 to 14 pounds," or "good looking."
Attribute levels not mutually exclusive: Every product must have exactly one level of each attribute. Researchers new to conjoint analysis sometimes fail to realize this, and use attributes for which many levels could describe each product. For example, with magazine subscription services, one might imagine an attribute listing magazines respondents could read, in which a respondent might want to read more than one. An attribute like that should be divided into several, each with levels of "yes" and "no."
Go back to Index

Using Conjoint Analysis to Recommend Products and Close Sales over the Web

Over the last three decades, conjoint analysis has blossomed as a valuable technique to measure people's preferences for products and services and to predict how markets might react to different product offerings. In 1985, Sawtooth Software created one of the first commercially available software package for conjoint analysis, called ACA (Adaptive Conjoint Analysis). ACA soon became the most widely used conjoint research technique and software system (based on industry-wide studies in 1992 and 1997).
Early in our company's history, we recognized that the statistical techniques we were developing to understand people's complex preferences could have wider application than just strictly for marketing research. We felt that conjoint (trade-off) analysis could help buyers sort through many product offerings to make quicker and better decisions in complex decision-making environments. For example, in the 1980s, we developed a computer based system for a Real Estate client, to provide a way of matching home buyers with homes for sale in their area that would be expected to most appeal to them. It seems that the Real Estate world wasn't ready for that kind of system then, and resistance by Realtors doomed the project. (About that same time, we also made early attempts for automobile purchases, college selections, financial products and group executive decisions.)
Times have changed. The Internet has revolutionized the way we do business and communicate with customers. Our economy is increasingly both comfortable with and dependent on technology. Buyers have become empowered by the amount of information available at their fingertips over the Web. More than 50% of homes in the US are online. But with the proliferation of information, buyers are experiencing information overload that is actually impeding their ability to make decisions about some purchases. It is not surprising that a number of websites have appeared to help buyers make decisions about products.
Many sites like Amazon.com use a technology called Collaborative Filtering, which recommends new books to customers based on their previous purchases and what other customers with similar tastes have purchased. Collaborative filtering requires large amounts of data from many buyers. These systems cannot make recommendations for new products or when few customers have purchased/evaluated a product. But, they have the advantage of recommending products to buyers based on other like-minded individuals' choices.
Other sites let respondents specify the kinds of products they are interested in buying using cutoffs or acceptable ranges. A query to the database of available products returns potential candidates. Often times, buyers place too many restrictions, thus disqualifying all available products. We have seen this happen within the context of ACA's Unacceptables section. We have found that respondents are quick to say that certain levels of an attribute are unacceptable, when in reality they are willing to bend those rules if enough other aspects of the product are very desirable. Also, a cut-off based decision rule provides no way of ranking qualified products based on overall suitability.
Still other sites collect respondent preferences, through self-explicated models or with conjoint analysis, and then use those preferences to recommend products that might satisfy the buyer. Preference models have the advantage of being able to recommend new products that other buyers have not yet experienced. They also can prioritize products better for respondents by sorting them from most likely to be preferred to least likely.
A few academics have published research regarding the use of electronic or on-line recommendation agents. So far, the results are very encouraging. In a recent article by Gerald Haubl and Valerie Trifts of the University of Alberta, the authors conclude:
"In sum, our findings suggest that interactive tools designed to assist consumers in the initial screening of available alternatives and to facilitate in-depth comparisons among selected alternatives in an online shopping environment may have strong favorable effects on both the quality and the efficiency of consumers' purchase decisions in online shopping environments--shoppers are able to make much better decisions while expending substantially less effort. This suggests that interactive decision aids have the potential to drastically transform the way in which consumers search for product information and make purchase decisions."
("Consumer Decision Making in Online Shopping Environments: The Effects of Interactive Decision Aids," Haubl and Trifts, Marketing Science, Winter 1999.)
The goals of Web-based recommendation agents are to:

personalize the process for the buyer,
help the buyer efficiently sift through large amounts of information to identify the products/services that are best for him/her,
instill the buyer with greater confidence in the decision and subsequent purchase,
increase sales,
provide feedback to the seller regarding the market's preferences for features.

As marketing scientists, we are excited about the opportunities in this area. Our work involves a fascinating blend of science and business strategy. We sincerely enjoy what we do and know the techniques we develop lead to better products for customers and more profitable industries.