12 Steps to a Successful Web-Based Conjoint Survey

The following article was submitted by Patrick Delana and Zach Curtis, POPULUS, Inc. We invite other users to submit case studies or other articles as well.

Recently a major communications firm commissioned us to determine demand for various configurations and pricing packages for its high speed data (HSD) services. The measurement objectives clearly called for some sort of conjoint analysis. Because there were relatively few attributes and because pricing was a key objective, we focused on either full profile conjoint (Sawtooth Software's CVA) or Choice-based Conjoint (Sawtooth Software's CBC).

The client's budget was modest; there were only four weeks until the analyzed findings were due. The stimuli were sufficiently complex that telephone interviewing would not be possible. Computer assisted self-administered interviews (CASI) were considered but there were not qualified field agencies located within all of the six geographic regions required by the client. Further the budget would not permit recruiting current Internet users for central location interviews.

The Internet was a logical means to interview people about an Internet service. So we chose to implement the study using Sawtooth Software's CVA Internet Module. It was a learning experience for us and we would like to share that learning with other Sawtooth Software users.

Here are the steps that we followed and recommend:

  1. Use Sawtooth Software's CVA System Internet Module. The Windows-based system uses a template, fill-in-the-blank approach that makes questionnaire development very straightforward (easier than Ci3). We used an Internet service provider (ISP) that gave us the permissions necessary to run the interview on its server. (Some ISPs do not permit users to run programs or to collect and store data on their servers.)
  2. Study a universe appropriate for a web-based survey. Despite higher claimed estimates, real web access penetration is only about 20%.
  3. Obtain an appropriate sample. In our recent study, we focused on only 5% of ZIP codes; these reflected the geographic areas in which our client would soon launch its service. Survey Sampling (www.ssisamples.com) was able to provide names and phone numbers of Internet subscribers within our targeted areas.
  4. Recruit and screen participants by telephone. Recruiting can be accomplished by a brief CATI survey, screening potential respondents for Internet access along with topical and security screens. A successful recruit resulted in verifying the respondent's name and obtaining e-mail addresses. Aiming for a final survey sample of 500, POPULUS obtained the names and e-mail addresses of 1,000 qualified respondents, assuming 50% cooperation for a completed interview.
  5. Train interviewers to properly record an e-mail address. Even the best of interviewers are accustomed to record open-end answers for meaning rather than precise wording. Interviewers must be instructed to read back an e-mail address, character by character.
  6. Send personalized e-mails within 24 hours to each recruited respondent. Otherwise people can quickly forget what they've promised to do.
  7. Create a unique password allowing each respondent access to the web site. The Sawtooth Software CVA Internet module allows for the creation of passwords. Respondents use the password to begin the survey. If necessary, a respondent can leave the survey and use the password to resume the survey at a later time. However, once a respondent had completed the survey, the password is rendered inoperable, preventing repeated access to the survey site.
  8. Use an e-mail package such as MailKingĀ® (http://www.mailking.com), a software program used to send personalized e-mail messages to each respondent. Each e-mail message should contain a hyperlink to the survey web site and the respondent's unique password. Each morning, simply load the results of the previous evening's recruiting into a spreadsheet, add a password to each record, and MailKing does the rest.
  9. Offer a generous incentive. In our case, respondents were informed that of those who completed the Web survey, one person from each of the six service area cells would be randomly selected to receive a check for $100. Notify winners via e-mail.
  10. Assume a 50% cooperation rate. POPULUS sent out 1,057 e-mail messages along with follow-up messages. Of these, 162 (15%) were returned as undeliverable. Completed surveys were obtained from 482 respondents within two weeks of the first e-mail mailing.
  11. Monitor the site regularly: daily, even hourly. It's easy to keep clients up-to-date regarding the number of completed web site interviews.
  12. Download interim data frequently. Use the Sawtooth Software DOS CVA program for the conjoint analysis and any statistical package for the rest of the data. Schedule the top-line meeting with your client the day after the survey site is closed.

The project was completed on time and on budget. The client's only concern was a reluctance to utilize the CVA simulator. "Wouldn't it be easier for you just to do the simulations for us?" he asked. "Just give us fifteen minutes," we answered. The program and data files were zipped up, appended to an e-mail, and installed five minutes later. After less than ten minutes of instruction in the simulator's use, we heard what we have heard many times before: "Wow, this is really neat. Thanks very much!"