|
|||||||
Technical Papers| Home |
Solutions |
Products |
Downloads |
Services |
Education |
Support |
|  SS Fall 2005Sawtooth Software Conference 2006We are pleased to announce the program for the 2006 Sawtooth Software Conference, to be held in Delray Beach, Florida, March 29-31, 2006.Our conferences are well-known for their practical focus, friendly environment, accessible presentations, and excellent food. They are not sales events for our software, but forums for discussing a variety of issues related to conjoint/choice analysis, computer/web interviewing, and other quantitative methods. Typically about one-fifth of the attendees are not Sawtooth Software users. To register for the conference, or to view more details (including abstracts), please visit www.sawtoothsoftware.com. The conference registration is only $750 for the 2.5-day event ($900 after Feb 1). Preliminary Program Outline:
Wednesday, Mar 29, 2006
Thursday, Mar 30, 2006
Friday, Mar 31, 2006
Optional Half-Day Tutorials (Tuesday, Mar 28, 2006) These four-hour tutorials provide an opportunity for more in-depth training. The cost is $195 for one tutorial, or $320 for two tutorials (add $25 each after Feb 1).
Morning:
Afternoon:
 Go back to IndexRich Johnson Honored in Latest JMR IssueThe latest issue of Journal of Marketing Research (JMR, August 2005) honored our chairman and founder, Rich Johnson, through a series of articles. Rich was invited to write his own short history, covering his early schooling and training, and subsequent career successes, “A Career between Theory and Practice.” Joel Huber of Duke University wrote a short article entitled, “What Has Marketing Learned from Richard Johnson?” Paul Green wrote a short tribute: “Theory and Practice Go Hand in Hand: A Tribute to Richard Johnson’s Contributions to Marketing Research Methodology.” Finally, Rich wrote a short piece encouraging greater cooperation between the academic and practitioner community, entitled “In Favor of Closer Ties.”With permission from JMR’s editor, this article is entirely composed of highlights from these articles. Interested readers may refer to the full texts from the August 2005 issue of JMR. Richard M. Johnson: I have had an interesting and rewarding career in marketing research that has been somewhere on the boundary between the academic and the practical. A lesson from my experience is that young people should keep their minds open about career choices and avoid foreclosing any possible paths of development. They may not learn until later in life what their most satisfying calling is. In the fourth grade, I encountered long division, which was taught as a rote procedure that consisted of repetitive steps with no apparent rhyme or reason to them. I was completely baffled. In later years, I realized that though I am reasonably good at figuring out what to do if I understand the underlying principles, I am no good at memorizing what appear to be arbitrary facts or procedures. I recall that I was convinced that I should henceforth avoid all unnecessary contact with numbers. I pursued a premedical program (at Haverford College) with a major in psychology. In my senior year, I took freshman math, which was a stunning experience. The course, which Cletus Oakley taught, was different from anything I had ever encountered. The basic principles were clear. The details followed naturally from the principles. I realized that I had misperceived the basic nature of mathematics, and I had come close to missing out on a subject for which I had a deep appreciation. A few months later, I was accepted at medical school, but I decided to pursue a graduate education in psychology. In 1956, I entered the doctoral program at the University of Washington. I was attracted to the psychometric program offered by Paul Horst. Horst had been a student of Thurstone. My most useful graduate instruction was a series of courses that Horst taught in matrix algebra: these courses provided me some of the most valuable tools that I have used throughout my career. Graduate students had no access to computers, but I became adept at using hand-operated mechanical calculating machines. For a summer project, I used such a machine to compute a 30 x 30 correlation matrix for an assortment of variables on patients in a mental hospital, and then I conducted a factor analysis of those correlations. Today, a personal computer could perform these calculations in a few seconds, but it took hundreds of hours of manual labor. Although this was tedious, it gave me an understanding of the basic processes involved in factor analysis, and it proved useful in later years. One career path for students of psychometrics was the development and validation of tests to aid in the selection of employees for particular types of industrial jobs. Horst suggested to his former colleagues at P&G that they take me on as a personnel psychologist. I arrived at P&G at about the same time as the first large-scale computer, an IBM 705. The computer was intended for accounting work rather than for psychometric research. However, there was hope that it might be available occasionally for statistical analyses, and several employees with statistical training became involved in trying to harness its capabilities. Among my first activities at P&G was the design of programs for regressions analysis, factor analysis, and other statistical procedures that could be run on the IBM 705. The early 1960s were an exciting time because the rapid expansion of computing capabilities made many things possible that had previously been unimaginable. I worked on many interesting problems in personnel psychology, but marketing research soon captured my interest. Now, I was able to use multivariate techniques such as perceptual mapping and cluster analysis with moderately large numbers of variables. Although I treasured my relationships with many excellent colleagues at P&G, I became frustrated with the sheer inertia of such a massive organization. I decided to try my hand at consulting, and in the late 1960s, I resigned from P&G. I found a firm that could provide computer access, and I busied myself as an applied statistician, computer programmer, and model developer. One of my clients was Chicago-based Market Facts, Inc. (MFI). My work with MFI expanded so rapidly that the company believed that it was uneconomic to continue the trend. I began a period of employment with MFI that would include the most productive years of my professional life. One of the problems that interested me was how to make use of product-ratings data to produce maps of product categories that described not only customer perceptions of products but also densities of customer ideal points in the same space. MFI conducted dozens of large projects using this approach in a wide variety of product categories, and there were many instances of repeat business from the same clients. Another thread of development work led eventually to the conjoint method known as adaptive conjoint analysis (ACA). Market Facts had a client in a durable goods business. Whenever this company seriously contemplated a new or modified product, a concept test was performed. The client was responsible for conducting the concept tests and answered to a product manager who commissioned them. However, before the client could report the results, the product manager would say, “Sorry, we did not have time to tell you about this; instead of two handles, it is going to have one, and instead of 20 units per minute, it will produce 22. Can you test that one in the next three weeks?” The client found there was never time to do the required concept tests quickly enough to affect the product design cycle, and thus the client came to MFI with what was considered an urgent problem—that is, the need to find a way to test all future product modifications at once. The client wanted to be able to tell the product manager, “Oh you say it is going to have one handle, with 22 units per minute, weight 30 pounds, and be green?” Well, that product would receive X share points. Any other questions?” Today, this is recognizable as a conjoint analysis problem, but Green and Rao had not yet published their historic article (1971). An answer to this problem is to consider a product a collection of separate attributes, each with a specified level. A new method of questioning was required to elicit information about values of attribute levels, and a new estimation procedure was required for converting that information into “utilities.” The solution came to be known as “trade-off analysis.” To collect data, respondents were presented with several empty tables, each crossing the levels of two attributes, and they were asked to rank the cells in each table in terms of preference. To estimate what are now called partworths, a nonmetric regression algorithm was used to find a set of values for each respondent. Although much was learned about how to improve the technique for future applications, this first study, which was conducted in 1970, was a success. In the early days, there was less communication between practitioners and academics than that which is enjoyed today. My early work at MFI was done almost in a vacuum and without the knowledge that a similar stream of development was taking place with Paul Green and his colleagues. Green and Srinivasan called such procedures conjoint analysis, and as time passed, it became clear that trade-off analysis was just a different variety of conjoint analysis. When time sharing and CRT (cathode ray tube) terminals first became available, I was excited about the possibility of using them to enhance the quality of market research interviews. I still remember an experience at MFI when I arranged a meeting with the company’s management to demonstrate the radical idea of computer-assisted interviewing. I had borrowed the most cutting-edge CRT terminal of the time, which consisted of a tiny three-inch screen in a large cabinet. I had shrouded the CRT with a cloth so I could introduce the idea of computer-assisted interviewing without distraction. The meeting went well until the unveiling, when, with a flourish, I removed the cloth to reveal the CRT. When management saw the tiny screen in the enormous cabinet, everyone in the room began to laugh, and they continued laughing until I ended the meeting. Fortunately, CRT terminals improved rapidly, and it was not long before computer-assisted interviewing became entirely feasible. By the mid-1970s, computer technology had advanced sufficiently that it became feasible to conduct computer-assisted trade-off analysis using pairwise questioning. The questioning format was dramatically easier for respondents than filling out trade-off matrices. A U.S. military service branch commissioned a project to study various recruiting incentives. We purchased what was then described as a “minicomputer”; it filled only a small room rather than a large one. Respondents used CRT terminals at interviewing sites around the United States that were connected to a central computer by telephone lines. The data turned out to be of high quality, and the study was judged to be a complete success. That study marked the end for the trade-off matrix. After ten years with MFI, I left the firm in 1978. Curt Jones and I founded the John Morton Company. Our plan was to apply emerging analytic techniques in a strategic marketing consulting practice. We purchased several dozen Apple computers, and this began a fascinating adventure of using them all over the world in many languages and in product categories of almost every description.
In 1982, I retired from the John Morton Company, and moved
Sawtooth Software’s first product was Ci2, which permitted researchers with no previous programming experience to compose and administer computer-based interviews. The second product was ACA.
In the early 1990s, it seemed that choice studies were likely to become more popular. Sawtooth Software was able to offer its CBC System for “choice-based conjoint” analysis. In 1995, Sawtooth moved to Sequim, Wash., which offered employees a comparable quality of life while avoiding the increasing cost of living in Sun Valley.
I believe that I have had a singularly satisfying career, though completely different from what I might have expected as a youth. I have worked as an applied mathematician, yet throughout my early schooling I was terrified of mathematics. I have flirted several times with the idea of trying academic life, but on each occasion, I have decided to remain a practitioner. I will probably always wonder whether I could have succeeded as a professor and whether such a path would have been as interesting as the one I pursued.
What success I have had has been due to my being positioned between theory and practice. Not being in an academic setting and being unaware of much prior academic work, I have had to invent my own approaches, many of which had already been invented by others. In retrospect, I might have accomplished more if I had been more of a scholar and less of an inventor, but that would not have been as much fun!
Joel Huber: Thanks to the efforts of Richard Johnson, the way marketing research is thought about and done has changed. Rich has made contributions in basic theory and applied knowledge, but most important, he has been a critical catalyst in bringing so many ideas into general use in the market research community. Sawtooth’s offering for choice-based conjoint (CBC) analysis made choice experiments accessible in ways that were possible only with state-of-the-art customized procedures. Perhaps the cleverest aspect of that product was the way it finessed the issue of choice design by selecting random but balanced choice sets for each individual. The net result is a program that is classic Johnson: It is efficient, robust, and perhaps most important, accessible to a broad range of researchers.
The other element that made the CBC system particularly valuable to the marketing research community was the availability of hierarchical Bayesian analysis. Richard Johnson had no formal training in Bayesian estimation, but he gained expertise on the procedure from pioneer researchers Greg Allenby and Peter Lenk. Richard Johnson took one particular Bayesian model and developed it to estimate the results of CBC. By limiting its applicability, he was able to increase the speed of convergence by an order of magnitude. In addition, the Sawtooth community performed many simulations and tests that have enabled the development of standards and procedures that transformed Bayesian estimation from a black art available only to the tutored few to a standard technique available to all.
Part of what is different about Rich is his fearlessness. The other part comes from his strategy of having a solid base that supports his risk taking. This support base includes both consulting clients and Rich’s intensely loyal and talented coworkers. It also comes from a group of academic researchers that include John Hauser, Dick Wittink, Peter Lenk, Greg Allenby, Paul Green and myself.
Perhaps the most critical supportive base is that of Sawtooth Software and its loyal users. Their bonding and mutual support play out in meetings and on their Web site. The open and sharing culture of that organization also contributes to its effectiveness. Sawtooth’s culture is unlike that of many market research companies, which are understandably reluctant to limit returns from their investments in methodology by making it easy for others to copy or criticize. Using a different, more open model, Sawtooth has become the dominant worldwide supplier of conjoint software to marketing researchers. The Sawtooth Software meetings share this transparency by inviting as speakers competitors or people with findings that are unfavorable to Sawtooth.
Paul Green: One of the most interesting features of Rich’s work is his ability to find new ways to solve “old” problems, whether they are in multidimensional scaling, conjoint analysis, or other areas of interest. I first noted examples of this skill in his early work in perceptual mapping and clustering. There are often twists that come from his fertile mind and have pragmatic value for applications. With his widely diverse skills and interests, it was only natural that he put these talents to work. Not content with his already important contributions to psychometrics, Rich and his associates started Sawtooth Software, a unique firm that specializes in psychometrics and related marketing research tools.
Rich remains an active and novel thinker and doer. (It would be difficult to imagine him as placid and content the role of country squire.) Rich is a rare blend of scientist, entrepreneur, risk taker, and sportsman. Rich and I are also both amateur piano players. Unfortunately, he is too shy about this nonrugged talent. P.S. I’m still waiting for a piano duet.
Richard M. Johnson: I would like to thank Paul Green and Joel Huber for their generous comments. There are no colleagues whose approval I value more.
I have heard academic colleagues describe particularly brilliant students, and when I have proposed that these students consider careers in applied research, my academic friends have shuddered at the thought. They believe that such students would be wasted in applied settings. Paul Green’s productivity is admirable, and I believe that he was aided by previous experience in an applied setting. Joel Huber observes that the academic side is becoming more specialized, and the practitioner side is becoming more proprietary. That presents a problem for both sides, and I believe something should be done about it.
I would like to encourage my academic colleagues to suggest that their best students plan to spend at least some time as practitioners. I believe that experience would give these students valuable perspectives and inventories of methodological challenges that could guide them in future academic work. I would also suggest that qualified practitioners be encouraged to spend parts of their careers in academe. This should not only provide them with increased knowledge but also build relationships from which both sides could benefit in the future. If such opportunities had existed years ago, I would surely have been interested in them, and I believe that these different kinds of experiences would have made me more productive.
We are pleased to announce that we are working very hard on a new software system called the MaxDiff System. It will be an integrated component within the SSI Web system. Users may conduct MaxDiff studies over the Web or via CAPI (computers not connected to the web) using this system. For paper-and-pencil implementation, we have been offering an experimental design program that includes a tool that prepares the data for estimation using CBC/HB software.
MaxDiff scaling is a trade-off method for measuring the importance or preference for multiple items, such as brands, product features, political platforms, advertising claims, etc. Any time you are considering using a rating scale, ranking scale, or constant sum scale for multiple items, you can consider using MaxDiff.
Here’s an example MaxDiff question, involving importance of server features:
Even though we haven’t offered a complete solution for MaxDiff studies yet, many of our users have been applying the technique using a combination of separate tools: our Best/Worst Experimental Designer (for generating questionnaire plans), and the Latent Class Module or CBC/HB Module for estimating item scores. The new MaxDiff system will integrate design of experiments, questionnaire programming, and quick estimation using hierarchical Bayes (HB) analysis all within SSI Web’s menu system.
We expect to complete this software in the fourth quarter.
More powerful student (lab) licenses ($1,000 covers entire school)
Previously, Choice-Based Conjoint (CBC) was limited to 3 attributes and 50 respondents for the academic lab licenses. Adaptive Conjoint Analysis (ACA) was limited to 5 attributes and 50 respondents. The new lab licenses support up to 10 attributes and 250 respondents, allowing you more flexibility to conduct non-commercial academic research. The lab system also includes capabilities for traditional full-profile conjoint (CVA) and perceptual mapping (CPM).
Until now, we haven’t offered a student lab version of our general web interviewing platform (CiW). Now, we offer a 500 data field version of CiW for students and professors to use on non-commercial academic research projects.
Research-on-Research Student Panel
There is a long history of involving students in methodological research projects. At Sawtooth Software, we would like to increase the amount of methodological research we are able to do on a variety of techniques, including conjoint/choice analysis and scaling. As a benefit to the academic community, we plan to offer scholarship opportunities for masters or Ph.D. students also to utilize the panel for promising research projects.
Please invite your students to join the Sawtooth Software Research-on-Research Student Panel, by visiting http://www.sawtoothsoftware.com/education/panel [no longer accessible]. We plan to survey panelists no more than 8 times per year. Students may enroll or unsubscribe at any time. We will not sell or rent this list, and will adhere to a strict privacy policy that is prominently displayed at each stage of participation in the panel.
Scholarship opportunities
We are certain that there are very bright masters and Ph.D. students who might rise to the occasion if offered a scholarship opportunity. As mentioned above, we would like to entertain proposals for methodological projects to be fielded among the Student Panel we are building. We are also looking for papers to be presented at our Sawtooth Software Conference, held about every 18 months. We are willing to fund travel and hotel expenses up to $1,000 and provide free registration into our Conference for a promising student that sends us a worthy paper to consider.
Twice a year (May 15 and November 15), we will award free software to two either masters or Ph.D students. Proposals are due April 30 and October 30 for consideration.
Free PowerPoint(TM) materials and curriculum suggestions for download
We offer free materials online for teaching conjoint analysis, including white papers. Please feel free to view these resources and adapt them to suit your curriculum. Visit: www.sawtoothsoftware.com/education/academicaids.shtml.
We are pleased to announce the availability of an introductory book on conjoint analysis by Sawtooth Software VP Bryan Orme entitled, “Getting Started with Conjoint Analysis: Strategies for Product Design and Pricing Research,” and published by Research Publishers LLC. This book is meant to fill the gap between heavier academic works and the needs of students and practitioners new to conjoint analysis.
This 164-page book includes a short foreword by “father of conjoint” and academic legend, Paul Green. Green writes: “Getting Started with Conjoint Analysis is a practical, no-nonsense guide to what happens when one designs, executes, and analyzes data from real marketplace problems. It should appeal to academics and consultant-practitioners alike. The book is easy to follow, while at the same time being almost encyclopedic in its coverage of topics ranging from study design to the presentation of results to clients.”
Objective
The objective of the survey was to provide the German public with a forum to voice their opinions about social and political topics. The survey was a joint initiative by a large German management consulting firm, the internet version of a large-circulation weekly magazine, and one of the largest internet providers in Germany.
The huge number of participants in the study owes to a broad media campaign. Numerous advertisements in print media and the internet advertised the survey as a way for Germans to express their opinion about economic reforms that would address the country’s pressing growth and unemployment problems. The media campaign stressed that the results would be broadly communicated, would be brought to the attention of key decision makers, and would be used to spark public debate. Therefore, we believe, many participants saw the survey as a unique opportunity to express their opinion about the most pressing public policy issues in Germany.
Study Design
The online survey was split into a core block that was presented to all participants and four theme blocks that were randomly assigned to participants. The four theme blocks covered the following areas: (1) work and leisure, (2) education, (3) family, and (4) civil engagement. The random-block design was chosen for two reasons. First, this design reduced average completion time for the online survey. Second, the random-block design reduced the risk of strategic self-selection into blocks that were of special interest to participants.
Technology Challenges and Testing
TNS Infratest faced the problem of conducting a conjoint study with five separate conjoint designs where the main block would have 500,000 respondents and the four theme blocks would have about 125,000 respondents each. In addition, the client wanted assurance that 8,000 interviews could be conducted simultaneously. Unfortunately, we found no one who had ever run such a large online survey and could tell us if the different software modules would handle such large numbers of respondents. It seems that we were pioneers in the field, conducting such a large conjoint study with half a million interviews over a relatively short time.
Our first decision was to split the study into four surveys covering the four theme blocks and also to run the main conjoint as well in four replicated surveys to avoid the risk running into some limitation of the data collection module. Extensive tests with Sawtooth Software’s SSI Web conjoint module for CBC encountered no problems managing such large samples. Also the export function to prepare the CHO files for utility calculation showed no problems with more than 500,000 test datasets.
The second problem was to find out if it was possible to handle 8,000 requested interviews at the same time. The IT department at TNS Infratest determined that a server cluster with six Pentium 2.8 GHz computers in combination with special load balancing software would be able to handle the load of 8,000 simultaneous CBC interviews.
Crossing Our Fingers
In October 2003 fieldwork started, and we weren’t quite sure what would happen over the following weeks—what aspects we forgot to test, what pitfalls were on the way. But it seems that we had good luck and all the hardware and software worked quit well. By the end of December 2003, over 870,000 respondents had started the conjoint in the main block and more than 500,000 had completed the survey.
During this time, we conducted several tests to figure out how many splits and what computers would be needed to run the hierarchical Bayes estimation. We thought the main conjoint block could be an HB run with all 500,000 respondents together. Several tests had shown that Sawtooth Software’s CBC/HB module would have absolutely no problem computing utility values for 500,000 respondents at the same time (from one enormous CHO file). Tests with different computers showed that an AMD Dual Opteron PC with a workspace of 2GB and the beta version of Microsoft Windows XP Professional x64 Edition performed best. The Sawtooth code doesn’t use the multiprocessor capabilities, but enabled us to simultaneously run the four theme block conjoints on one Opteron CPU while the other CPU was running the main conjoint. Due to that parallel processing, the utility scores for the main conjoint and the four theme blocks all finished after 128 hours 13 minutes and 46 seconds (a little over five days). We used 15,000 burn-in iterations, followed by 5,000 “used” draws per respondent, skipping every 10th draw, for a total of 65,000 iterations. Our overall conclusion is that we are satisfied with the software and are very impressed with the stability and capabilities of Sawtooth’s programs, especially the CBC/HB module.
Some of the results were expected—for instance, the large gap in the quality of life between the former East and West Germany—but the survey also uncovered some important new insights. Most striking was the large proportion of respondents willing to reform Germany’s social and political system. For instance, the results suggest that Germans may be more ready than their politicians for economic reforms that could solve some of the country’s economic problems, such as high unemployment and unsustainable pensions. Germans also seem to be more willing to contemplate lower levels of government support than politicians generally acknowledge. These and many other results received considerable media attention, not the least because of the large number online participants.
|
|||
|
About Us | Contact Us | Privacy | FAQ's | Site Map | Survey Software | Conjoint Analysis | Forum |
|
© 2013 Sawtooth Software, Inc. All rights reserved. |