Sawtooth Software: The Survey Software of Choice

How Many Holdout Tasks for Model Validation (2015)

Analysts sometimes add holdout questions to their conjoint surveys to test the way they have specified their models. Holdouts can also be used for out-of-sample validation. The author, Keith Chrzan, uses synthetic CBC respondents to test how many holdouts are needed to identify a true vs. a misspecified model reliably. The misspecified models were manipulated to involve known small, medium-sized, and large errors. Under medium-sized and large errors, just a handful (about 5) holdout tasks seems to provide enough data to reliably indicate that the true model is better than the misspecified one. Under relatively small misspecification error, no number of holdouts tested (up to 15) can reliably point to the correct model. Chrzan’s findings demonstrate that just 1 or 2 holdouts is probably not enough to give practitioners enough evidence to compare competing models, but 5 or more will often bring enough evidence to reliably point to which of two competing models provides the best fit.

  Download PDF   View in new window

Sawtooth Software

6:30 AM to 5:30 PM Mountain Time
(GMT-6; GMT-7 Autumn/Winter)
Monday through Friday

Phone: +1 801 477 4700
Fax: +1 801 337 7410
Email: support@sawtoothsoftware.com

SKIM Software

9:00 AM to 5:30 PM Central European
(GMT+2; GMT+1 Autumn/Winter)
Monday through Friday

Phone: +31 10 282 3500
Fax: +31 10 282 3560
Email: software@skimgroup.com
Online: www.skimgroup.com

We're open!

Lighthouse Studio

Lighthouse Studio is our flagship software for producing and analyzing online and offline surveys. It contains modules for general interviewing, choice-based conjoint, adaptive choice-based conjoint, adaptive choice analysis, choice-value analysis, and maxdiff exercises.

Try Lighthouse Studio

  Buy Lighthouse Studio

Getting Started with Conjoint Analysis

New to conjoint analysis? Pick up a copy of Bryan Orme's best-selling book today!

Learn More