Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

CBC/HB used on heterogeneous small sample

Dear Sawtooth Team,

I have run a CBC/HB estimation using a relatively small sample of 20 people. I am now facing criticism that the small sample in conjunction with the assumed heterogeneity is causing biased individual utilities. However, I argue that CBC/HB is used for that case specifically, where heterogeneity is assumed and should be identified.

May I ask how you see the matter?

Many thanks for your help,

Joerg
asked Mar 14 by Joerg

1 Answer

0 votes
HB (under the right conditions) can be used with very small sample sizes with CBC, ACBC, and MaxDiff questionnaires.  The right conditions are relatively few parameters to estimate (attributes & levels) and a relatively large number of choice tasks per respondent.  For example, if those 20 people represented nearly a census of the population (such that sampling error was held at a minimum), I wouldn't hesitate to use HB if the CBC study had 5 attributes each with 2 to 4 levels, and I had 15 to 20 choice tasks per respondent.

In the early years of the introduction of HB, Greg Allenby was at our conference and the audience pushed him regarding how low one could go with sample size and HB and obtain equally good or better estimates as with traditional methods given the same sample size (which at the time meant aggregate logit).  He said as few as 10 respondents, or maybe even lower, if I remember correctly.  And, he felt HB would be better than aggregate logit in those situations (given the same sample size restrictions).

On my own, I have tested ACBC with subsamples as small as n=9 and gotten good results.  I wouldn't expect "biased" results given small sample sizes.  There is less certainty on the upper-level model in that case, so the lower-level model (individual) tends to carry a large weight.  So, bias?  I'm not seeing it.  Noise, yes, as there always is for such models.
answered Mar 14 by Bryan Orme Platinum Sawtooth Software, Inc. (159,810 points)
...