Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

High standard errors in CBC logit


I´m currently running my CBC study and wanted to close the survey soon. Before I started running my survey, I used the Test Design Efficiency function and was able to generate standard errors below 0.05 for all main effects with my expected sample size of 100 (sample size per CBC is only modest as I have two treatment groups for which I run two seperate CBC´s).

Now I even was able to collect more data (n=150/treatment group) but standard errors with logit exceed 0.05 for some attribute levels (highest standard error is 0.06477). This only concerns attributes with more (three) levels. The remaining attributes  only have two levels. Standard errors for these attribute levels are below 0.05.

As far as I can see, the higher standard errors are caused through the high proportion of tasks in which respondents selected "None". It is about 32% (treatment group 1) and 30% (treatment group 2). When testing the design, I assumed the typical None-rate of 15% would apply.

Is there any way to further reduce standard errors? What came to my mind was to
(a) try to collect more data. However, as I have two treatment groups, I might need many respondents until I can achieve standard errors below 0.05.
(b) exclude respondents, which very often chose the None option. Is this even a reasonable approach, as this also means that I delete some data, that could be used for estimation?
Are there any other possibilities?

I want to do the final analysis using HB. What would standard errors above 0.05 in logit imply for my analysis in HB? Am I even "allowed" to use and analyse the data?

Thanks so much for your help!
asked Mar 23, 2018 by AnjaWe (270 points)
reshown Jun 19, 2018 by AnjaWe

1 Answer

0 votes
0.05 target is merely a guideline.  There aren't cliffs in statistics, where if you go slightly above 0.05 suddenly everything is terrible.

Your bigger concerns with HB are how many tasks are asked per respondent and how many times each level appeared per respondent, which the aggregate logit (rule of thumb) test says nothing about.  With HB, the general guideline is that each level be shown on average about 6x or more per respondent across all the tasks & concepts in the CBC questionnaire.  But again, there are no cliffs.

With sample size, other concerns have to deal with how many respondents you've collected to use in projecting to the population, how well you've sampled them, how much non-response bias you have, etc.  If you are analyzing by segment, you'll be concerned with whether you have enough respondents per segment to draw conclusions.  These types of issues can be thought of in terms of general sample size requirements for standard market research surveys.
answered Mar 23, 2018 by Bryan Orme Platinum Sawtooth Software, Inc. (169,815 points)