Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Questionnaire versions with 4 treatment groups

My study has 4 treatment groups. I expect that approximately 200 respondents will participate in each version. The difference between the versions is a informative sentence which is likely to effect the choice of the respondents in the choice experiment.

How many questionnaire versions should I choose for  each survey? Logically, I would say 1 version because then we observe the effect of the treatment.

On the other hand, I know that it is recommended to include at least 100 questionnaire versions to prevent order bias. Would it be theoretically acceptable to create 200 versions for each treatment? Would Lighthouse Studio create the same 200 versions for each survey? Or would they be different from each other?

Many thanks for your support!
asked Jan 29, 2018 by anonymous

1 Answer

0 votes
If you one (different) version for each of your treatment groups then you will not be able to distinguish your treatment effect from a version effect - so please DO NOT do that.  

We usually recommend a large number of versions so that any version effects are spread so thinly across your sample that they can't do any harm.
answered Jan 29, 2018 by Keith Chrzan Platinum Sawtooth Software, Inc. (70,025 points)
Thank you so much! Just to make it clear for me.. Can Sawtooth create only one (same) version for 4 different groups? As I understand from your message, still it makes more sense to use 200 different versions since the version effects are spread thinly. Am I correct?
I don't know how many attributes and levels you have, but if your experiment is small enough then you can make a single-version experiment, yes.  You can let the software make it for you or you can make it outside of our software and simply import it if that works better for you.
Many thanks, really helpful. I have 5 attributes (4*4*2*2*2 levels). I expect approximately 200 respondents for each treatment group. My last question: Would you categorize it as small?
It depends on how you make the experiment.  If you use a design based on an orthogonal main-effects plan you're looking at 16 questions per respondent.   If you had many versions I would go with an efficient design with just 8-10 questions per respondent but if you create an efficient design with one version you will need an absolute minimum of 10 questions and I don't know how efficient it will be - I suspect you'll need more than the absolute minimum number of questions and that you might even be back up at 16 questions/respondent.
Unfortunately, it is not possible to include 16 questions (I have only 15 minutes for each survey). Then, it makes more sense to have 200 versions given that it is acceptable theoretically .
I have finally decided to use 10 questions per respondent, 4 concepts per task with complete enumeration. Strength of design is 814,  I achieved standard errors smaller than .05 for my main effect utilities.

Thanks for your help!
You should still look into what kind of strength of design you could get with just one version - if it's not too far off of 814 then you still might want to go that route.