I am at the moment conducting a CBC study. I have now received 340 respondents for a CBC including 10 random tasks (with 4 concepts + none-option) and 3 fixed holdout tasks, where 2 tasks are the same concepts after random task 3 and 9 to see whether respondents answered the same way when having the same offering. The third holdout is to test validity (after random task 6).
The following questions advanced prior to estimating the data with HB:
1. Does it make sense to include one of the holdout tasks that are used for reliability to the estimation? If so why?
2. I checked for reliability by giving a 1 if respondent selected the same concept in both holdout tasks and 0 if not. I only had a reliability of about 60 %. Is this a very low figure? How could I interprete it?
3. In Assessing the Validity of Conjoint Analysis – Continued (1997) (https://www.sawtoothsoftware.com/download/techpap/assess2.pdf
) the formula of Wittink and Johnson (1992)at page 5 of the paper was used to calculate the highest possible hit rate given a certain test-retest reliability. Is this also applicable for CBC and validation based on HB estimation (I would calculate validity as explained in https://sawtoothsoftware.com/forum/12998/computing-holdout-validity-for-the-overall-sample?show=12998#q12998
4. I want to use the holdout tasks that I use to check test-retest reliability to also use to calculate validity with hit rate. Is it advisable ? Can I include both tasks that I used for reliability assessment or only one (as they have the same concepts onl in different order)
Thank you in advance for helping me out.