Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Comparing two groups using (3-way-)interactions with CBC / HB

Hello there,

I am dealing with the following challenge and would be grateful for advice. I collected data for two CBCs from two independent groups of respondents and the CBCs’ designs were identical in terms of attributes, number of versions etc. The only difference between the two groups was that group 1 was told that the attributes shown related to product A and group 2 was told that they related to product B (so, the attributes shown were not product related). This is done because I want to find out if the attributes impact choices differently for two different types of products. Each (identical) design is 2x4x2x3x4 and interactions between the attributes are important to be estimated as well. So, I want to compare groups by interacting attributes and interactions with a dummy product variable (product A and B).

The output from CBC / HB (latest version 5.5.6) that I need is the main effects for all attribute levels for both products and some interaction effects between attributes also for both products. So what I want is basically the output that I would get if I ran estimations for both groups independently from each other, only that I estimate both groups together to make it easier to check for significant differences between both groups’ parameter estimates. This is how I proceeded:

-    The design file contained two fixed tasks, 13 random tasks, and 15 versions. So I duplicated those 15 versions (copying and pasting them in the lines below them) and changed the version numbers for the duplicated choice tasks (for product B) from originally 1-15 to 16-30.
-    Then, I also changed up the attribute levels in all concepts of the versions 16-30 (product B) as following: Attribute 1: From 1, 2 to 3, 4; Attribute 2: From 1, 2, 3, 4 to 5, 6, 7, 8. etc… So, the attributes basically get turned into merged meta-attributes (2-way-interaction between the attributes from the CBC and a dummy-coded product attribute with the levels 1 and 2).
-    In the response file, I merged the two data sets from the two CBCs and adapted the column stating the version number for group 2 accordingly (from 1-15 to 16-30). So, taking the example of attribute 1, it is assumed that respondents from group 1 saw the first 15 versions (containing level 1 and 2 of attribute 1) and respondents from group 2 saw versions 16-30 (containing level 3 and 4 of attribute 1 which are the interaction terms of both attribute levels times product B).
-    The following difficulty occurs when I want to estimate the (3-way-)interaction effects between the attributes. For example, let’s look at the new attributes 1 and 3 which both consist of 4 levels each (2 original attribute levels x 2 product levels (A and B)):
-    Attribute 1: (lev 1): 1 x A, (lev 2): 2 x A, (lev 3) 1 x B, (lev 4) 2 x B
-    Attribute 3: (lev 1): 1 x A, (lev 2): 2 x A, (lev 3) 1 x B, (lev 4) 2 x B
-    An interaction between these two attributes results in 16 combinations. Half of them are nonsense and there is not even data available for them (e.g., (a1 lev 1) “1 x A” and (a3 lev 3) “1 x B”). Since the software doesn’t allow 3-way-interactions (which would result in only 8 combinations between attribute 1, attribute 3, and the product attribute (2x2x2)), I read that merged meta-attributes was the way to go but this doesn’t seem to work here.
-    So my questions are the following:

1.    Do you have any idea of how to solve this problem?
2.    How could I perform a fixed holdout validation, now? Because adding the product dimension means that those two fixed tasks would turn into four fixed tasks where group 1 saw the first two and group 2 saw the second two tasks. So, I can’t run a simulation across the entire (merged) sample anymore.
3.    Is there a different way of comparing both of the groups (ideally with a merged data set of both groups)?

Thank you very much in advance!
asked Aug 15, 2018 by Leo (310 points)

Your solution to the original question

Please only use this to answer the original question. Otherwise please use comments.
Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
Anti-spam verification:

To avoid this verification in future, please log in or register.