Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Looking for interactions between levels?

Dear Forum,

I am conducting a CBC.
- I have 10 random tasks with 3 concepts per task.
- I have 5 attributes with each having between 3 and 5 levels
- I am using the Dual Response option and Balanced Overlap

The main objective of my research is reporting on interactions. Finding out if a combination would lead to a higher preference.

When doing an interaction search, I can find the interactions between attributes. However, I would also like to see the interactions between levels.
 
How can I report for (significant) interactions between levels?

Thank you very much in advance.

Kind regards,

Joost
asked Oct 14 by Joost (140 points)

1 Answer

0 votes
Hi, Joost,

Think of the stat tests for significance of the interactions between attributes as something of an omnibus test, one which would have to be significant before looking at the details of interactions among levels.  Only if an interaction between two attributes is significant does it make sense to look at the specific interactions involving pairs of levels.

Moreover, if you're running a large number of interactions, you run the risk of counting experiment-wise error as a significant interaction.  In other words, while the chance of a false positive in a single stat test run at 95% confidence is 5% (by definition) if you run 10 interactions, the chance of a false positive rises to 40%, because there are 10 opportunities each with a 5% chance of a false positive.  You may want to correct for this fact using an adjustment like the Benjamini-Hochberg procedure for combating false discovery.

One more thing - a lot of times what look like interactions when running aggregate logit models turn out to be non-significant when you run the analysis using HB:  sometimes heterogeneity at the individual level masquerades as interactions at a more aggregate level.  So once you identify candidate interactions with the aggregate logit tests (suitably corrected for having run multiple tests) you probably want to confirm that they're still present when you estimate your model with HB, using appropriate tests with respondent level utilities.
answered Oct 14 by Keith Chrzan Platinum Sawtooth Software, Inc. (80,100 points)
Hi Keith,

Thank you very much for your response. I would like to re-iterate the question and answer to see if I have understood it correctly. I also have additional questions.

-    I will use the Interaction Search analysis type to look for significant interactions between attributes.
o    The Counts analysis type might also give me some “quick and dirty” suggestions for significant interaction effects.

-    If I find such a significant interaction between attributes, I can include this interaction in both the Logit and HB analysis type.
o    For the Logit: It is then possible to compare the overall fit (log likelihood) of the model before and after the inclusion of the interaction terms to see if it might significantly improve the ability to predict respondent choices.
o    Question: When including an interaction in HB, I receive a utility for every second-order interaction (between levels). How do I interpret these utilities?

-    Main Question: What is not clear to me, is how to test for significance of interaction terms in Logit and HB?
o    In other words: how do I know which interaction between what levels is significant? Or do I need to interpret the results differently: Since there is a significant interaction between attributes, all interactions between those levels are also significant. The utility score shows whether it is positive or negative.

Thanks in advance for reading and answering my question.

Kind regards,

Joost
Joost,  

I would start with the Interaction Search, which uses an aggregate logit to search for significant interactions.  I would submit those to a procedure that takes into account experimentwise error.  This is a better test than the counts analysis test, which is no more than exploratory.  

As the Interaction Search has already used the logit model to test for interactions, there is no need to do that again.  When you include the interaction between two attributes you do get utilities for each combinations of levels - but that is what the interaction between attributes IS - it's not like the interaction utilities for the pairs of levels are something over and above the interaction between the two attributes.  The best way to interpret them is to realize that they're additive with the main effects and not to look at them separately.  Better still is to see how they change the results of your simulations.  

For your last question, again, the interaction just IS the set of utilities you get between levels of the two attributes.  I suppose if you want to see which ones are significant, you can just divide the utility (or the mean utility in the case of HB) but the standard error to get a t-value, from which you could estimate a p-value.  But this is a somewhat artificial effort, I think:  note that these follow-on t-tests are entirely dependent on analyst choices about the coding of the attributes, and you would get one answer if you use effects coding, and more if you use dummy coding with one or another level set to the reference level.
...