Hi, Joost,

Think of the stat tests for significance of the interactions between attributes as something of an omnibus test, one which would have to be significant before looking at the details of interactions among levels. Only if an interaction between two attributes is significant does it make sense to look at the specific interactions involving pairs of levels.

Moreover, if you're running a large number of interactions, you run the risk of counting experiment-wise error as a significant interaction. In other words, while the chance of a false positive in a single stat test run at 95% confidence is 5% (by definition) if you run 10 interactions, the chance of a false positive rises to 40%, because there are 10 opportunities each with a 5% chance of a false positive. You may want to correct for this fact using an adjustment like the Benjamini-Hochberg procedure for combating false discovery.

One more thing - a lot of times what look like interactions when running aggregate logit models turn out to be non-significant when you run the analysis using HB: sometimes heterogeneity at the individual level masquerades as interactions at a more aggregate level. So once you identify candidate interactions with the aggregate logit tests (suitably corrected for having run multiple tests) you probably want to confirm that they're still present when you estimate your model with HB, using appropriate tests with respondent level utilities.

Thank you very much for your response. I would like to re-iterate the question and answer to see if I have understood it correctly. I also have additional questions.

- I will use the Interaction Search analysis type to look for significant interactions between attributes.

o The Counts analysis type might also give me some “quick and dirty” suggestions for significant interaction effects.

- If I find such a significant interaction between attributes, I can include this interaction in both the Logit and HB analysis type.

o For the Logit: It is then possible to compare the overall fit (log likelihood) of the model before and after the inclusion of the interaction terms to see if it might significantly improve the ability to predict respondent choices.

o Question: When including an interaction in HB, I receive a utility for every second-order interaction (between levels). How do I interpret these utilities?

- Main Question: What is not clear to me, is how to test for significance of interaction terms in Logit and HB?

o In other words: how do I know which interaction between what levels is significant? Or do I need to interpret the results differently: Since there is a significant interaction between attributes, all interactions between those levels are also significant. The utility score shows whether it is positive or negative.

Thanks in advance for reading and answering my question.

Kind regards,

Joost