Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Significance of attributes and attribute levels

I have two questions regarding the significance and t-test of the chosen attributes and attribute levels:
-    To get a general overview about the main results of my CBC study, I’ve done a counts analysis. It shows me that one of my five attributes is not significant. I am not quite sure how I have to interpret this for a CBC analysis. Do you have some advice for that issue?

-    Also, I’ve run a HB analysis with my data. For the t-test, I’ve taken the following steps:
1.    I took the average utilities and the average standard deviation for each attribute level (which is one output besides the individual utilities and importances)
2.    Then, I calculated the square root of the sample size minus one (=SQRT(N-1)).
3.    Next, I computed the standard error by taking the standard deviation of each attribute level divided by the square root (n-1)
4.    Finally, I took the average utility for each attribute level and divided it by the standard error to get the t-value for each attribute level

First, my question is whether these steps are correct to figure out whether the attribute levels are significantly different from zero. If this is the correct way, I’ve found that one attribute level is not significant. Similar to my first issue, how can I interpret this finding?
It would be great to hear from you soon.

Best regards!
asked Jul 17, 2017 by briniminii (490 points)

1 Answer

0 votes
For stat testing I would use the HB utilities rather than the counts - the counts analysis allows only an approximate test.  

Your steps in the your stat test on the HB utilities looks correct.

It's not too surprising that some levels will not be significantly different from zero.  Think about, it, we're zero-centering the data, so the average utility for levels of each attribute is zero.  If you have a 3-level variable, for example, it won't be at all surprising if one of the levels has a large positive utility, one a large negative and one a utility in-between, close to zero.  Also the coding is arbitrary - if you did dummy coding rather than effects coding, your zero would be in a different place.   So don't think this is like some regression model where you might consider dropping levels that aren't significant - that would be a mistake.
answered Jul 17, 2017 by Keith Chrzan Platinum Sawtooth Software, Inc. (90,475 points)
Thank you very much for your answer!

So there is no need to further interpret the attribute level that is not signficant?

How can I measure whether an Attribute (not attribute level) is signficant within HB then?

Thanks a lot for your help!