Testing for Differences between Scores

Top  Previous  Next

The confidence interval is sometimes used as a way to test whether an item is "significantly different" from another item. The easy (but not technically correct) "eyeball" method is to observe whether the 95% confidence intervals overlap for the two items. If two items reflect a statistically significant difference using this "eyeball" test, then they will also pass the more rigorous tests described below.


A second, more technically correct way, to test whether two item mean scores are "significantly different" is to divide the difference in the scores by the pooled standard error of the two items, where the pooled standard error is equal to: sqrt (SEa^2 + SEb^2) where SEa is the standard error of the first item and SEb is the standard error of the second item. If this resulting T-value is greater than 1.96 in absolute magnitude (assuming large sample), we are 95% confident that the mean for one item is different from the other. However, this test assumes independent samples, when really we're dealing with matched samples, and could use an even more sensitive test.


An even more sensitive and technically correct way is to use the matched samples T-test. The scores may be exported and opened in Excel. A new column is defined by taking the difference between the two scores (for each respondent). Next, the standard deviation of the values in that new column is taken, and that standard deviation is divided by the square root of the sample size, resulting in the standard error of the difference between scores. The matched samples T-value is the mean difference in scores divided by this standard error.

Page link: https://www.sawtoothsoftware.com/help/analysis/manual/index.html?testing-for-differences-betwee.html