Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

ACBC Hit Rates significantly higher than chance?

Dear forum,

can you recommend a test procedure to show that the predictions of hold out card choices (hit rates) are significantly better based on the ACBC´s PWU results compared to chance?

I use three profiles per set, so chance would be 1/3. I read about McNemar´s test, but I find it hard to model the data appropriately. Is there a way to use one overarching model or is it necessary to assess the significance for each hold out card separately?

Thanks very much for your support.


asked Nov 6, 2018 by Chris Berlin (480 points)

1 Answer

+1 vote
Hi, Christian,

McNemar is specific to testing 2x2 tables, so if you have 3 profiles per set it's going to be inappropriate.  

What you can do is build a simulator with the ACBC part worth utilities.  Then for each holdout question, your first choice prediction will either be right or wrong for each respondent (you'll have the actual and the predicted choice for each respondent and they'll either match or they won't).  So if you have 250 respondents and 4 holdout choice sets, you have 1,000 observations.  By chance alone 33.33% of them should be correct, and from your simulation you'll have some percentage of correct predictions that you can compare to 33.33% using a Z-test for proportions.  

Another way to do this would be to count the number of hits per respondent, which would range from 0 to 4, right?  Then you take a mean of that number of hits across respondents and compare it to 1.333 (an average of 4/3 correct predictions by chance alone) using a one group t-test for means with a sample size of 250.    

So you have a couple of options for comparing your hit rate to chance.
answered Nov 6, 2018 by Keith Chrzan Platinum Sawtooth Software, Inc. (70,125 points)