Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Choose the right HB setting

Hello,

I have run 3 HB with different setting in Lighthouse Studio, but I am not sure which setting ist the best to choose.

Included None-Option with 20.000 Iterations in total:
- Pct. Cert. = 0.571
- RLH = 0.576
-Avg Variance = 1.189
-RMS= 1,923

Excluded None-Option with 20.000 Iterations in total:
- Pct. Cert. = 0.610
- RLH = 0.651
-Avg Variance = 0.776
-RMS= 1.382

Excluded None-Option with 60.000 Iterations in total:
- Pct. Cert. = 0.620
- RLH = 0.659
-Avg Variance = 0.049
-RMS= 1.425

As far as I understand should be every of these values be maximized.
But either the Pct. Cert. and the RLH is higher or the Avg. Variance and the RMS.

Can anybody help me?

Thank you and best regards,
Beatrice
asked May 20, 2019 by bea007 (255 points)

1 Answer

+2 votes
Trying to achieve the best within-sample RLH (fit statistic) is not necessarily the goal that you should try to maximize when running HB.  What you are trying to do is use all the information you can in the data in an appropriate way to estimate utilities that will be successful in predicting new choices by new people made outside the CBC questions used in utility estimation.

In the manual, it speaks of not estimating a None alternative (if you included a None option in the questionnaire):

"We generally recommend always estimating the none parameter (but perhaps ignoring it during
later simulation work). However, you can omit the "none" parameter by unchecking this box. In
that case, any tasks where None has been answered are skipped. The None parameter
(column) and None alternative are omitted from the design matrix."

So, if you decide to not estimate a None parameter when you included a None option in your CBC questionnaire, you are throwing away probably good information.  You are throwing away any task in which the respondent selected None.  (Presumably, the levels of attributes shown in competition to the None tended not to be good in this task; and that's probably good information to retain.)

The None utility may or may not have much variance or magnitude across respondents, which could affect the RMS and Avg Variance when include the None utility or not in the estimation.  So, that's not necessarily an apples-to-apples comparison when including the None or not in the estimation.
answered May 20, 2019 by Bryan Orme Platinum Sawtooth Software, Inc. (167,790 points)
...