Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Design Efficiency When Combining Prohibition and Alternative-Specific Attributes

Hi everyone,

within my current CBC study I found that for presenting realistic choice sets and still following the goal of the study, it might be necessary to use a combination of prohibitions and alternative-specific attributes.

Going through the Lighthouse help I found that in general such combinations should be avoided as they can lead to unexpected results.

Now I'm looking for some advice if the current setup of the study will yield valid results.
Can this be analyzed by looking at the design report? (see below)

Prohibition: Att1/Lev1 not in combination with Att2/Lev2,3
Alternative-Specific: Att5 only in combination with Att2/Lev2

Thank you very much in advance for any hints/explanations.

Kind regards,

Build includes 120 respondents.

Total number of choices in each response category:
Category   Number  Percent
       1     412   34.33%
       2     404   33.67%
       3     384   32.00%

There are 1200 expanded tasks in total, or an average of 10.0 tasks per respondent.

Iter    1  Log-likelihood = -1313.14471  Chi Sq = 10.38007  RLH = 0.33478
Iter    2  Log-likelihood = -1312.87947  Chi Sq = 10.91055  RLH = 0.33485
Iter    3  Log-likelihood = -1312.86773  Chi Sq = 10.93404  RLH = 0.33486
Iter    4  Log-likelihood = -1312.86725  Chi Sq = 10.93500  RLH = 0.33486
Iter    5  Log-likelihood = -1312.86723  Chi Sq = 10.93504  RLH = 0.33486
Iter    6  Log-likelihood = -1312.86723  Chi Sq = 10.93504  RLH = 0.33486

          Std Err    Attribute Level
  1       0.10658    1 1 Immer Grün
  2       0.06425    1 2 Stadtwerke Musterstadt
  3       0.06318    1 3 Pink Strom

  4       0.09970    2 1 Standard Strom ohne Label
  5       0.06036    2 2 Öko Basis
  6       0.06242    2 3 Öko Nachhaltig

  7       0.05302    3 1 0 Prozent
  8       0.05302    3 2 5 Prozent
  9       0.05286    3 3 10 Prozent
 10       0.05375    3 4 15 Prozent

 11       0.04123    4 1 0 Prozent
 12       0.04043    4 2 50 Prozent
 13       0.04112    4 3 100 Prozent

 14       0.08617    5 1 Grüner Strom
 15       0.08674    5 2 TÜV Süd
 16       0.08562    5 3 OK Power
asked Nov 26, 2019 by Philipp

1 Answer

0 votes
The advice int he help menu is that you should be careful to test your design when you combine prohibitions with alternative specific effects, to make sure you don't have big problems.  

Looking at the design report, it looks like all of your effects are estimable (you have standard errors and not *******s, and your model converged without a warning about a deficient design or a ridge adjustment).  

You might want to look at the D-efficiency (the "strength of design" for your model compared to a model with no prohibitions and no alternative-specific effects just to see the reduction in your effective sample size, but given this design report I'd be comfortable fielding this study.
answered Nov 26, 2019 by Keith Chrzan Platinum Sawtooth Software, Inc. (85,325 points)
Thank you very much for the quick reply Keith!

In addition, I'd like to ask something about the D-efficiency that you mentioned. As I understood, those values can not be compared between different designs.

How about two designs that only differ in the number of random choice tasks? Is a direct comparison of the D-efficiencies between those two designs allowed?

Thank you very much!
Actually, D-efficiency CAN be compared across designs for the same number of attributes and levels.  In fact, the easiest way to interpret it is as a relative measure.  For example if one design has a D-efficiency of 400 and another has a D-efficiency of 300, we can say that the former is 33% more efficient than the latter.  

You can compare designs with different numbers of choice tasks.  

What you cannot do is compare designs with different sets of parameters (attributes, levels, interactions).
Hello Keith,

thanks for the additional information. I'd like to add another topic that needs clarification on my side.

For the combination of prohibitions and alternative-specific attributes, complete enumeration should be used as standard approach, giving the best D-Efficiency also in my case.

However, compared to balanced overlap, the choice is harder to process for the respondent, as all levels differ for the different attributes.

Now with respondents eventually applying a heuristical approach and eliminating attributes from their choice process, the results of the study might not reflect the real results.

My question is, if there's maybe some rule of thumb, explaining on when to pick complete enumeration despite this possible drawback over balanced overlap.

Thanks again for your support!

As you note, if respondents are using non-compensatory  rules, a design with overlap is likely to incline respondents to deeper attribute processing than is a design without overlap.  I don't know of a rule of thumb about how often to expect respondents to do non-compensatory consideration of attributes, but in academic reviews I think the proportion of such respondents in a range of studies ran as high as 70% and as low as 20% - in other words, it seems very common.  My preference is to use balanced overlap as a default and to move away from it only if the benefits of other methods are large.
Thanks Keith.

Within my current study, Complete Enumeration gives me a 16% higher D-Efficiency compared to Balanced Overlap.
Just looking at this bare number, would you consider Balanced Overlap to be the method of choice.

Thank you very much!
Efficiency means one good thing but overlap seeks to provide another, and there's really no easy way to get the best design for both objectives in most studies.

Of course an efficient design will be more efficient than an overlap design.  But an overlap design leads to deeper attribute processing among non-compensatory choosers than will an efficient design with no overlap on important attributes.  

If I think respondents may choose lexicographically, then an efficient design with no overlap will allow you only to learn what the most important attribute is and you will be better with a design with overlap.  If your respondents choose based on the total utility of an alternative, and not with any non-compensatory rules, then you're better off with the efficient design.  

Without knowing the utilities and decision rules in advance, it's a little hard to say which design will have been better for you once you know the answer, but this is a problem we all face.

That said, knowing how prevalent lexicographic choosing is, I usually prefer to play it safe and use Balanced Overlap.  Indeed, that's why it's the default design method in our software.
Thank you for the feedback. I'm now also tending towards the use of Balanced Overlap. (eventually compensating the efficiency disadvantage by adding another choice task).

However I did not consider the difference in std. errors so far. While for Complete Enumeration I'm staying below the recommended thresholds of .05/.10, for Balanced Overlap I'm getting std. errors of up to .13/.14 (for alternative-specific attributes).

Might I ask one more time for your take on this Keith? Thanks so much!
Philipp, well, it's not ideal, but it's also not too bad, since you're only a little over the 0.10 rule of thumb for alternative specific effects.  That rule of thumb is sensitive to sample size and really dates back to when we ran CBC with aggregate logit - since most folks run HB MNL analysis on their CBCs these days, these aren't the standard errors you're going to get anyway - but they're helpful as a rule of thumb.  

Plus, I don't know about you, but I don't live in the ideal world (apparently like you, I work for a living instead of reclining on a beach reading a book on my own private tropical island) - so I've sort of resolved myself to accepting some non-ideal things.  

Good luck!