Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

The optimal number of fixed blocks? Sequential Dual Response?

Dear community,

I have - once again - two questions for you, which I would be very happy to answer.

But first a few words about my design:
I have four attributes with three and four levels respectively. In total there are 108 different cards. I draw 36 cards with the highest information content using the Federov exchange algorithm. Then I generate the choice sets using various methods and divide them into blocks. The design consists of three blocks with 12 choice sets each. Each respondent is assigned to one of three blocks. Thus the choice sets of respondent 1 are the same as those of respondent 4, the choice sets of respondent 2 are the same as those of respondent 5, and so on. The designs are created using different R-Packages, e.g. 'support.CEs' (Mix&Match2), 'choiceDes', an own function for sampling without replacement, ...

Now I would like to test these designs with Lighthouse Studio and the simulated respondents and especially compare them with the Sawtooth methods (Balanced Overlap, Shortcut, ...).

Now the questions:
----------------------------------
1) When I insert these designs into Lighthouse Studio and test them (Legacy (OLS) Efficiency Test), I get a warning on each: „We strongly encourage you to further investigate the efficiency of your design prior to fielding this study.“.
And this, although the efficiency is very close to 1. I suppose this is because I only have 3 blocks of 36 cards each?

Example:
Legacy (OLS) Efficiency Test
-------------------------------------------------------------
Att/Lev   Freq.   Actual    Ideal      Effic.
 1    1        36 (this level has been deleted) DVU
 1    2        36   0.2357   0.2357   1.0000    KVU
 1    3        36   0.2357   0.2357   1.0000    BEG

 2    1        27 (this level has been deleted) Mix_Default
 2    2        27   0.2887   0.2887   1.0000    Mix_EE
 2    3        27   0.2887   0.2887   1.0000    Mix_Wind
 2    4        27   0.2887   0.2887   1.0000    Mix_PV

 3    1        36 (this level has been deleted) Reg_0
 3    2        36   0.2357   0.2357   1.0000    Reg_50
 3    3        36   0.2357   0.2357   1.0000    Reg_100

 4    1        36 (this level has been deleted) Preis_0
 4    2        36   0.2357   0.2357   1.0000    Preis_5
 4    3        36   0.2357   0.2357   1.0000    Preis_10

The frequencies for Balanced Overlap are 3600 and 2700 respectively.

With the "Logit Efficiency Test Using Simulated Data", the standard errors of all my methods are ok and comparable to the Balanced Overlap method. Now my question is, how do I deal with this warning? For the estimation of Main Effects and 2-Way Interaction Effects 36 cards (out of 108 possible) should be sufficient? It is not quite clear to me why 300 blocks, i.e. one individual block per respondent, would be superior to the 3 fixed blocks.

The help function of Lighthouse Studio says:
“We generally recommend that you include enough versions of the questionnaire so that the number of random choice tasks times the number of questionnaire versions is greater than or equal to 80 (assuming no prohibitions, and typical attribute level specifications).”

In my case 12*3 = 36 < 80, so the number of blocks is too small?
Maybe I should ask a more general question: Why are more blocks better and how many are actually necessary?


2) Is it possible to use the so-called "Sequential Dual Response (SDR)" with Sawtooth? With this method, so-called "forced choice sets" are performed first. In my case the 12 choice sets with 3 cards each, but without none-option. The second step uses these results by displaying only the selected card of a choice set + none-option (so called "free choice sets"). In contrast to traditional dual response, these two phases are separate, i.e. first all forced choice sets and then all free choice sets. For reasons of time, I imagine this in my case as follows:
i) A respondent answers the 12 forced choice sets.
ii) Only 6 of the 12 choice sets are displayed again as free choice sets.

Is it possible to implement something like this with some programming effort? If so, how?


Thanks in advance!
asked Jun 27 by Nico (290 points)

2 Answers

+1 vote
I have some comments about your first question, but I don't have a solution for your second question (I expect other people probably have solutions for that).

We have implemented some basic warning checks in our Design Tester within our Lighthouse Studio CBC software, just to make sure people are thinking through their design.  A warning doesn't necessarily mean you have an error.  It is just a warning to have you double-check your thinking.

First, I would not rely on the "legacy OLS" test alone.  It only speaks to main effects and it uses a test that doesn't leverage assumptions and math as used in multinomial logit (MNL).  I would focus more on the simulated data with aggregate logit test, with a sample size matching the size you plan to obtain with the real data.

You say you have run the simulated respondents aggregate logit test, but you did not comment on whether you asked that test procedure to estimate all main effects AND interaction effects (there is an advanced option in the simulated logit test to add all the first-order interaction effects among your 4 attributes).  In your message, you indicate that you want to estimate main effects and potentially all first-order interaction effects.  So, did you examine the main effects and see that the estimated standard errors for them (given the target sample sizes) was 0.05 or less; then also look at the estimated effects for all first-order interaction effects (you have six of those, considering all pairings of four attributes in your design) to determine that just three blocks with 12 choice sets each can obtain precise estimates of both main effects plus all potential first order interactions?  Our recommendation is that the standard errors for interaction effects should be 0.10 or less from simulated data in aggregate logit.  

As you have learned, Sawtooth Software's CBC system is flexible to either design its own choice tasks, or to import a design that you have generated elsewhere.  You are taking the latter approach.  Sawtooth Software's standard approach is to create a default 300 blocks of the design.  That strategy is based on the idea that more variation in choice task composition across blocks can lead to 1) better control of order/psychological context effects and 2) potentially better precision of both main & interaction effects than designs with few blocks.  However, we recognize that carefully designed plans with fewer blocks may be equally good as well, so long as you do the proper checks for precision of the design for both main effects and potential first-order interaction effects; and so long as you think through how many blocks and randomizations within blocks are useful to reduce order effects to a reasonable level.

It isn't necessary to use so many as 300 blocks.  300 blocks typically is overkill; but since computers make it so easy to generate many blocks and distribute them across respondents, we consider 100s of blocks a good thing.  In terms of controlling for psychological order effects and obtaining precise estimates of both main and first-order effects, fewer blocks such as 10 or 12 also may be perfectly fine.  Maybe for your specific limited design (just 4 attributes with few levels per attribute), three blocks would be fine...you'll need to examine the main effects and first-order interactions aggregate logit report with your target sample size to verify this for your specific situation.

Regarding design efficiency, you will find that Complete Enumeration and Shortcut are quite good for simulated respondent data and precision of main effects.  But, Balanced Overlap will typically be superior for first-order interactions.  Yet, be careful that simulated data and aggregate logit precision does not exactly reflect what you'll get with real respondents.  Real respondents don't always answer CBC questionnaires according to the assumptions of additivity (compensatory decision-making) and for respondents who screen on must-have or must-avoid features, some level repeating (level overlap) within choice tasks is a desirable property (balanced overlap allows some level overlap) for improving the efficiency of the CBC questionnaire.
answered Jun 27 by Bryan Orme Platinum Sawtooth Software, Inc. (164,715 points)
+1 vote
Hello, Cybey.  

First your questions about the design warning.  Seeing the small number of versions,  the designer gave you a standard warning to check your efficiency statistics.  You did that and it looks like your design works fine, so you  an ignore that warning.  

As for more versions, often our clients want to be able to test for and implement interactions among attributes, but most small fixed designs don't allow that, so creating a design with more versions is likely to enable one to do so.  If you're happy with the effects your design allows you to capture, and if there's no chance you might later wish you'd have been able to capture an interaction you're not currently planning on, then your design is a good one and you can ignore that warning as well.  

Those warnings are general cautions for all users of the software and we decided to use them to err on the side of caution as it's better to have a warning come up a little too often than even one time not often enough.

As for your dual response situation, yes, you could handle those.  The easiest way would be to export your data as a "single CSV" file, which pulls your response and design data into a single export.  Then you could append the 6 new none-tasks by manipulating the resulting data file in Excel or R so that the choice data would include those extra 6 observations properly formatted.  This is not a trivial job and the exact steps you take to get from the raw data export to the revised design file will vary from analyst to analyst depending on what software you use for data manipulation.  Just make sure your resulting data file is formatted appropriately for import into our CBC/HB program for analysis (or for import into our latent class program if you plan on aggregate MNL or latent class MNL analysis).
answered Jun 27 by Keith Chrzan Platinum Sawtooth Software, Inc. (77,750 points)
Dear Bryan, dear Keith,

Thank you for the quick answers! You are great.

The small number of blocks was/is caused by the fact that I want (or wanted) to implement the design manually on a survey platform. Right now, I'm torn between using Sawtooth after all. Therefore, question 2 is also very important for me.
With the survey platform, 12 choice sets per block must be implemented manually (without holdouts), because each block results in a new branch of a tree to which a respondent can be assigned. Also with my methods, e.g. optimal blocking using the algorithm of Cook & Nachtsheim (1989), the OLS standard errors decrease with increasing number of blocks. It is interesting to see that the number of blocks is similar to the number of respondents: At a certain number, there is hardly any change. The marginal effect of an increase becomes smaller and smaller.
3 blocks, 12 choice sets per respondent:
Logit Efficiency Test Using Simulated Data
-------------------------------------------------------------
Main Effects: 1 2 3
Interactions: 1x2 1x3 2x3
Build includes 300 respondents.

Total number of choices in each response category:
Category   Number  Percent
-----------------------------------------------------
       1    1241   34.47%
       2    1168   32.44%
       3    1191   33.08%

There are 3600 expanded tasks in total, or an average of 12.0 tasks per respondent.

Iter    1  Log-likelihood = -3943.89181  Chi Sq = 22.22486  RLH = 0.33436
Iter    2  Log-likelihood = -3943.47800  Chi Sq = 23.05248  RLH = 0.33440
Iter    3  Log-likelihood = -3943.46139  Chi Sq = 23.08570  RLH = 0.33440
Iter    4  Log-likelihood = -3943.46072  Chi Sq = 23.08703  RLH = 0.33440
Iter    5  Log-likelihood = -3943.46070  Chi Sq = 23.08708  RLH = 0.33440
Iter    6  Log-likelihood = -3943.46070  Chi Sq = 23.08709  RLH = 0.33440
*Converged

          Std Err    Attribute Level
  1       0.03528    1 1 DVU
  2       0.03152    1 2 KVU
  3       0.03183    1 3 BEG

  4       0.03755    2 1 Mix_Default
  5       0.03763    2 2 Mix_EE
  6       0.03933    2 3 Mix_Wind
  7       0.03915    2 4 Mix_PV

  8       0.03292    3 1 Reg_0
  9       0.03357    3 2 Reg_50
 10       0.02966    3 3 Reg_100

 11       0.06105    DVU x Mix_Default
 12       0.05419    DVU x Mix_EE
 13       0.05129    DVU x Mix_Wind
 14       0.06130    DVU x Mix_PV
 15       0.05344    KVU x Mix_Default
 16       0.05484    KVU x Mix_EE
 17       0.05096    KVU x Mix_Wind
 18       0.05684    KVU x Mix_PV
 19       0.05542    BEG x Mix_Default
 20       0.04654    BEG x Mix_EE
 21       0.05784    BEG x Mix_Wind
 22       0.05817    BEG x Mix_PV

 23       0.04576    DVU x Reg_0
 24       0.04130    DVU x Reg_50
 25       0.04220    DVU x Reg_100
 26       0.04793    KVU x Reg_0
 27       0.04551    KVU x Reg_50
 28       0.04240    KVU x Reg_100
 29       0.04716    BEG x Reg_0
 30       0.04380    BEG x Reg_50
 31       0.04165    BEG x Reg_100

 32       0.05176    Mix_Default x Reg_0
 33       0.05426    Mix_Default x Reg_50
 34       0.05130    Mix_Default x Reg_100
 35       0.05262    Mix_EE x Reg_0
 36       0.04852    Mix_EE x Reg_50
 37       0.05326    Mix_EE x Reg_100
 38       0.05497    Mix_Wind x Reg_0
 39       0.05633    Mix_Wind x Reg_50
 40       0.05114    Mix_Wind x Reg_100
 41       0.04978    Mix_PV x Reg_0
 42       0.05221    Mix_PV x Reg_50
 43       0.05088    Mix_PV x Reg_100


As you can see, all main effects as well as first-order interaction effects can be estimated. The Balanced Overlap is still better, but the difference is small.

@ Keith
Unfortunately, I was not able to follow your descriptions. :-/
Here again, what I would like to implement:
In the first phase, a respondent gets 12 choice sets with three electricity tariffs each to choose from. For each choice set, he has to choose a tariff. There is no none-option. That’s where the name "forced choice sets" comes from.

An example:
In the seventh choice set, a respondent chooses tariff 2. In the eighth choice set, a respondent chooses tariff 1. After the 12 forced choice sets, phase two starts: The first so-called "free choice set" shows only the selected answer of forced choice set seven, i.e. tariff 2 vs. none-option. The second so-called "free choice set" shows only the selected answer of forced choice set eight, i.e. tariff 1 vs. none-option.
It is not necessary to use the responses from the choice sets 7-12. A random drawing of the forced choice sets would also be possible. The important thing is that there are no more than 6 draws. With the traditional dual response, all answers (=12) would be queried again. In addition, the two phases forced/free are not separated.
I hope you understand what I mean. The question is whether this method can be implemented with Lighthouse Studio?

The concept comes from:
Christian Schlereth, Bernd Skiera (2017) Two New Features in Discrete Choice Experiments to Improve Willingness-to-Pay Estimation That Result in SDR and SADR: Separated (Adaptive) Dual Response. Management Science 63(3):829-842. https:// doi.org/10.1287/mnsc.2015.2367

The authors write on page 841:
“The separation feature ensures that context effects with respect to the no-purchase option are not a concern, because the forced choice questions do not contain a no-purchase option, and all free choice questions contain only one product alternative, disconnected from the forced choice questions. In addition, because of the strict separation of forced and free choice questions, both SDR and SADR avoid choice deferral, a major problem for dual response experiments.”
Lighthouse Studio does not have a standard feature that would allow you to implement your separated dual response.  That said, you can implement the SDR by (a) exporting the CBC survey data as a single .csv file, (b) manually adding the SDR tasks and then (c) analyzing in the standalone CBC/HB software.   So you would NOT be implementing this entirely within Lighthouse Studio, no.  

Step (b) is a bit laborious and you are correct that I didn't provide step-by-step instructions (for fear of leaving a step out and confusing you).  But it's easy enough to do and I've done it on occasion so I know it can be done.
Dear Keith,

unfortunately, I do not really understand what you mean by "CBC survey data”. Do you mean the design matrix with the column headings "Version (0 = Fixed Task)", “Task”, “Concept”, …? I would be glad, if you could explain the steps a little bit more in detail, because as a newbie, I'm currently stuck.

In the meantime I wasn't idle and did a little research and found this forum entry: https://sawtoothsoftware.com/forum/7616/showing-response-question-next-question-when-there-page-break
Respondent Brian McEwan writes that I probably need JavaScript to view previous answers from Choice Sets. I tried something and managed to display the previous answer as intended with Sequential Dual Response (SDR) using the Free Format Question:

[code]
[% Begin Unverified Perl

GETVALUE("CBC_Random1")

End Unverified %]
[\code]

In my case, a 1, 2 or 3 is simply displayed, depending on the choice of the respondent in the regarding choice set. To my knowledge, I have to use this answer/value to ask the Sequential Dual Response (SDR) question. For example, like this:

[code]
[% Begin Unverified Perl

SETVALUE("SDR1_PreviousChoice", GETVALUE("CBC_Random1"));

End Unverified %]

<!-- [% SDR1_PreviousChoice %] -->

<table class="classic_cbc" align="center">
    <!—Electricity Mix, Attribute 2 -->
    <tr>
        <td class="att_label">[%CBCAttributeLabel(CBC_Random1, 2)%]</td>
        <td class="top_row">[%CBCDesignLevelText (CBC_Random1, SDR1_PreviousChoice, 2)%]</td>
    </tr>
    <!-- Supplier, Attribute 1 -->
    <tr>
        <td class="att_label">[%CBCAttributeLabel(CBC_Random1, 1)%]</td>
        <td>[%CBCDesignLevelText (CBC_Random1, SDR1_PreviousChoice, 1)%]</td>
    </tr>
    <!—Regional Electricity Share, Attribute 3 -->
    <tr>
        <td class="att_label">[%CBCAttributeLabel(cbc_Random1, 3)%]</td>
        <td>[%CBCDesignLevelText (CBC_Random1, SDR1_PreviousChoice, 3)%]</td>
    </tr>
   
    <!-- Choice Buttons -->
    <tr>
        <td>&nbsp;</td>
        <td class="input_cell clickable">
            Wechseln<br> [%RadioSelect(SDR1_response, 1)%]
        </td>
    </tr>
    <tr>
        <td>&nbsp;</td>
        <td class="input_cell clickable">
            Nicht wechseln<br> [%RadioSelect(SDR1_response, 0)%]
        </td>
    </tr>
</table>…
[\code]

The Free Format question is called "SDR1", the variable "SDR1_PreviousChoice". It is a hidden variable of type integer.

Is this the way to go? At least it seems to work in test mode.
If that perl code works for you then yes, it seems to be a good way to go.
...