Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Quota, Randomization, Retest Reliability and long questionnaire

Hi,

I have a quite long questionnaire that is composed mainly of 5 parts:

P1: Quota screening questions
P2: CBC questions
P3: MaxDiff Questions
P4: Third Methodology "Part of the questions only shown to respondents and that was managed by constructed list"
P5: General information and Demographics

P2-4 are shown in a random order based on Randomized blocks marking the first and last questions in each part.

Then after answering the questionnaire, the respondent is asked if they want to join in a retest experiment after 3-4 weeks of the original questionnaire:
Yes: They will finish at a text question and their account will be marked as "Incomplete" And after 3 weeks they will be invited to log in again with the same password and continue on the next page a duplicate of the original questionnaire (methodology recommended by Mr. Justin Luster"
No: they will be directed to a terminate page thanking them for the participation & they will be marked as "Completes".

I got several feedbacks that the questionnaire is too long and the drop-outs will be enormous. Thus, I want to work around its length by dividing the respondents into 4 groups, all following the original quota requirements:

G1: Answers all the questionnaire (around 100 people)
G2: Skip P1 (around 175 people)
G3: Skip P2 (around 175 people)
G4: Skip P3 (around 175 people)

I am really confused about how to go on with such a design.

Some Ideas:

Make 3 sub questionnaires, for G2-G4, and Direct respondents to these links first then once the  3 sub questionnaires are filled, I ask the following respondents to complete the whole questionnaire?
expected problems are the Incomplete status of the respondents who sign up for the retest reliability exercise which might never be filled on time.

Manage that in the original questionnaire but I am not clear about where to go from here.

Any help is really appreciated and all insights are helpful

Regards'
asked Jun 6, 2019 by AMYN Bronze (2,115 points)

1 Answer

+2 votes
 
Best answer
This sounds doable with just the one survey.

You will need to create two lists for this, assuming you haven't already created these.  The first list should be a predefined list with one item representing each part of the survey to be randomized / skipped.  The second should be a constructed list that uses that first list as its parent.  Here's an example of what the instructions for the constructed list should look like:

Begin Unverified Perl
my $plist = 'list1';
my $group = GETVALUE('group');

if ($group == 1) {
    ADD($plist);
}
elsif ($group == 2) {
    ADD($plist, 2, 3);
}
elsif ($group == 3) {
    ADD($plist, 1);
    ADD($plist, 3);
}
else {
    ADD($plist, 1, 2);
}
End Unverified
Randomize()


Line 2 must be updated with the name of the parent list.  Line 3 must be updated with whatever variable in your survey controls which group respondents go into.  Then the "if" / "elsif" / "else" controls which parts of the survey the respondents in that group will see.

Finally, you need to set your randomized block set to use this constructed list.
answered Jun 6, 2019 by Zachary Platinum Sawtooth Software, Inc. (134,850 points)
selected Jun 14, 2019 by AMYN
Thank you very much for your suggestion.
I have a question regarding the application of this constructed list:
For The variable in line 3, I have no specific variable for that, I want people to be assigned randomly to each of the 3 groups by a rate of 1:5:5:5 (1= whole survey; 5: any of the combinations of only 2 parts).
Can this be another predefined list that takes a random value from 1-16 for example? OR a randomly generated variable?
Begin Unverified Perl
my $plist = 'QuotaRand';
my $group = RandNum (2019,1,16);
 
if ($group == 1) {
    ADD($plist);
}
elsif ($group == 2-6) {
    ADD($plist, 2, 3);
}
elsif ($group == 7-11) {
    ADD($plist, 1);
    ADD($plist, 3);
}
else {
    ADD($plist, 1, 2);
}
End Unverified
Randomize()

Thank you
That could be done, but a quota question with Least Fill (Percentages) might be better than just using a random number.

If you are going to use a random number, though, you'll either want to use SysRand or use RandNum seeded with the respondent number.  If you seed it with 2019 everytime, you'll always get the same result.

As for the conditionals, the code would be:

$group >= 2 && $group <= 6
Thank you for the reply and suggestion.
First of all your suggestion of the least fill is great, can you tell me how to apply it?
My concerns with that method are:
1-since the first group (that answer the whole questionnaire) is expected to have relatively higher drop-outs, it will be offered more frequent and then this might affect the completion time of the project
2- the survey is conducted over 2 waves & will remain unfinished for some of the respondents during the transition time between the waves ( for retest reliability check) this might affect how the least fill quota cells are calculated

ON the other hand, I wanted to have a constant seed for each respondent, not for the whole survey and that I may be able to regenerate the same number for each respondent as I need to reuse this code in the retest part to make sure they are randomized to the same parts again.  I thought that the respondent number is added automatically and the number I add is like a marker.
your advice and insight is much appreciated
Least fill quotas are a feature new to Lighthouse Studio 9.7.0, but it could be hacked in with some unverified Perl in earlier versions.  However, considering what you've said about drop outs, perhaps least fill isn't the best option for you.  If respondents in group 1 kept dropping out, least fill would just keep sending more respondents into that same group.  If you want 1 in 16 respondents to merely be assigned to group one (rather than 1 in 16 completes being from group one), then you won't want to use least fill.

Wanting to be able to repeat random choices again in the future is the exact reason one should choose RandNum over SysRand.  While Lighthouse Studio does use the respondent number internally in some locations for seeding, nothing to that effect is done in the RandNum function.  "[% RandNum(2019, 1, 16) %]" will always output the same result.  You'll need to manually put the respondent number in like this:

[% RandNum(RespNum(), 1, 16) %]
So the final correct code should be
Begin Unverified Perl
my $plist = 'QuotaRand';
my $group = [% RandNum(RespNum(), 1, 16) %];
  
if ($group == 1) {
    ADD($plist);
}
elsif ($group >= 2 && $group <= 6) {
    ADD($plist, 2, 3);
}
elsif ($group >= 7 && $group <= 11) {
    ADD($plist, 1);
    ADD($plist, 3);
}
else {
    ADD($plist, 1, 2);
}
End Unverified
Randomize()
You'll need to do this when nesting Sawtooth Script inside unverified Perl:

my $group = RANDNUM(RESPNUM(), 1, 16);
thank you for your patience
...