Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Quota Problem in Pliot

Hi Sawtooth supporter,

I met a problem about my quota during the pilot. It was fine during the test.

Our survey has three parts of the experiment, and each part has few treatment groups (specifically, 3 treatments in part1; 2 treatments in part 2; 4 treatments in part 3; 2 treatments in part 4). details of the quota setting like this: Quota 1-3 belong to part 1; Quota 4-5 belong to part2; quota 6-9 belong to part 3; and quota 10-11 belong to part 4.

From the pilot results, we found that respondents from the last quotas of  the first three parts cannot complete the survey. More specifically, the complete rate of quota 3, quota 5, quota9 are 67%, 94%, and 44% respectively. But other quotas' complete rate is 100%.

I am guessing this may be because once the survey system considers the last stage quotas are full, it will not allow the people who are doing the survey to complete. We got the total number of respondents that equals to the number we set for each part limit, but, we cannot get enough the complete number in the last quotas of the first three part.

I don't know if this problem is caused by I chose ''do not use predictive quota control'' in advanced quota setting. In order to solve this problem, should I change to ''use predictive quota control'' and select ''use projected completion rate'' and setting 60 percent of in progress?
Please let me know if you need more details.

Many thanks
Hanjin
asked Mar 16, 2018 by hanjin (350 points)

1 Answer

0 votes
If you have multiple quotas, you are correct that a respondent has to qualify for a cell in a quota in order to continue on in the survey.  We define qualify as both the logic matches up and there are spots remaining in the quota.  For example, if I have a quota on hearing about a brand, if that cell fills up, the next person who takes the survey and indicates they have heard of the correct brand will still follow the same logic and someone who did not hear about it (typically sent to a terminate point).

Perhaps you need to either add an "overflow" cell to a quota, where the logic is set to "always qualify" and the limit is very high?  Or maybe you need to set up some skip logic so that a respondent skips over a quota if the quota is full?  The way to answer this question is to step through the logic of your survey, especially thinking about what happens if someone hits a quota that has filled up.  You might also leverage the data generator built in to the software to simulate respondents going through the survey to perhaps illuminate if someone can get through the survey or not?

Predictive quota control just closes a cell early based on the assumption of how likely a respondent who is in process will complete.  For example, if I have 50 males and 50 females, the survey will let people pass through those quotas until the 50 completes has been met in each group.  With predictive quota control, the quota would close early if the number of completes plus the number of in progress males equals 50 (or a little more depending on your projected completion rate).
answered Mar 16, 2018 by Brian McEwan Gold Sawtooth Software, Inc. (41,270 points)
...