Optimal Sort - any limitation on num of responses?
laura lucchesi
Is there any limitation on the number of completed responses Optimal Sort can handle? Are there any known performance issues with the tool itself when dealing with large sample sizes of about 3000 users.
Comments are currently closed for this discussion. You can start a new one.
Support Staff 2 Posted by William on 11 Jul, 2011 11:32 PM
Hi Laura,
While we don't force an upper limit on Optimal Sort responses, there are a few things to keep in mind:
The first thing I should point out is that in an open card sorting exercise where participants are naming their own categories, the more responses you have the harder it is to analyze your results once the survey is finished.
Of course, this is going to depend on the number of cards you have, and is somewhat mitigated in the case of a closed sort. But if you make the assumption that participants will create between 5 and 10 categories in an open sort, that's potentially 20,000 category names to analyze.
We personally recommend testing between 50-200 people (especially for your first study). Statistically, there's good evidence (see http://home.comcast.net/~tomtullis/publications/UPA2004CardSorting.pdf) that the correlation between how your tested population groups the cards and how the 'entire' population would group the cards quickly approaches an upper limit where no further insight is necessarily gained.
The other caveat I should mention is that while we have surveys with 3000 participants in our database, they're very much the exception rather than the rule. For the sake of safety, I'd appreciate it if you could 'throttle' the users you send the survey to, at least initially. We've never had 3000 users attempt to complete a study at the same time, so I can't make any guarantees about the load that might put on our system.
The number of responses you get will also affect how long the results will take to load, but we've recently tested this functionality and if you're using a modern browser I'm confident you won't run into any problems loading your survey results.
Best regards
William
Optimal Workshop
William closed this discussion on 11 Jul, 2011 11:32 PM.