Optimal Sort - any limitation on num of responses?

laura lucchesi's Avatar

laura lucchesi

11 Jul, 2011 03:39 PM

Is there any limitation on the number of completed responses Optimal Sort can handle? Are there any known performance issues with the tool itself when dealing with large sample sizes of about 3000 users.

  1. 2 Posted by William on 11 Jul, 2011 11:32 PM

    William's Avatar

    Hi Laura,

    While we don't force an upper limit on Optimal Sort responses, there are a few things to keep in mind:

    The first thing I should point out is that in an open card sorting exercise where participants are naming their own categories, the more responses you have the harder it is to analyze your results once the survey is finished.

    Of course, this is going to depend on the number of cards you have, and is somewhat mitigated in the case of a closed sort. But if you make the assumption that participants will create between 5 and 10 categories in an open sort, that's potentially 20,000 category names to analyze.

    We personally recommend testing between 50-200 people (especially for your first study). Statistically, there's good evidence (see http://home.comcast.net/~tomtullis/publications/UPA2004CardSorting.pdf) that the correlation between how your tested population groups the cards and how the 'entire' population would group the cards quickly approaches an upper limit where no further insight is necessarily gained.

    The other caveat I should mention is that while we have surveys with 3000 participants in our database, they're very much the exception rather than the rule. For the sake of safety, I'd appreciate it if you could 'throttle' the users you send the survey to, at least initially. We've never had 3000 users attempt to complete a study at the same time, so I can't make any guarantees about the load that might put on our system.

    The number of responses you get will also affect how long the results will take to load, but we've recently tested this functionality and if you're using a modern browser I'm confident you won't run into any problems loading your survey results.

    Best regards
    William
    Optimal Workshop

  2. William closed this discussion on 11 Jul, 2011 11:32 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac