From time to time I hear people say that they prefer online card sorting to offline card sorting or vice versa. I think they complement each other! (and you should do both)

Let’s start with a quick rundown on the major differences in outcomes from moderated and unmoderated card sorting.

Remote & Unmoderated Card Sorting (Online):

• Unlimited scale. You can have as many participants as required to get the answer you need.
• Much closer to “fire & forget”. Set up a study, fire it out to potential participants, enjoy the afternoon in the sun.
• Relatively cheap. Compared to the cost of having a facilitator, note taker, clients on site, reception, coffees, compensation… remote testing is clearly cheaper to conduct.
• It can be difficult to know why things happen. Qualitative results are not nearly as apparent because participants are not facilitated, moderated, steered and often not recorded. You don’t get to hear them thinking out loud or discussing decisions.
• Great for gathering quantitative results. If you have a hunch of your own or as a result of qualitative tests then remote and unmoderated user testing is a great way to back it up with some numbers.

In-Person & Moderated Card Sorting (Offline):

• Limited scale. You can only bring in as many participants as you can afford in terms of time and budget.
• Relatively heavy investment per participant. Each participant will have associated costs and will create work for you. (I’m not saying it isn’t worthwhile, it generally is, I’m just pointing out the differences)
• Great for gathering qualitative results. This is where you get insight into how people feel about what they’re doing or saying in the study.
• It is usually too expensive to get quantitative results from moderated testing. Yes, you will undoubtedly uncover most of the problems and convince yourself that something must be done, but many situations call for more.

So what should you do?

I recommend that people conduct from 1 to 5 offline, in-person and moderated card sorts to get a good understanding for themselves of how other people would organise their content and the rationale for it. Then I suggest that people conduct an online study using OptimalSort to put some numbers behind the hunches. By the way, I don’t mean to belittle any professional observations by calling them hunches, I’m just making the point that however duly convinced you might be it is usually not unreasonable for a stakeholder to want more data if a change will impact thousands or potentially millions of other people (or dollars for that matter).

If you are fortunate enough to have crystal clear direction from your qualitative research to propose an immediate way forward then I suggest you could skip the online card sort and move directly to validating your proposed new information architecture using tree testing. Either way you should be validating your chosen labels and content hierarchy using Treejack after a card sort.

We believe there is so much value in both qualitative and quantitative research techniques that we want you to do both. To assist you with this we have recently implemented an important change to OptimalSort: You can now print your OptimalSort cards (from a generated PDF) for moderated and in-person paper based card sorting and easily get the results back into OptimalSort for analysis alongside your quantitative research data. Hooray!

Step 1: Print the cards

OptimalSort's printed cards are printed with crop marks for easy cutting

Step 2: Sort the cards

Please don't analyse this card sort. I just laid them out to look pretty.

Step 3: Scan the groups back into OptimalSort

OptimalSort works with common barcode scanners so that you can quickly get your results into the tool for analysis

I’d love to know what you think of this new feature and whether it will be useful to you in your own card sorts. It certainly beats trying to moderate card sorts around a screen or retrospectively entering participant sorts by doing multiple sorts yourself (you know who you are!).

Two days ago I presented this poster to the crowd at the IA Summit 2011 in Denver, Colorado for the Poster Session. We’re trying to address two issues with Card Sort Analysis and this poster is a discussion piece for a proposed new algorithm for analysis. The two issues:

1. Current methods for Card Sort Analysis are essentially qualitative. Although this is very useful, there are times when it is desirable to use a larger data set. Quantitative Card Sort Analysis using current methods is difficult, or damn near impossible with hundreds or thousands of results.
2. Current visualizations for presenting Card Sort Analysis (dendrograms and similarity matrices) are not very helpful at showing alternate popular mental models that might come through in the raw data. Understanding alternate models can help you decide what to put in a sidebar or footer (for example) or provide valuable insights for second tier navigation or even copy writing. Traditionally you would need to wade through a spreadsheet to uncover these insights.

Participant Centric Information Architecture Analysis

In short, we test each card sort result against all the others and come up with an “acceptability score” which represents the degree to which each participant agrees with the other results. In this way we can establish which particular results is most acceptable to the population, and from there, we can answer the question: “Of those who do not agree with this particular IA, how would they prefer to group the cards?”.

We have already developed a working prototype of our Participant Centric Analysis Method and hope to integrate the new visualization into OptimalSort in the near future. We’d love to hear any feedback you might have on this new method.

Download the poster (PDF)