The key purpose of running a card sort is to learn something new about how people conceptualize and organize the information that’s found on your website. The insights you gain from running a card sort can then help you develop a site structure with content labels or headings that best represent the way your users think about this information. Card sorts are in essence a simple technique, however it’s the details of the sort that can determine the quality of your results.
Adding context to cards in OptimalSort – descriptions, links and images
In most cases, each item in a card sort has only a short label, but there are instances where you may wish to add additional context to the items in your sort. Currently, the cards tab in OptimalSort allows you to include a tooltip description, a link within the tooltip description or to format the card as an image (with or without a label).
We generally don’t recommend using tooltip descriptions and links, unless you have a specific reason to do so. It’s likely that they’ll provide your participants with more information than they would normally have when navigating your website, which may in turn influence your results by leading participants to a particular solution.
Legitimate reasons that you may want to use descriptions and links include situations where it’s not possible or practical to translate complex or technical labels (for example, medical, financial, legal or scientific terms) into plain language, or if you’re using a card sort to understand your participants’ preferences or priorities.
If you do decide to include descriptions in your sort, it’s important that you follow the same guidelines that you would otherwise follow for writing card labels. They should be easy for your participants to understand and you should avoid obvious patterns, for example repeating words and phrases, or including details that refer to the current structure of the website.
A quick survey of how card descriptions are used in OptimalSort
I was curious to find out how often people were including descriptions in their card sorts, so I asked our development team to look into this data. It turns out that around 15% of cards created in OptimalSort have at least some text entered in the description field. In order to dig into the data a bit further, both Ania and I reviewed a random sample of recent sorts and noted how descriptions were being used in each case.
We found that out of the descriptions that we reviewed, 40% (6% of the total cards) had text that should not have impacted the sort results. Most often, these cards simply had the card label repeated in the description (to be honest, we’re not entirely sure why so many descriptions are being used this way! But it’s now in our roadmap to stop this from happening — stay tuned!). Approximately 20% (3% of the total cards) used descriptions to add context without obviously leading participants, however another 40% of cards have descriptions that may well lead to biased results. On occasion, this included linking to the current content or using what we assumed to be the current top level heading within the description.
Testing the effect of card descriptions on sort results
So, how much influence could potentially leading card descriptions have on the results of a card sort? I decided to put it to the test by running a series of card sorts to compare the effect of different descriptions. As I also wanted to test the effect of linking card descriptions to existing content, I had to base the sort on a live website. In addition, I wanted to make sure that the card labels and descriptions were easily comprehensible by a general audience, but not so familiar that participants were highly likely to sort the cards in a similar manner.
I selected the government immigration website New Zealand Now as my test case. This site, which provides information for prospective and new immigrants to New Zealand, fit the above criteria and was likely unfamiliar to potential participants.
Navigating the New Zealand Now website
When I reviewed the New Zealand Now site, I found that the top level navigation labels were clear and easy to understand for me personally. Of course, this is especially important when much of your target audience is likely to be non-native English speaking! On the whole, the second level headings were also well-labeled, which meant that they should translate to cards that participants were able to group relatively easily.
There were, however, a few headings such as “High quality” and “Life experiences”, both found under “Study in New Zealand”, which become less clear when removed from the context of their current location in the site structure. These headings would be particularly useful to include in the test sorts, as I predicted that participants would be more likely to rely on card descriptions in the cases where the card label was ambiguous.
I selected 30 headings to use as card labels from under the sections “Choose New Zealand”, “Move to New Zealand”, “Live in New Zealand”, “Work in New Zealand” and “Study in New Zealand” and tweaked the language slightly, so that the labels were more generic.
I then created four separate sorts in OptimalSort:Round 1: No description: Each card showed a heading only — this functioned as the control sort
Round 2: Site section in description: Each card showed a heading with the site section in the description
Round 3: Short description: Each card showed a heading with a short description — these were taken from the New Zealand Now topic landing pages
Round 4:Link in description: Each card showed a heading with a link to the current content page on the New Zealand Now website
For each sort, I recruited 30 participants. Each participant could only take part in one of the sorts.
What the results showed
An interesting initial finding was that when we queried the participants following the sort, only around 40% said they noticed the tooltip descriptions and even fewer participants stated that they had used them as an aid to help complete the sort.
Of course, what people say they do does not always reflect what they do in practice! To measure the effect that different descriptions had on the results of this sort, I compared how frequently cards were sorted with other cards from their respective site sections across the different rounds.Let’s take a look at the “Study in New Zealand” section that was mentioned above. Out of the five cards in this section,”Where & what to study”, “Everyday student life” and “After you graduate” were sorted pretty consistently, regardless of whether a description was provided or not. The following charts show the average frequency with which each card was sorted with other cards from this section. For example in the control round, “Where & what to study” was sorted with “After you graduate” 76% of the time and with “Everyday day student life” 70% of the time, but was sorted with “Life experiences” or “High quality” each only 10% of the time. This meant that the average sort frequency for this card was 42%.
On the other hand, the cards “High quality” and “Life experiences” were sorted much less frequently with other cards in this section, with the exception of the second sort, which included the site section in the description.These results suggest that including the existing site section in the card description did influence how participants sorted these cards — confirming our prediction! Interestingly, this round had the fewest number of participants who stated that they used the descriptions to help them complete the sort (only 10%, compared to 40% in round 3 and 20% in round 4).Also of note is that adding a link to the existing content did not seem to increase the likelihood that cards were sorted more frequently with other cards from the same section. Reasons for this could include that participants did not want to navigate to another website (due to time-consciousness in completing the task, or concern that they’d lose their place in the sort) or simply that it can be difficult to open a link from the tooltip pop-up.
What we can take away from these results
This quick investigation into the impact of descriptions illustrates some of the intricacies around using additional context in your card sorts, and why this should always be done with careful consideration. It’s interesting that we correctly predicted some of these results, but that in this case, other uses of the description had little effect at all. And the results serve as a good reminder that participants can often be influenced by factors that they don’t even recognise themselves!If you do decide to use card descriptions in your cards sorts, here are some guidelines that we recommend you follow:
Avoid repeating words and phrases, participants may sort cards by pattern-matching rather than based on the actual content
Avoid alluding to a predetermined structure, such as including references to the current site structure
If it’s important that participants use the descriptions to complete the sort, you should mention this in your task instructions. It may also be worth asking them a post-survey question to validate if they used them or not
We’d love to hear your thoughts on how we tested the effects of card descriptions and the results that we got. Would you have done anything differently?Have you ever completed a card sort only to realize later that you’d inadvertently biased your results? Or have you used descriptions in your card sorts to meet a genuine need? Do you think there’s a case to make descriptions more obvious than just a tooltip, so that when they are used legitimately, most participants don’t miss this information?
This morning Ieavesdroppeda conversation between Amy Worley (@worleygirl) and The SemanticWill™ (@semanticwill) on "the twitters".Aside from recommending two books by Donna Spencer (@maadonna), I asked Nicole Kaufmann, one of the friendly consultants at Optimal Usability, if she had any advice for Amy about reorganising 404 books into categories that make more sense.I don't know Amy's email address and this is much too long for a tweet. In any case I thought it might be helpful for someone else too so here's what Nicole had to say:In general I would recommend having at least three sources of information (e.g. 1x analytics + 1 open card sort + 1 tree test, or 2 card sorts + 1 tree test) in order to come up with a useful and reliable categorisation structure.Here are four options for how you could consider approaching it (starting with my most preferred to least preferred):
Option A
Pick the 20-25 cards you think will be the most difficult and 20-25 cards that you think will be the easiest to sort and test those in one open card sort.
Based on the results create one or two sets of categories structures which you can test in a one or two closed card sorts. Consider replacing about half of the tested cards with new ones.
Based on the results of those two rounds of card sorting, create a categorisation structure and pick a set of difficult cards which you can turn into tasks which you can test in a tree test.
Plus: Categorisation is revised between studies. Relative easy analysis.
Minus: Not all cards have been tested. Depending on the number of studies needs about 80-110 participants. Time intensive.
Option B
Pick the 20-25 cards you think will be the most difficult and 20-25 cards that you think will be the easiest to sort and test those in one open card sort.
Based on the results do a closed card sort(s) excluding the easiest cards and adding some new cards which haven't been tested before.
Plus: Card sort with reasonable number of cards, only 40-60 participants needed, quick to analyse.
Minus: Potential bias and misleading results if the wrong cards are picked.
Option C
Create your own top level categories (5-8) (could be based on a card sort) and assign cards to these categories, then pick random cards within those categories and set up a card sort for each (5-8).
Based on the results create a categorisation structure and a set of task which will be tested in a tree test.
Plus: Limited set of card sorts with reasonable number of cards, quick to analyse. Several sorts for comparison.
Minus: Potential bias and misleading results if the wrong top categories are picked. Potentially different categorisation schemes/approaches for each card sort, making them hard to combine into one solid categorisation structure.
Option D
Approach: Put all 404 cards into 1 open card sort, showing each participant only 40-50 cards.
Plus: All cards will have been tested
Do a follow up card sort with the most difficult and easiest cards (similar to option B).
Minus: You need at least 200-300 completed responses to get reasonable results. Depending on your participant sources it may take ages to get that many participants.
A great information architecture (IA) is essential for a great user experience (UX). And testing your website or app’s information architecture is necessary to get it right.
Card sorting and tree testing are the very best UX research methods for exactly this. But the big question is always: which one should you use, and when? Very possibly you need both. Let’s find out with this quick summary.
What is card sorting and tree testing? 🧐
Card sorting is used to test the information architecture of a website or app. Participants group individual labels (cards) into different categories according to criteria that makes best sense to them. Each label represents an item that needs to be categorized. The results provide deep insights to guide decisions needed to create an intuitive navigation, comprehensive labeling and content that is organized in a user-friendly way.
Tree testing is also used to test the information architecture of a website or app. When using tree testing participants are presented with a site structure and a set of tasks they need to complete. The goal for participants is to find their way through the site and complete their task. The test shows whether the structure of your website corresponds to what users expect and how easily (or not) they can navigate and complete their tasks.
What are the differences? 🂱 👉🌴
Card sorting is a UX research method which helps to gather insights about your content categorization. It focuses on creating an information architecture that responds intuitively to the users’ expectations. Things like which items go best together, the best options for labeling, what categories users expect to find on each menu.
Doing a simple card sort can give you all those pieces of information and so much more. You start understanding your user’s thoughts and expectations. Gathering enough insights and information to enable you to develop several information architecture options.
Tree testing is a UX research method that is almost a card sort in reverse. Tree testing is used to evaluate an information architecture structure and simply allows you to see what works and what doesn’t.
Using tree testing will provide insights around whether your information architecture is intuitive to navigate, the labels easy to follow and ultimately if your items are categorized in a place that makes sense. Conversely it will also show where your users get lost and how.
What method should you use? 🤷
You’ve got this far and fine-tuning your information architecture should be a priority. An intuitive IA is an integral component of a user-friendly product. Creating a product that is usable and an experience users will come back for.
If you are still wondering which method you should use - tree testing or card sorting. The answer is pretty simple - use both.
Just like many great things, these methods work best together. They complement each other, allowing you to get much deeper insights and a rounded view of how your IA performs and where to make improvements than when used separately. We cover more reasons why card sorting loves tree testing in our article which dives deeper into why to use both.
Ok, I'm using both, but which comes first? 🐓🥚
Wanting full, rounded insights into your information architecture is great. And we know that tree testing and card sorting work well together. But is there an order you should do the testing in? It really depends on the particular context of your research - what you’re trying to achieve and your situation.
Tree testing is a great tool to use when you have a product that is already up and running. By running a tree test first you can quickly establish where there may be issues, or snags. Places where users get caught and need help. From there you can try and solve potential issues by moving on to a card sort.
Card sorting is a super useful method that can be instigated at any stage of the design process, from planning to development and beyond. As long as there is an IA structure that can be tested again. Testing against an already existing website navigation can be informative. Or testing a reorganization of items (new or existing) can ensure the organization can align with what users expect.
However, when you decide to implement both of the methods in your research, where possible, tree testing should come before card sorting. If you want a little more on the issue have a read of our article here.
Check out our OptimalSort and Treejack tools - we can help you with your research and the best way forward. Wherever you might be in the process.
Despite the abundance of user research methods, card sorting remains one of the best ways to get into the minds of your users and discover how they understand and categorize information.
Given that one of our most popular tools is an online card sorting tool called OptimalSort (You may have heard of it), we thought that we’d answer some common questions about online card sorting – the research method that OptimalSort uses.
Let’s begin!
1. What’s the difference between online and offline card sorting?
Traditional card sorting can be done using paper cards and hosted in-person, hence “offline”. Online card sorting is pretty much what it sounds like: a card sort hosted over the internet. But there’s a little more to it.
Primarily, running a card sort online as opposed to hosting one in person means that the process becomes much easier to facilitate. Instead of needing to schedule a time for your participants to come into an office, you can simply send them a link to your card sort. Then, they can complete the test in their own time.
Note that the very benefits of online card sorting mean that you can lose certain insights gained from an in-person card sort, like understanding why your participants sort cards in a certain way. There are ways around this, however. For example, you could pair your card sorting tool with an online video recording solution.
2. When should you run an online card sort?
Card sorting is best suited to answering specific, information-related questions. For example, maybe you want to rearrange the layout of your magazine? Or perhaps you need to add several new shopping categories to your website.
In the latter example, card sorting is the perfect technique to find out where people would commonly expect to find those categories on your website. In the card sort, you present participants with a list of cards containing the names of items within certain categories and task them with sorting those items into groups that make sense to them. The end result? You have a clear picture of how your users or customers would arrange the content on your website.
Card sorting is useful when you’ve got the information you need to organize, but you’re just not sure how to organize it.
3. Do I need to compensate participants for taking part in my card sort?
Compensation is tricky when it comes to online testing methods like card sorting. While there are no hard and fast rules, you may find that it’s the best way to incentivize people to take part in your study. Now, taking part in an online card sort is much easier than trekking across town to sit down for a user interview, so you may want to offer participants the chance to win a prize for taking part instead of compensating them directly.
Note: Offering a discount for your product or service is a great way to compensate users and encourage the use of your product.
4. How do I make sense of the data?
Most card sorting tools offer powerful analysis functionality built right into the tool itself, so all you have to worry about is actually putting the card sort together, sending out the links and promoting it.
Using OptimalSort as an example, let’s take a look at some of the analysis functionality and why it’s useful. Other card sorting tools will likely have different analysis options available.
Participants Table: Review all of the people who took part in your card sort and segment or exclude them.
Participant-Centric Analysis (PCA): See the most popular grouping strategies as well as the alternatives among those people who disagreed with the first strategy.
Dendrograms: Quickly spot popular groups of cards and get a sense of how similar or different your participant’s card sorts were.
5. Is online card sorting expensive?
Online card sorting tools can be expensive, but it’s all relative. As just one example of this, online research platforms mean that you’ll likely be gaining access to a whole host of other tools by signing up for an online card sorting tool.
There’s also the fact that it’s a cheaper exercise overall than in-person card sorting as you won’t have to pay as much for compensation, or even use as much of your own time. Time is money!
6. Can I still get qualitative insights from an online card sort?
You can draw qualitative insights directly from the results of an online card sort, but you can also use online card sorting tools alongside participant recording software to build a more holistic understanding. By using recording software, you’ll be able to watch participants as they complete a card sort, and ask them to talk through what they’re doing to learn why they placed cards in a certain way.
7. How many participants do I need?
In a nutshell, a larger number than you’d probably bring in for a user interview. Aim for between 20 and 30 participants.
Card sorting (whether it’s performed online or offline) is what’s known as a generative user testing method. This means that you’re typically starting without a design, and you’re using the method to get an idea of how people think with regards to the problem you’re trying to solve. A good example of this would be that you’re building a new website, and are using card sorting to learn how people think the content should be grouped and arranged.
Here’s a great quote from Nielsen Norman Group: “There is great variability in different people’s mental models and in the vocabulary they use to describe the same concepts. We must collect data from a fair number of users before we can achieve a stable picture of the users’ preferred structure and determine how to accommodate differences among users”.
People will be more likely to complete your card sort.
You’ll only be able to include the most relevant cards, and be forced to discard the rest.
You’ll get enough useful data and insights to make informed decisions about your website, app or project.
9. What online card sorting tools are available?
There are a number of online card sorting tools available, including our very own OptimalSort, which is one of the tools included in our platform. OptimalSort has a number of useful features to make it easy to set up and run a card sort with participants based all over the world. Once you’ve gathered all of your responses, built-in analysis features can then help you make sense of the data.
Of course, there are other options available. Take a look at this tools map from User Interviews for a comprehensive overview of the major research tools.
10. What do I do after a card sort?
With your card sort done and dusted, it’s time to take that data and build a draft structure of your website or mobile app. Once you’ve put this rough structure together, you can use tree testing to to see how people navigate through it. We’ve got a guide for that too, which you can read here.
Wrap-up
So that’s 10 questions about online card sorting – answered! If you’re interested in diving straight into a card sort of your own, we obviously recommend giving OptimalSort a try (which you can do for free).
