You’ve just finished running your card sort. The study has closed and the data is waiting to be analyzed. It’s time to take a look at the analysis side of card sorting, specifically in our tool OptimalSort. Let’s get started.

‍

A note on analysis 📌

‍

When it comes to analysis, there are essentially two types. There’s exploratory analysis (when you look through data to get impressions, pull out useful ideas and be creative) and statistical analysis (which really just comes down to the numbers). These two types of analysis also go by qualitative and quantitative, respectively.

‍

You’re able to get fantastic insights from both forms.

‍

“Remember that you are the one who is doing the thinking, not the technique… you are the one who puts it all together into a great solution. Follow your instincts, take some risks, and try new approaches.” Donna Spencer, Maadmob.

‍

Getting started with analysis 🏁

‍

Whenever you wrap up a study using our card sorting tool, you’ll want to kick off your analysis by heading to the Results Overview section. It’s here that you’ll be able to see how many people actually took part in the study, the average time taken and general statistics about the study itself.

‍

This is useful data to include in presentations to interested stakeholders, just to give them a more holistic view of your research.

‍

Digging into your participant data ⛏

‍

With the Results Overview section out of the way, you can make your way over to the Participants Table. This is where you can find information about the individual people who took part in your card sort. You can also start to filter your data here.

‍

Here are just a few of the different actions that you can take:

‍

• Review your participants, and include or exclude certain individuals based on their card sorts. This is a useful tool if you want to use your data in different ways.
• Segment and reload your results. This function can allow you to view data from individuals or groups of your choosing.
• Add additional card sorts. If you also decided to run manual (in-person) card sorts using printed cards, you can add this data here.

‍

Analysing open and hybrid card sort data 🕵️♂

‍

The Categories tab is the best place to go for open and hybrid card sort results. Take some time to scan the categories people came up with and you’ll be able to quickly build up a good understanding of their ‘mental models’, or how they perceived the theme of your cards.

‍

Consider how different the categories might look for cards containing food items, for example. Some participants might create categories reflecting supermarket aisles, while others might create categories reflecting food groups.

‍

A good place to get started here is by refining your data. Standardize any categories that have similar labels (whether that’s wording, spelling or capitalizations etc). Hybrid card sorts have some set categories, and these will already be standardized.

‍

Note: Before you start throwing categories with similar labels together, take a closer look to see if people had the same conceptual approach. Here’s an example from our card sorting 101 guide:

‍

Of the 15 groups with the word ‘Animal’ in the label, 13 had a similar set of cards, but two participants had labeled their categories slightly differently (Animals and Environment’ and ‘Animals and Nature’) and had thus included extra cards the others didn’t have (‘Glaciers melting faster than previously thought’, for example).

‍

Reviewing the Similarity Matrix 🤔

‍

One really useful tool for understanding how your participants think is the Similarity Matrix. This view shows you the percentage of people who grouped 2 cards together.

The most closely related pairings are clustered along the right edge. Higher agreement between participants on which cards go together equates to darker and larger clusters.

‍

‍

There are a few different ways to use the insights from the Similarity Matrix:

‍

• Put together a draft website structure based on the clusters you see on the right.
• Identify which card pairings are most common (and as a result should probably go together on your website).
• Identify which card pairings are least common so you don’t need to waste time considering how they might work on your website.

‍

Spotting popular card groupings 🔍

‍

Dendrograms are a tool to enable you to spot popular groups of cards, as well to get a general feel of how similar or different your participants’ card sorts were to each other.

‍

There are two dendrograms to explore:

‍

• More than 30 card sort participants: The Actual Agreement Method (AAM) dendrogram gives you the data straight: “X% of participants agree with this exact grouping”.
• Fewer than 30 card sort participants: The Best Merge Method (BMM) tells you “X% of participants agree with parts of this grouping”, and so enables you to extract as much as you can from the data.

‍

Looking for alternative approaches 👀

‍

The Participant-Centric Analysis (PCA) view can be useful when you have a lot of results. It’s quite simple. Basically, it aims to find the most popular grouping strategy, and then find two more popular alternatives among participants who agreed with the first strategy.

‍

This approach is called Participant-Centric Analysis because every response (from every participant) is treated as a potential solution, and then ranked for similarity with other responses. What this is telling you is that if you see a card sort with a 11/43 agreement score, this means 10 other participants sorted their cards into groups similar to these ones. 

‍

Taking the next step: Run a card sort and try analysis for yourself 🃏

‍

Now that we’ve taken a bit of a deep dive into the analysis side of card sorting in OptimalSort, it’s time to take the tool for a spin and start generating your own data.

‍

Getting started is easy. If you haven’t already, simply sign up for a free account (you don’t need a credit card) and start a card sort. You can also practice by creating a card sort and sending it out to your coworkers, friends or family. Once you start to see results trickling in, you can start to make sense of the data.

‍

For more information, check out the card sorting 101 guide that we’ve put together, or our introduction to card sorting on the Optimal Workshop Blog.

‍

Happy testing! 

Despite the abundance of user research methods, card sorting remains one of the best ways to get into the minds of your users and discover how they understand and categorize information.

Given that one of our most popular tools is an online card sorting tool called OptimalSort (You may have heard of it), we thought that we’d answer some common questions about online card sorting – the research method that OptimalSort uses.

Let’s begin!

1. What’s the difference between online and offline card sorting?

Traditional card sorting can be done using paper cards and hosted in-person, hence “offline”. Online card sorting is pretty much what it sounds like: a card sort hosted over the internet. But there’s a little more to it.

Primarily, running a card sort online as opposed to hosting one in person means that the process becomes much easier to facilitate. Instead of needing to schedule a time for your participants to come into an office, you can simply send them a link to your card sort. Then, they can complete the test in their own time.

Note that the very benefits of online card sorting mean that you can lose certain insights gained from an in-person card sort, like understanding why your participants sort cards in a certain way. There are ways around this, however. For example, you could pair your card sorting tool with an online video recording solution.

2. When should you run an online card sort?

Card sorting is best suited to answering specific, information-related questions. For example, maybe you want to rearrange the layout of your magazine? Or perhaps you need to add several new shopping categories to your website.

In the latter example, card sorting is the perfect technique to find out where people would commonly expect to find those categories on your website. In the card sort, you present participants with a list of cards containing the names of items within certain categories and task them with sorting those items into groups that make sense to them. The end result? You have a clear picture of how your users or customers would arrange the content on your website.

Card sorting is useful when you’ve got the information you need to organize, but you’re just not sure how to organize it.

3. Do I need to compensate participants for taking part in my card sort?

Compensation is tricky when it comes to online testing methods like card sorting. While there are no hard and fast rules, you may find that it’s the best way to incentivize people to take part in your study. Now, taking part in an online card sort is much easier than trekking across town to sit down for a user interview, so you may want to offer participants the chance to win a prize for taking part instead of compensating them directly.

Note: Offering a discount for your product or service is a great way to compensate users and encourage the use of your product. 

4. How do I make sense of the data?

Most card sorting tools offer powerful analysis functionality built right into the tool itself, so all you have to worry about is actually putting the card sort together, sending out the links and promoting it.

Using OptimalSort as an example, let’s take a look at some of the analysis functionality and why it’s useful. Other card sorting tools will likely have different analysis options available.

• Participants Table: Review all of the people who took part in your card sort and segment or exclude them.
• Participant-Centric Analysis (PCA): See the most popular grouping strategies as well as the alternatives among those people who disagreed with the first strategy.
• Dendrograms: Quickly spot popular groups of cards and get a sense of how similar or different your participant’s card sorts were.

5. Is online card sorting expensive?

Online card sorting tools can be expensive, but it’s all relative. As just one example of this, online research platforms mean that you’ll likely be gaining access to a whole host of other tools by signing up for an online card sorting tool.

There’s also the fact that it’s a cheaper exercise overall than in-person card sorting as you won’t have to pay as much for compensation, or even use as much of your own time. Time is money!

6. Can I still get qualitative insights from an online card sort?

You can draw qualitative insights directly from the results of an online card sort, but you can also use online card sorting tools alongside participant recording software to build a more holistic understanding. By using recording software, you’ll be able to watch participants as they complete a card sort, and ask them to talk through what they’re doing to learn why they placed cards in a certain way. 

7. How many participants do I need?

In a nutshell, a larger number than you’d probably bring in for a user interview. Aim for between 20 and 30 participants.

Card sorting (whether it’s performed online or offline) is what’s known as a generative user testing method. This means that you’re typically starting without a design, and you’re using the method to get an idea of how people think with regards to the problem you’re trying to solve. A good example of this would be that you’re building a new website, and are using card sorting to learn how people think the content should be grouped and arranged.

Here’s a great quote from Nielsen Norman Group: “There is great variability in different people’s mental models and in the vocabulary they use to describe the same concepts. We must collect data from a fair number of users before we can achieve a stable picture of the users’ preferred structure and determine how to accommodate differences among users”.

8. How many cards should I use?

We recommend aiming for between 30 and 60 cards, as per our comprehensive 101 guide. Why? Because:

• People will be more likely to complete your card sort.
• You’ll only be able to include the most relevant cards, and be forced to discard the rest.
• You’ll get enough useful data and insights to make informed decisions about your website, app or project.

9. What online card sorting tools are available?

There are a number of online card sorting tools available, including our very own OptimalSort, which is one of the tools included in our platform. OptimalSort has a number of useful features to make it easy to set up and run a card sort with participants based all over the world. Once you’ve gathered all of your responses, built-in analysis features can then help you make sense of the data.

Of course, there are other options available. Take a look at this tools map from User Interviews for a comprehensive overview of the major research tools.

10. What do I do after a card sort?

With your card sort done and dusted, it’s time to take that data and build a draft structure of your website or mobile app. Once you’ve put this rough structure together, you can use tree testing to to see how people navigate through it. We’ve got a guide for that too, which you can read here.

Wrap-up

So that’s 10 questions about online card sorting – answered! If you’re interested in diving straight into a card sort of your own, we obviously recommend giving OptimalSort a try (which you can do for free).

Happy testing!

Few methods surpass card sorting when you need to figure out how your users understand and categorize information.

Whether you’re working on a new website, mobile app, intranet or even a physical store, this user research method is a powerful way of getting into the minds of the people you’re trying to serve.

For those of you unfamiliar with this method or just needing a recap, a card sort involves participants sorting cards containing different items into groups. It’s as simple as that. You can use the results to then determine how to group and label the information on your app or website in a way that makes sense to the people using it.

When to use card sorting

Imagine for a moment that you’re the owner of a small fitness apparel store called FitSmart, and you’ve just ordered in a shipment of new fitness trackers. You need to add these new products to your website, but you’re not exactly sure how they fit in alongside the shoes, clothing, equipment and supplements that you also offer.

This is where card sorting comes into play.

With this user research method, you can quickly determine where people might expect to find a fitness tracker category on your website. The card sort will present the test participants with a list of cards containing the names of items found on your website, and ask users to sort the cards into groups that make sense to them.

Remember: You are not your user, and while it is possible to take an educated guess as to where to position these products, using a card sort will take out the guesswork.

One of the other benefits of card sorting is that you can routinely come back to the method whenever you need to update your website.

How to run your first card sort

Running a card sort is quite straightforward – and much simpler when using an online, unmoderated tool. Yes, you can run a card sort in person using paper cards, but you’ll then also have to coordinate with participants to have them meet you in a physical location, actually host each card sort and carry out all of the analysis manually.

Using an online tool like OptimalSort eliminates all of the admin and instead allows you to get on with the task of actually analyzing your results and making effective decisions. If you’d like to try card sorting, you can give OptimalSort a go for free and then follow this guide to set up your first test. We’d love to hear how you get on!

Pair card sorting with tree testing

When you’re working in the realm of information architecture – whether that’s reorganizing the way your website is laid out or trying to arrange categories in a mobile shopping app – card sorting isn’t the be-all and end-all.

Card sorting can show you which things should go together, but you also need to be able to work out how people make their way through a website structure. This is where a method called tree testing comes into play.

Also known as reverse card sorting, you can use a tree test to evaluate the findability of different items. In a tree test, you task participants with completing a certain action (like finding fitness trackers) and observe them as they navigate through a text-only version of your website structure. By recording every step that they take, including any wrong turns and how long it takes them to complete the task, you can make more informed changes to the specific placement of different pages and elements.

It’s best to use card sorting and tree testing together.

• If you’re building a new website, start with a card sort to group items together, and then use a tree test to put your structure through its paces.
• If you’re trying to fix or update an existing website, start with a tree test to assess the current structure, then move to a card sort to make changes based on the results of the tree test.

Wrap up

Card sorting is one of the most effective ways of building products and services that are intuitive for the people using them. If you’re interested in learning more about this research method, check out the hub page for card sorting on the Optimal Workshop Blog.

When it comes to designing and testing in the world of information architecture, it’s hard to beat card sorting. As a usability testing method, card sorting is easy to set up, simple to recruit for and can supply you with a range of useful insights. But there’s a long-standing debate in the world of card sorting, and that’s whether it’s better to run card sorts in person (moderated) or remotely over the internet (unmoderated).

‍

This article should give you some insight into the world of online card sorting. We've included an analysis of the benefits (and the downsides) as well as why people use this approach. Let's take a look!

‍

How an online card sort works

‍

Running a card sort remotely has quickly become a popular option just because of how time-intensive in-person card sorting is. Instead of needing to bring your participants in for dedicated card sorting sessions, you can simply set up your card sort using an online tool (like our very own OptimalSort) and then wait for the results to roll in.

‍

So what’s involved in a typical online card sort? At a very high level, here’s what’s required. We’re going to assume you’re already set up with an online card sorting tool at this point.

‍

1. Define the cards: Depending on what you’re testing, add the items (cards) to your study. If you were testing the navigation menu of a hotel website, your cards might be things like “Home”, “Book a room”, “Our facilities” and “Contact us”.
2. Work out whether to run a closed or open sort: Determine whether you’ll set the groups for participants to sort cards into (closed) or leave it up to them (open). You may also opt for a mix, where you create some categories but leave the option open for participants to create their own.
3. Recruit your participants: Whether using a participant recruitment service or by recruiting through your own channels, send out invites to your online card sort.
4. Wait for the data: Once you’ve sent out your invites, all that’s left to do is wait for the data to come in and then analyze the results.

‍

That’s online card sorting in a nutshell – not entirely different from running a card sort in person. If you’re interested in learning about how to interpret your card sorting results, we’ve put together this article on open and hybrid card sorts and this one on closed card sorts.

‍

Why is online card sorting so popular?

‍

Online card sorting has a few distinct advantages over in-person card sorting that help to make it a popular option among information architects and user researchers. There are downsides too (as there are with any remote usability testing option), but we’ll get to those in a moment.

‍

Where remote (unmoderated) card sorting excels:

‍

• Time savings: Online card sorting is essentially ‘set and forget’, meaning you can set up the study, send out invites to your participants and then sit back and wait for the results to come in. In-person card sorting requires you to moderate each session and collate the data at the end.
• Easier for participants: It’s not often that researchers are on the other side of the table, but it’s important to consider the participant’s viewpoint. It’s much easier for someone to spend 15 minutes completing your online card sort in their own time instead of trekking across town to your office for an exercise that could take well over an hour.
• Cheaper: In a similar vein, online card sorting is much cheaper than in-person testing. While it’s true that you may still need to recruit participants, you won’t need to reimburse people for travel expenses.
• Analytics: Last but certainly not least, online card sorting tools (like OptimalSort) can take much of the analytical burden off you by transforming your data into actionable insights. Other tools will differ, but OptimalSort can generate a similarity matrix, dendrograms and a participant-centric analysis using your study data.

‍

Where in-person (moderated) card sorting excels:

‍

• Qualitative insights: For all intents and purposes, online card sorting is the most effective way to run a card sort. It’s cheaper, faster and easier for you. But, there’s one area where in-person card sorting excels, and that’s qualitative feedback. When you’re sitting directly across the table from your participant you’re far more likely to learn about the why as well as the what. You can ask participants directly why they grouped certain cards together.

‍

Online card sorting: Participant numbers

‍

So that’s online card sorting in a nutshell, as well as some of the reasons why you should actually use this method. But what about participant numbers? Well, there’s no one right answer, but the general rule is that you need more people than you’d typically bring in for a usability test.

‍

This all comes down to the fact that card sorting is what’s known as a generative method, whereas usability testing is an evaluation method. Here’s a little breakdown of what we mean by these terms:

‍

Generative method: There’s no design, and you need to get a sense of how people think about the problem you’re trying to solve. For example, how people would arrange the items that need to go into your website’s navigation. As Nielsen Norman Group explains: “There is great variability in different people's mental models and in the vocabulary they use to describe the same concepts. We must collect data from a fair number of users before we can achieve a stable picture of the users' preferred structure and determine how to accommodate differences among users”.

‍

Evaluation method: There’s already a design, and you basically need to work out whether it’s a good fit for your users. Any major problems are likely to crop up even after testing 5 or so users. For example, you have a wireframe of your website and need to identify any major usability issues.

‍

Basically, because you’ll typically be using card sorting to generate a new design or structure from nothing, you need to sample a larger number of people. If you were testing an existing website structure, you could get by with a smaller group.

‍

Where to from here?

‍

Following on from our discussion of generative versus evaluation methods, you’ve really got a choice of 2 paths from here if you’re in the midst of a project. For those developing new structures, the best course of action is likely to be a card sort. However, if you’ve got an existing structure that you need to test in order to usability problems and possible areas of improvement, you’re likely best to run a tree test. We’ve got some useful information on getting started with a tree test right here on the blog.

In Part 1 of this series we looked at how to interpret results from open and hybrid card sorts and now in Part 2, we’re going to talk about closed card sorts. In closed card sorts, participants are asked to sort the cards into predetermined categories and are not allowed to create any of their own. You might use this approach when you are constrained by specific category names or as a quick checkup before launching a new or newly redesigned website.In Part 1, we also discussed the two different - but complementary - types of analysis that are generally used together for interpreting card sort results: exploratory and statistical. Exploratory analysis is intuitive and creative while statistical analysis is all about the numbers. Check out Part 1 for a refresher or learn more about exploratory and statistical analysis in Donna Spencer’s book.

‍

‍

Getting started

‍

Closed card sort analysis is generally much quicker and easier than open and hybrid card sorts because there are no participant created category names to analyze - it’s really just about where the cards were placed. There are some similarities about how you might start to approach your analysis process but overall there’s a lot less information to take in and there isn’t much in the way of drilling down into the details like we did in Part 1.Just like with an open card sort, kick off your analysis process by taking an overall look at the results as a whole. Quickly cast your eye over each individual card sort and just take it all in. Look for common patterns in how the cards have been sorted. Does anything jump out as surprising? Are there similarities or differences between participant sorts?

‍

If you’re redesigning an existing information architecture (IA), how do your results compare to the current state? If this is a final check up before launching a live website, how do these results compare to what you learned during your previous research studies?If you ran your card sort using information architecture tool OptimalSort, head straight to the Overview and Participants Table presented in the results section of the tool. If you ran a moderated card sort using OptimalSort’s printed cards, you’ve probably been scanning them in after each completed session, but now is a good time to double check you got them all. And if you didn’t know about this handy feature of OptimalSort, it’s something to keep in mind for next time!

‍

The Participants Table shows a breakdown of your card sorting data by individual participant. Start by reviewing each individual card sort one by one by clicking on the arrow in the far left column next to the Participants numbers. From here you can easily flick back and forth between participants without needing to close that modal window. Don’t spend too much time on this — you’re just trying to get a general impression of how the cards were sorted into your predetermined categories. Keep an eye out for any card sorts that you might like to exclude from the results. For example participants who have lumped everything into one group and haven’t actually sorted the cards.

‍

Don’t worry- excluding or including participants isn’t permanent and can be toggled on or off at anytime.Once you’re happy with the individual card sorts that will and won’t be included in your results visualizations, it’s time to take a look at the Results Matrix in OptimalSort. The Results Matrix shows the number of times each card was sorted into each of your predetermined categories- the higher the number, the darker the shade of blue (see below).

‍

‍

This table enables you to quickly and easily get across how the cards were sorted and gauge the highest and lowest levels of agreement among your participants. This will tell you if you’re on the right track or highlight opportunities for further refinement of your categories.If we take a closer look (see below) we can see that in this example closed card sort conducted on the Dewey Decimal Classification system commonly used in libraries, The Interpretation of Dreams by Sigmund Freud was sorted into ‘Philosophy and psychology’ 38 times in study a completed by 51 participants.

‍

‍

In the real world, that is exactly where that content lives and this is useful to know because it shows that the current state is supporting user expectations around findability reasonably well. Note: this particular example study used image based cards instead of word label based cards so the description that appears in both the grey box and down the left hand side of the matrix is for reference purposes only and was hidden from the participants.Sometimes you may come across cards that are popular in multiple categories. In our example study, How to win friends and influence people by Dale Carnegie, is popular in two categories: ‘Philosophy & psychology’ and ‘Social sciences’ with 22 and 21 placements respectively. The remaining card placements are scattered across a further 5 categories although in much smaller numbers.

‍

‍

When this happens, it’s up to you to determine what your number thresholds are. If it’s a tie or really close like it is in this case, you might review the results against any previous research studies to see if anything has changed or if this is something that comes up often. It might be a new category that you’ve just introduced, it might be an issue that hasn’t been resolved yet or it might just be limited to this one study. If you’re really not sure, it’s a good idea to run some in-person card sorts as well so you can ask questions and gain clarification around why your participants felt a card belonged in a particular category. If you’ve already done that great! Time to review those notes and recordings!You may also find yourself in a situation where no category is any more popular than the others for a particular card. This means there’s not much agreement among your participants about where that card actually belongs. In our example closed card sort study, the World Book Encyclopedia was placed into 9 of 10 categories. While it was placed in ‘History & geography’ 18 times, that’s still only 35% of the total placements for that card- it’s hardly conclusive.

‍

Sometimes this happens when the card label or image is quite general and could logically belong in many of the categories. In this case, an encyclopedia could easily fit into any of those categories and I suspect this happened because people may not be aware that encyclopedias make up a very large part of the category on the far left of the above matrix: ‘Computer science, information & general works’. You may also see this happening when a card is ambiguous and people have to guess where it might belong. Again - if you haven’t already - if in doubt, run some in-person card sorts so you can ask questions and get to the bottom of it!After reviewing the Results Matrix in OptimalSort, visit the Popular Placements Matrix to see which cards were most popular for each of your categories based on how your participants sorted them (see below 2 images).

‍

‍

‍

The diagram shades the most popular placements for each category in blue making it very easy to spot what belongs where in the eyes of your participants. It’s useful for quickly identifying clusters and also highlights the categories that didn’t get a lot of card sorting love. In our example study (2 images above) we can see that ‘Technology’ wasn’t a popular card category choice potentially indicating ambiguity around that particular category name. As someone familiar with the Dewey Decimal Classification system I know that ‘Technology’ is a bit of a tricky one because it contains a wide variety of content that includes topics on medicine and food science - sometimes it will appear as ‘Technology & applied sciences’. These results appear to support the case for exploring that alternative further!

‍

‍

Where to from here?

‍

Now that we’ve looked at how to interpret your open, hybrid and closed card sorts, here are some next steps to help you turn those insights into action!Once you’ve analyzed your card sort results, it’s time to feed those insights into your design process and create your taxonomy which goes hand in hand with your information architecture. You can build your taxonomy out in Post-it notes before popping it into a spreadsheet for review. This is also a great time to identify any alternate labelling and placement options that came out of your card sorting process for further testing.From here, you might move into tree testing your new IA or you might run another card sort focussing on a specific area of your website. You can learn more about card sorting in general via our 101 guide.

‍

When interpreting card sort results, don’t forget to have fun! It’s easy to get overwhelmed and bogged down in the results but don’t lose sight of the magic that is uncovering user insights.I’m going to leave you with this quote from Donna Spencer that summarizes the essence of card sort analysis quite nicely:Remember that you are the one who is doing the thinking, not the technique... you are the one who puts it all together into a great solution. Follow your instincts, take some risks, and try new approaches. - Donna Spencer

‍

‍

Further reading

‍

• Card Sorting 101 – Learn about the differences between open, closed and hybrid card sorts, and how to run your own using OptimalSort.

‍

• Which comes first: card sorting or tree testing? – Tree testing and card sorting can give you insight into the way your users interact with your site, but which comes first?

‍

• How to pick cards for card sorting – What should you put on your cards to get the best results from a card sort? Here are some guidelines.

Cards have been created, sorted and sorted again. The participants are all finished and you’re left with a big pile of awesome data that will help you improve the user experience of your information architecture. Now what?Whether you’ve run an open, hybrid or closed card sort online using an information architecture tool or you’ve run an in person (moderated) card sort, it can be a bit daunting trying to figure out where to start the card sort analysis process.

‍

‍

About this guide

‍

This two-part guide will help you on your way! For Part 1, we’re going to look at how to interpret and analyze the results from open and hybrid card sorts.

• In open card sorts, participants sort cards into categories that make sense to them and they give each category a name of their own making.

• In hybrid card sorts, some of the categories have already been defined for participants to sort the cards into but they also have the ability to create their own.

‍

Open and hybrid card sorts are great for generating ideas for category names and labels and understanding not only how your users expect your content to be grouped but also what they expect those groups to be called.In both parts of this series, I’m going to be talking a lot about interpreting your results using Optimal Workshop’s online card sorting tool, OptimalSort, but most of what I’m going to share is also applicable if you’re analyzing your data using a spreadsheet or using another tool.

‍

‍

Understanding the two types of analysis: exploratory and statistical

‍

Similar to qualitative and quantitative methods, exploratory and statistical analysis in card sorting are two complementary approaches that work together to provide a detailed picture of your results.

‍

• Exploratory analysis is intuitive and creative. It’s all about going through the data and shaking it to see what ideas, patterns and insights fall out. This approach works best when you don’t have the numbers (smaller sample sizes) and when you need to dig into the details and understand the ‘why’ behind the statistics.

‍

• Statistical analysis is all about the numbers. Hard data that tells you exactly how many people expected X to be grouped with Y and more and is very useful when you’re dealing with large sample sizes and when identifying similarities and differences across different groups of people.

‍

Depending on your objectives - whether you are starting from scratch or redesigning an existing IA - you’ll generally need to use some combination of both of these approaches when analyzing card sort results. Learn more about exploratory and statistical analysis in Donna Spencer’s book.

‍

‍

Start with the big picture

‍

When analyzing card sort results, start by taking an overall look at the results as a whole. Quickly cast your eye over each individual card sort and just take it all in. Look for common patterns in how the cards have been sorted and the category names given by participants. Does anything jump out as surprising? Are there similarities or differences between participant sorts? If you’re redesigning an existing IA, how do your results compare to the current state?If you ran your card sort using OptimalSort, your first port of call will be the Overview and Participants Table presented in the results section of the tool.If you ran a moderated card sort using OptimalSort’s printed cards, now is a good time to double check you got them all. And if you didn’t know about this handy feature of OptimalSort, it’s something to keep in mind for next time!The Participants Table shows a breakdown of your card sorting data by individual participant. Start by reviewing each individual card sort one by one by clicking on the arrow in the far left column next to the Participants numbers.

‍

‍

From here you can easily flick back and forth between participants without needing to close that modal window. Don’t spend too much time on this — you’re just trying to get a general impression of what happened.Keep an eye out for any card sorts that you might like to exclude from the results. For example participants who have lumped everything into one group and haven’t actually sorted the cards. Don’t worry - excluding or including participants isn’t permanent and can be toggled on or off at anytime.If you have a good number of responses, then the Participant Centric Analysis (PCA) tab (below) can be a good place to head next. It’s great for doing a quick comparison of the different high-level approaches participants took when grouping the cards.The PCA tab provides the most insight when you have lots of results data (30+ completed card sorts) and at least one of the suggested IAs has a high level of agreement among your participants (50% or more agree with at least one IA).

‍

‍

The PCA tab compares data from individual participants and surfaces the top three ways the cards were sorted. It also gives you some suggestions based on participant responses around what these categories could be called but try not to get too bogged down in those - you’re still just trying to gain an overall feel for the results at this stage.Now is also a good time to take a super quick peek at the Categories tab as it will also help you spot patterns and identify data that you’d like to dive deeper into a bit later on!Another really useful visualization tool offered by OptimalSort that will help you build that early, high-level picture of your results is the Similarity Matrix. This diagram helps you spot data clusters, or groups of cards that have been more frequently paired together by your participants, by surfacing them along the edge and shading them in dark blue. It also shows the proportion of times specific card pairings occurred during your study and displays the exact number on hover (below).

‍

‍

In the above screenshot example we can see three very clear clusters along the edge: ‘Ankle Boots’ to ‘Slippers’ is one cluster, ‘Socks’ to ‘Stockings & Hold Ups’ is the next and then we have ‘Scarves’ to ‘Sunglasses’. These clusters make it easy to spot the that cards that participants felt belonged together and also provides hard data around how many times that happened.Next up are the dendrograms. Dendrograms are also great for gaining an overall sense of how similar (or different) your participants’ card sorts were to each other. Found under the Dendrogram tab in the results section of the tool, the two dendrograms are generated by different algorithms and which one you use depends largely on how many participants you have.

‍

If your study resulted in 30 or more completed card sorts, use the Actual Agreement Method (AAM) dendrogram and if your study had fewer than 30 completed card sorts, use the Best Merge Method (BMM) dendrogram.The AAM dendrogram (see below) shows only factual relationships between the cards and displays scores that precisely tell you that ‘X% of participants in this study agree with this exact grouping’.In the below example, the study shown had 34 completed card sorts and the AAM dendrogram shows that 77% of participants agreed that the cards highlighted in green belong together and a suggested name for that group is ‘Bling’. The tooltip surfaces one of the possible category names for this group and as demonstrated here it isn’t always the best or ‘recommended’ one. Take it with a grain of salt and be sure to thoroughly check the rest of your results before committing!

‍

‍

The BMM dendrogram (see below) is different to the AAM because it shows the percentage of participants that agree with parts of the grouping - it squeezes the data from smaller sample sizes and makes assumptions about larger clusters based on patterns in relationships between individual pairs.The AAM works best with larger sample sizes because it has more data to work with and doesn’t make assumptions while the BMM is more forgiving and seeks to fill in the gaps.The below screenshot was taken from an example study that had 7 completed card sorts and its BMM dendrogram shows that 50% of participants agreed that the cards highlighted in green down the left hand side belong to ‘Accessories, Bottoms, Tops’.

‍

‍

Drill down and cross-reference

‍

Once you’ve gained a high level impression of the results, it’s time to dig deeper and unearth some solid insights that you can share with your stakeholders and back up your design decisions.Explore your open and hybrid card sort data in more detail by taking a closer look at the Categories tab. Open up each category and cross-reference to see if people were thinking along the same lines.Multiple participants may have created the same category label, but what lies beneath could be a very different story. It’s important to be thorough here because the next step is to start standardizing or chunking individual participant categories together to help you make sense of your results.In open and hybrid sorts, participants will be able to label their categories themselves. This means that you may identify a few categories with very similar labels or perhaps spelling errors or different formats. You can standardize your categories by merging similar categories together to turn them into one.OptimalSort makes this really easy to do - you pretty much just tick the boxes alongside each category name and then hit the ‘Standardize’ button up the top (see below). Don’t worry if you make a mistake or want to include or exclude groupings; you can unstandardize any of your categories anytime.

‍

‍

Once you’ve standardized a few categories, you’ll notice that the Agreement number may change. It tells you how many participants agreed with that grouping. An agreement number of 1.0 is equal to 100% meaning everyone agrees with everything in your newly standardized category while 0.6 means that 60% of your participants agree.Another number to watch for here is the number of participants who sorted a particular card into a category which will appear in the frequency column in dark blue in the right-hand column of the middle section of the below image.

‍

‍

‍

From the above screenshot we can see that in this study, 18 of the 26 participant categories selected agree that ‘Cat Eye Sunglasses’ belongs under ‘Accessories’.Once you’ve standardized a few more categories you can head over to the Standardization Grid tab to review your data in more detail. In the below image we can see that 18 participants in this study felt that ‘Backpacks’ belong in a category named ‘Bags’ while 5 grouped them under ‘Accessories’. Probably safe to say the backpacks should join the other bags in this case.

‍

‍

So that’s a quick overview of how to interpret the results from your open or hybrid card sorts.Here's a link to Part 2 of this series where we talk about interpreting results from closed card sorts as well as next steps for applying these juicy insights to your IA design process.

‍

‍

Further reading

• Card Sorting 101 – Learn about the differences between open, closed and hybrid card sorts, and how to run your own using OptimalSort.
• Which comes first: card sorting or tree testing? – Tree testing and card sorting can give you insight into the way your users interact with your site, but which comes first?
• How to pick cards for card sorting – What should you put on your cards to get the best results from a card sort? Here are some guidelines.