Optimal Blog

Whether it’s Fortune 500 companies or tiny startups, people are recognizing the value of building products with a user-first methodology.But it’s not enough to merely say “we’re doing research”, it has to be the right UX research. Research that combines richness of different people's experiences and behavioral insights with tangible numbers and metrics. Key to this is an approach called mixed methods research.

‍

Here, we’ll dive into the what and why of mixed methods research and cover a few examples of the approach.

‍

What is mixed methods research? 🔬

‍

Mixed methods isn’t some overly complicated practice that’ll take years to master — it simply refers to answering research questions through a combination of qualitative and quantitative data. This might mean running both interviews and surveys as part of a research project or complementing diary study data with analytics looking at the usage of a particular feature.A basic mixed methods question could be: “What are the key tasks people perform on my website?”. To answer this, you’d look at analytics to understand how people navigate through the page and conduct user interviews to better understand why they use the page in the first place. We’ve got more examples below.

‍

It makes sense: using both qualitative and quantitative methods to answer a single research question will mean you’re able to build a more complete understanding of the topic you’re investigating. Quantitative data will tell you what is happening and help you understand magnitude, while qualitative data can tell you why something is happening. Each type of data has its shortcomings, and by using a mixed methods approach you’re able to generate a clearer overall picture.

‍

When should you use mixed methods? 🧐

‍

There’s really no “time to do mixed methods research”. Ideally, for every research question you have, evaluate which qualitative and quantitative methods are most likely to give you the best data. More often than not, you’ll benefit from using both approaches.We’ve put together a few examples of mixed methods research to help you generate your own UX research questions.

‍

Examples of mixed methods research 👨🏫

‍

Imagine this. You’re on the user research team at BananaBank, a fictional bank. You and your team want to investigate how the bank’s customers currently use their digital banking services so your design team can make some user-focused improvements.We’ve put together a few research questions based on this goal that would best be served by a mixed methods approach.

‍

Question 1: How does people’s usage of online banking differ between desktop and the app?

‍

• The value of quantitative methods: The team can view usage analytics (How many people use the desktop app versus the mobile app) and look at feature usage statistics.
• The value of qualitative methods: Interviews with users can answer all manner of questions. For example, the research team might want to find out how customers make their way through certain parts of the interface. Usability testing is an opportunity to watch users as they attempt various tasks (for example, making a transaction).

‍

Question 2: How might you better support people to reach their savings goals?

‍

• The value of quantitative methods: The team can review current saving behavior patterns, when people stop saving, the longevity of term deposits and other savings-related actions.
• The value of qualitative methods: Like the first question, the team can carry out user interviews, or conduct a diary study to better understand how people set and manage savings goals in real life and what usually gets in the way.

‍

Question 3: What are the problem areas in our online signup form?

‍

• The value of quantitative methods: The team can investigate where people get stuck on the current form, how frequently people run into error messages and the form fields that people struggle to fill out or leave blank.
• The value of qualitative methods: The team can observe people as they make their way through the signup form.

‍

Mixed methods = holistic understanding 🤔

‍

As we touched on at the beginning of this article, mixed methods research isn’t a technique or methodology, it’s more a practice that you should develop to gain a more holistic understanding of the topic you’re investigating. What’s more, using both types of methods will often mean you’re able to validate the output of one method by using another.When you plan your next research activity, consider complementing it with additional data to generate a more comprehensive picture of your research problem.

‍

Further reading 📚🎧☕

‍

• Which comes first: card sorting or tree testing? – We run through why you should use card sorting and tree testing, and which one to use first.
• How to write great questions for your research – Whether you opt for a qualitative or quantitative method – or both – learning how to write great research questions is key.

As the focus on user-centered design continues to grow in organizations around the world, we’ll also need effective leaders to guide UX teams. But what makes a great UX leader?

Leadership may come as naturally as breathing to some people, but most of us need some guidance along the way. We created this article to pull together a few tips for effectively running UX teams, but be sure to leave a comment if you think we’ve missed anything. After all, part of what makes a great leader is being able to take feedback and to learn from others!

The difference between a manager and a leader

There’s a pretty clear distinction between managers and leaders. As a leader, your job isn’t necessarily to manage and tell people what to do, but instead to lead. You should enable your team to succeed by providing them with the tools and resources they need.

Know your team’s strengths and weaknesses

Intel’s Andy Grove, who infamously ruled the Silicon Valley semiconductor company with an iron fist, may be a polarizing figure in the leadership sphere, but he did institute (or at least help popularize) some techniques that are still widely practised today. One of these was to sit in an office cube with his fellow employees, instead of in a siloed office by himself. There’s a good lesson here. Instead of sealing yourself away from your team, immerse yourself in their environment and their work. You’ll develop a much better understanding of the types of problems they deal with on a daily basis and as a result be in a better position to help them.

You can also take this a step further and conduct an audit of your team’s strengths and weaknesses. Also known as a skills audit, this process is more commonly performed in organizations at scale, but it’s a good way to show you where your capabilities lie – even in small teams. With an intimate understanding of your UX team you’ll be in a good position to assess which projects your team can and can’t take on at any given time.

Taking this process even further, you can undertake a skills audit of yourself. If you want to develop yourself as a leader, you have to understand your own strengths and weaknesses.

This quote by Donald Rumsfeld, although it applies to crisis management, provides a great way to self-audit: “There are known knowns: there are things we know we know. We also know there are known unknowns: That is to say, we know there are some things we do not know. But there are also unknown unknowns: the things we don't know we don't know". You can see a visual example of this in the Johari Window:

Source: Wikipedia

Here’s how you can take this approach and use it for yourself:

• Identify your known unknowns: Skills you don’t currently possess that you’re able to recognize you need yourself.
• Identify your unknown unknowns: Skills you don’t know you don’t currently have, but which your team can identify by asking them.

When it comes to projects, be inclusive

NASA astronaut Frank Borman, echoing a sentiment since shared by many people who’ve been to space, said: “When you're finally up on the moon, looking back at the earth, all these differences and nationalistic traits are pretty well going to blend and you're going to get a concept that maybe this is really one world and why the hell can't we learn to live together like decent people?”.

On an admittedly much smaller scale, the same learning can and should be applied to UX teams. When it comes time to take on a new project and define the vision, scope and strategy, bring in as many people as possible. The idea here isn’t to just tick an inclusivity box, but to deliver the best possible outcome.

Get input from stakeholders, designers, user researchers and developers. You certainly don’t have to take every suggestion, but a leader’s job is to assess every possible idea, question the what, why and how, and ultimately make a final decision. ‘Leader’ doesn’t necessarily have to mean ‘devil’s advocate’, either, but that’s another role you’ll also want to consider when taking suggestions from a large number of people.

Make time for your team

Anyone who’s ever stepped into a leadership role will understand the significant workload increase that comes along with it – not to mention the meetings that seemingly start to crop up like weeds. With such time pressures it can be easy to overlook things like regular one-on-ones, or at the very least making time for members to approach you with any issues.

Even with the associated pressures that come along with being a leader, stand-ups or other weekly meetings and one-on-ones should not be ignored.

Sit down with each member of your team individually to stay up to date on what they’re working on and to get a feel for their morale and motivation. What’s more, by simply setting some time aside to speak with someone individually, they’re more likely to speak about problems instead of bottling them away. Rotating through your team every fortnight will mean you have a clear understanding of where everyone is at.

Hosting larger stand-ups or weekly meetings, on the other hand, is useful in the way that large team meetings have always been useful. You can use the forum as a time for general status updates and to get new team members acclimated. If there’s one piece of advice we can add on here, it’s to have a clear agenda. Set the key things to cover in the meeting prior to everyone stepping into the room, otherwise you’re likely to see the meeting quickly get off track.

Keep a level head

You know the feeling. It’s Wednesday afternoon and one of the product teams has just dropped a significant amount of work on your team’s plate – a plate that’s already loaded up. While it can be tempting to join in with the bickering and complaining, it’s your job as the leader of your UX team to keep a level head and remain calm.

It’s basic psychology. The way you act and respond to different situations will have an impact of everyone around you – most importantly, your team. By keeping calm in every situation, your team will look to you for guidance in times of stress. There’s another benefit to keeping a level head: your own leaders are more likely to recognize you as a leader as well as someone who can handle difficult situations.

Two leadership development consultants ran a study of over 300,000 business leaders, and sorted the leadership skills they found most important for success into a numbered list. Unsurprisingly, an ability to motivate and inspire others was listed as the most important trait.

Be the voice for your team

While no user researcher or designer will doubt the value of UX research, it’s still an emerging industry. As a result, it can often be misunderstood. If you’re in charge of leading a UX team, it’s up to you to ensure that your team always has a seat at the table – you have to know when to speak up for yourself and your team.

If you a problem, you need to voice your concern. Of course, you need to be able to back up your arguments, but that’s the point of your role as a leader. Those new to leadership can find this aspect of the the job one of the hardest parts to master – it’s no surprise one of the key qualities in a great leader is an ability to speak up if they feel it’s the right thing to do.

Finally, you’ve got to assume the role of a buffer. This is another general leadership quality, and it’s similarly important. Take the flak from executives, upper management or the board of directors and defend your UX team, even if they’re not aware of it. If you need to take some information or feedback from these people and give it to your team, pay close attention to how you relay it to them. As an example, you want to be sure that a message about reducing customer churn is made relevant and actionable.

Master your own skill set

Stepping into a UX leadership position isn’t an excuse to stop developing yourself professionally. After all, it was those skills that got you there in the first place. Continue to focus on upskilling yourself, staying up to date on movements and trends in the industry and immersing yourself in the work your team carries out.

A leader with the skills to actually function as a member of their team is that much more capable – especially when another pair of hands can help to get a project over the line.

Wrap up

The field of user research continues to grow and mature, meaning the need for effective leaders is also increasing. This means there are near-limitless opportunities for those willing to step into UX leadership roles, provided they’re willing to put in the work and become capable leaders.

As we stated earlier, many of the skills that make a great leader also translate to UX leadership, and there’s really no shortage of resources available to provide guidance and support. In terms of UX specifically, here are a few of our favorite articles – from our blog and elsewhere:

• The essential qualities of a UX leader – Leaders in the UX industry share their thoughts as to what makes a good UX leader.
• Building and managing UX teams: A 360 degree guide – A slightly different approach to the one we’ve taken here, this article explains how to build and manage a UX team.
• Effective UX leaders – A discussion on the characteristics of effective UX leaders. It’s full of useful points.
• What does a truly inspirational design team leader look like? – A talk by Ashlea McKay at UX New Zealand 2015 in which she shares 8 qualities of inspirational design team leaders.

What an epic year. It’s certainly been one of significant change for us. We’ve welcomed a number of talented new people into our family (our team has grown by 64%), traveled around the world to visit and learn from our community, and refined and expanded our tools.Here’s what’s been going on at Optimal Workshop this year.Changing how we workOne of the biggest internal changes we made this year was to switch from primary working groups based on roles to smaller cross-functional teams called squads. Each of the squads has a set of objectives tied to the overarching goals of the company and they’re left to determine how to best meet these objectives. What’s more, squads are also self-managing, meaning they have no assigned manager. It’s different, and it’s working well for us. People are reporting higher levels of autonomy, enjoyment and focus.

Our Community Education squad hard at work.

We’ve also learned more about the importance of clarity this year, which I think is to be expected given our growth. I read a great article from Brené Brown, where she notes that “clear is kind, unclear is unkind”. Building a shared understanding is hard, and it’s well worth it.Happy and healthy

A highlight reel of some of the amazing smoothies and juices we’ve sampled this year.

What started as an initiative to cut back on our coffee consumption (and the subsequent afternoon slumps) has turned into a daily tradition here. We continued to make fresh fruit and vegetable juices daily this year, promoting healthy dietary decisions and giving everyone something to look forward to in the morning. I think we’ll just keep doing this as long as it seems like a good idea! If you’d like to hear my rationale for these kinds of crazy initiatives, here it is: If we expect people to come in and do their best work, we need to create an environment that’s conducive to people working at their best. Read the Stuff article about us here for more information and a video.Hitting the road (again)

Karl repping the Optimal Workshop team at the DesignOps Summit in New York City this year.

We’ve been to a lot of events this year, both at home and abroad. In fact, our team traveled a cumulative 205,349 miles in 2018 to connect with our community face to face. While that’s not quite the distance to the Moon, we were pretty close! I guess that’s the price of living in New Zealand, tucked away at the bottom of the globe (the bottom right corner on most maps).Moving house

Our new (still under construction) home.

In what’s possibly the biggest piece of news in this post, we’re moving into a new office late next year. Allen Street has been good to us, but it’s time to grow into a new space. Where are we moving, I hear you ask? Well, we’re actually taking over a piece of Wellington history and setting up shop in the converted Paramount picture theater. We’re really excited to share more with you – and even more excited to move in there ourselves!Our getaway to Riversdale

It’s always a good idea to bring a drone with you!

It’s no secret we like to do things a little different here – and the end of the year is no exception. Instead of hanging around in the office on a Friday afternoon or going out to a bar, we arose bright and early and clambered aboard a bus to head over the hill to the Wairarapa for a very traditional Kiwi beach day. Highlights included paintball, ping pong, some lovely team meals, freezing swims in the ocean and much celebration. It was certainly a great way to see the end of the working year in.Until next year

Anyway. That’s all for my end of year update. We really love what you do and we can’t wait to get right back into making this suite of tools the best, most cohesive home for your research that it can be. I’ve said it before, but we want to be the place where you and your team find signals in the noise and meaning in the mess. After all, we’re all about helping you create meaningful experiences.Keep your eyes peeled. We’ve got many more exciting changes on the way in 2019.As ever, we’re just getting started.Andrew MayfieldCEO

In Part 1 of this series we looked at how to interpret results from open and hybrid card sorts and now in Part 2, we’re going to talk about closed card sorts. In closed card sorts, participants are asked to sort the cards into predetermined categories and are not allowed to create any of their own. You might use this approach when you are constrained by specific category names or as a quick checkup before launching a new or newly redesigned website.In Part 1, we also discussed the two different - but complementary - types of analysis that are generally used together for interpreting card sort results: exploratory and statistical. Exploratory analysis is intuitive and creative while statistical analysis is all about the numbers. Check out Part 1 for a refresher or learn more about exploratory and statistical analysis in Donna Spencer’s book.

‍

‍

Getting started

‍

Closed card sort analysis is generally much quicker and easier than open and hybrid card sorts because there are no participant created category names to analyze - it’s really just about where the cards were placed. There are some similarities about how you might start to approach your analysis process but overall there’s a lot less information to take in and there isn’t much in the way of drilling down into the details like we did in Part 1.Just like with an open card sort, kick off your analysis process by taking an overall look at the results as a whole. Quickly cast your eye over each individual card sort and just take it all in. Look for common patterns in how the cards have been sorted. Does anything jump out as surprising? Are there similarities or differences between participant sorts?

‍

If you’re redesigning an existing information architecture (IA), how do your results compare to the current state? If this is a final check up before launching a live website, how do these results compare to what you learned during your previous research studies?If you ran your card sort using information architecture tool OptimalSort, head straight to the Overview and Participants Table presented in the results section of the tool. If you ran a moderated card sort using OptimalSort’s printed cards, you’ve probably been scanning them in after each completed session, but now is a good time to double check you got them all. And if you didn’t know about this handy feature of OptimalSort, it’s something to keep in mind for next time!

‍

The Participants Table shows a breakdown of your card sorting data by individual participant. Start by reviewing each individual card sort one by one by clicking on the arrow in the far left column next to the Participants numbers. From here you can easily flick back and forth between participants without needing to close that modal window. Don’t spend too much time on this — you’re just trying to get a general impression of how the cards were sorted into your predetermined categories. Keep an eye out for any card sorts that you might like to exclude from the results. For example participants who have lumped everything into one group and haven’t actually sorted the cards.

‍

Don’t worry- excluding or including participants isn’t permanent and can be toggled on or off at anytime.Once you’re happy with the individual card sorts that will and won’t be included in your results visualizations, it’s time to take a look at the Results Matrix in OptimalSort. The Results Matrix shows the number of times each card was sorted into each of your predetermined categories- the higher the number, the darker the shade of blue (see below).

‍

‍

This table enables you to quickly and easily get across how the cards were sorted and gauge the highest and lowest levels of agreement among your participants. This will tell you if you’re on the right track or highlight opportunities for further refinement of your categories.If we take a closer look (see below) we can see that in this example closed card sort conducted on the Dewey Decimal Classification system commonly used in libraries, The Interpretation of Dreams by Sigmund Freud was sorted into ‘Philosophy and psychology’ 38 times in study a completed by 51 participants.

‍

‍

In the real world, that is exactly where that content lives and this is useful to know because it shows that the current state is supporting user expectations around findability reasonably well. Note: this particular example study used image based cards instead of word label based cards so the description that appears in both the grey box and down the left hand side of the matrix is for reference purposes only and was hidden from the participants.Sometimes you may come across cards that are popular in multiple categories. In our example study, How to win friends and influence people by Dale Carnegie, is popular in two categories: ‘Philosophy & psychology’ and ‘Social sciences’ with 22 and 21 placements respectively. The remaining card placements are scattered across a further 5 categories although in much smaller numbers.

‍

‍

When this happens, it’s up to you to determine what your number thresholds are. If it’s a tie or really close like it is in this case, you might review the results against any previous research studies to see if anything has changed or if this is something that comes up often. It might be a new category that you’ve just introduced, it might be an issue that hasn’t been resolved yet or it might just be limited to this one study. If you’re really not sure, it’s a good idea to run some in-person card sorts as well so you can ask questions and gain clarification around why your participants felt a card belonged in a particular category. If you’ve already done that great! Time to review those notes and recordings!You may also find yourself in a situation where no category is any more popular than the others for a particular card. This means there’s not much agreement among your participants about where that card actually belongs. In our example closed card sort study, the World Book Encyclopedia was placed into 9 of 10 categories. While it was placed in ‘History & geography’ 18 times, that’s still only 35% of the total placements for that card- it’s hardly conclusive.

‍

Sometimes this happens when the card label or image is quite general and could logically belong in many of the categories. In this case, an encyclopedia could easily fit into any of those categories and I suspect this happened because people may not be aware that encyclopedias make up a very large part of the category on the far left of the above matrix: ‘Computer science, information & general works’. You may also see this happening when a card is ambiguous and people have to guess where it might belong. Again - if you haven’t already - if in doubt, run some in-person card sorts so you can ask questions and get to the bottom of it!After reviewing the Results Matrix in OptimalSort, visit the Popular Placements Matrix to see which cards were most popular for each of your categories based on how your participants sorted them (see below 2 images).

‍

‍

‍

The diagram shades the most popular placements for each category in blue making it very easy to spot what belongs where in the eyes of your participants. It’s useful for quickly identifying clusters and also highlights the categories that didn’t get a lot of card sorting love. In our example study (2 images above) we can see that ‘Technology’ wasn’t a popular card category choice potentially indicating ambiguity around that particular category name. As someone familiar with the Dewey Decimal Classification system I know that ‘Technology’ is a bit of a tricky one because it contains a wide variety of content that includes topics on medicine and food science - sometimes it will appear as ‘Technology & applied sciences’. These results appear to support the case for exploring that alternative further!

‍

‍

Where to from here?

‍

Now that we’ve looked at how to interpret your open, hybrid and closed card sorts, here are some next steps to help you turn those insights into action!Once you’ve analyzed your card sort results, it’s time to feed those insights into your design process and create your taxonomy which goes hand in hand with your information architecture. You can build your taxonomy out in Post-it notes before popping it into a spreadsheet for review. This is also a great time to identify any alternate labelling and placement options that came out of your card sorting process for further testing.From here, you might move into tree testing your new IA or you might run another card sort focussing on a specific area of your website. You can learn more about card sorting in general via our 101 guide.

‍

When interpreting card sort results, don’t forget to have fun! It’s easy to get overwhelmed and bogged down in the results but don’t lose sight of the magic that is uncovering user insights.I’m going to leave you with this quote from Donna Spencer that summarizes the essence of card sort analysis quite nicely:Remember that you are the one who is doing the thinking, not the technique... you are the one who puts it all together into a great solution. Follow your instincts, take some risks, and try new approaches. - Donna Spencer

‍

‍

Further reading

‍

• Card Sorting 101 – Learn about the differences between open, closed and hybrid card sorts, and how to run your own using OptimalSort.

‍

• Which comes first: card sorting or tree testing? – Tree testing and card sorting can give you insight into the way your users interact with your site, but which comes first?

‍

• How to pick cards for card sorting – What should you put on your cards to get the best results from a card sort? Here are some guidelines.

Cards have been created, sorted and sorted again. The participants are all finished and you’re left with a big pile of awesome data that will help you improve the user experience of your information architecture. Now what?Whether you’ve run an open, hybrid or closed card sort online using an information architecture tool or you’ve run an in person (moderated) card sort, it can be a bit daunting trying to figure out where to start the card sort analysis process.

‍

‍

About this guide

‍

This two-part guide will help you on your way! For Part 1, we’re going to look at how to interpret and analyze the results from open and hybrid card sorts.

• In open card sorts, participants sort cards into categories that make sense to them and they give each category a name of their own making.

• In hybrid card sorts, some of the categories have already been defined for participants to sort the cards into but they also have the ability to create their own.

‍

Open and hybrid card sorts are great for generating ideas for category names and labels and understanding not only how your users expect your content to be grouped but also what they expect those groups to be called.In both parts of this series, I’m going to be talking a lot about interpreting your results using Optimal Workshop’s online card sorting tool, OptimalSort, but most of what I’m going to share is also applicable if you’re analyzing your data using a spreadsheet or using another tool.

‍

‍

Understanding the two types of analysis: exploratory and statistical

‍

Similar to qualitative and quantitative methods, exploratory and statistical analysis in card sorting are two complementary approaches that work together to provide a detailed picture of your results.

‍

• Exploratory analysis is intuitive and creative. It’s all about going through the data and shaking it to see what ideas, patterns and insights fall out. This approach works best when you don’t have the numbers (smaller sample sizes) and when you need to dig into the details and understand the ‘why’ behind the statistics.

‍

• Statistical analysis is all about the numbers. Hard data that tells you exactly how many people expected X to be grouped with Y and more and is very useful when you’re dealing with large sample sizes and when identifying similarities and differences across different groups of people.

‍

Depending on your objectives - whether you are starting from scratch or redesigning an existing IA - you’ll generally need to use some combination of both of these approaches when analyzing card sort results. Learn more about exploratory and statistical analysis in Donna Spencer’s book.

‍

‍

Start with the big picture

‍

When analyzing card sort results, start by taking an overall look at the results as a whole. Quickly cast your eye over each individual card sort and just take it all in. Look for common patterns in how the cards have been sorted and the category names given by participants. Does anything jump out as surprising? Are there similarities or differences between participant sorts? If you’re redesigning an existing IA, how do your results compare to the current state?If you ran your card sort using OptimalSort, your first port of call will be the Overview and Participants Table presented in the results section of the tool.If you ran a moderated card sort using OptimalSort’s printed cards, now is a good time to double check you got them all. And if you didn’t know about this handy feature of OptimalSort, it’s something to keep in mind for next time!The Participants Table shows a breakdown of your card sorting data by individual participant. Start by reviewing each individual card sort one by one by clicking on the arrow in the far left column next to the Participants numbers.

‍

‍

From here you can easily flick back and forth between participants without needing to close that modal window. Don’t spend too much time on this — you’re just trying to get a general impression of what happened.Keep an eye out for any card sorts that you might like to exclude from the results. For example participants who have lumped everything into one group and haven’t actually sorted the cards. Don’t worry - excluding or including participants isn’t permanent and can be toggled on or off at anytime.If you have a good number of responses, then the Participant Centric Analysis (PCA) tab (below) can be a good place to head next. It’s great for doing a quick comparison of the different high-level approaches participants took when grouping the cards.The PCA tab provides the most insight when you have lots of results data (30+ completed card sorts) and at least one of the suggested IAs has a high level of agreement among your participants (50% or more agree with at least one IA).

‍

‍

The PCA tab compares data from individual participants and surfaces the top three ways the cards were sorted. It also gives you some suggestions based on participant responses around what these categories could be called but try not to get too bogged down in those - you’re still just trying to gain an overall feel for the results at this stage.Now is also a good time to take a super quick peek at the Categories tab as it will also help you spot patterns and identify data that you’d like to dive deeper into a bit later on!Another really useful visualization tool offered by OptimalSort that will help you build that early, high-level picture of your results is the Similarity Matrix. This diagram helps you spot data clusters, or groups of cards that have been more frequently paired together by your participants, by surfacing them along the edge and shading them in dark blue. It also shows the proportion of times specific card pairings occurred during your study and displays the exact number on hover (below).

‍

‍

In the above screenshot example we can see three very clear clusters along the edge: ‘Ankle Boots’ to ‘Slippers’ is one cluster, ‘Socks’ to ‘Stockings & Hold Ups’ is the next and then we have ‘Scarves’ to ‘Sunglasses’. These clusters make it easy to spot the that cards that participants felt belonged together and also provides hard data around how many times that happened.Next up are the dendrograms. Dendrograms are also great for gaining an overall sense of how similar (or different) your participants’ card sorts were to each other. Found under the Dendrogram tab in the results section of the tool, the two dendrograms are generated by different algorithms and which one you use depends largely on how many participants you have.

‍

If your study resulted in 30 or more completed card sorts, use the Actual Agreement Method (AAM) dendrogram and if your study had fewer than 30 completed card sorts, use the Best Merge Method (BMM) dendrogram.The AAM dendrogram (see below) shows only factual relationships between the cards and displays scores that precisely tell you that ‘X% of participants in this study agree with this exact grouping’.In the below example, the study shown had 34 completed card sorts and the AAM dendrogram shows that 77% of participants agreed that the cards highlighted in green belong together and a suggested name for that group is ‘Bling’. The tooltip surfaces one of the possible category names for this group and as demonstrated here it isn’t always the best or ‘recommended’ one. Take it with a grain of salt and be sure to thoroughly check the rest of your results before committing!

‍

‍

The BMM dendrogram (see below) is different to the AAM because it shows the percentage of participants that agree with parts of the grouping - it squeezes the data from smaller sample sizes and makes assumptions about larger clusters based on patterns in relationships between individual pairs.The AAM works best with larger sample sizes because it has more data to work with and doesn’t make assumptions while the BMM is more forgiving and seeks to fill in the gaps.The below screenshot was taken from an example study that had 7 completed card sorts and its BMM dendrogram shows that 50% of participants agreed that the cards highlighted in green down the left hand side belong to ‘Accessories, Bottoms, Tops’.

‍

‍

Drill down and cross-reference

‍

Once you’ve gained a high level impression of the results, it’s time to dig deeper and unearth some solid insights that you can share with your stakeholders and back up your design decisions.Explore your open and hybrid card sort data in more detail by taking a closer look at the Categories tab. Open up each category and cross-reference to see if people were thinking along the same lines.Multiple participants may have created the same category label, but what lies beneath could be a very different story. It’s important to be thorough here because the next step is to start standardizing or chunking individual participant categories together to help you make sense of your results.In open and hybrid sorts, participants will be able to label their categories themselves. This means that you may identify a few categories with very similar labels or perhaps spelling errors or different formats. You can standardize your categories by merging similar categories together to turn them into one.OptimalSort makes this really easy to do - you pretty much just tick the boxes alongside each category name and then hit the ‘Standardize’ button up the top (see below). Don’t worry if you make a mistake or want to include or exclude groupings; you can unstandardize any of your categories anytime.

‍

‍

Once you’ve standardized a few categories, you’ll notice that the Agreement number may change. It tells you how many participants agreed with that grouping. An agreement number of 1.0 is equal to 100% meaning everyone agrees with everything in your newly standardized category while 0.6 means that 60% of your participants agree.Another number to watch for here is the number of participants who sorted a particular card into a category which will appear in the frequency column in dark blue in the right-hand column of the middle section of the below image.

‍

‍

‍

From the above screenshot we can see that in this study, 18 of the 26 participant categories selected agree that ‘Cat Eye Sunglasses’ belongs under ‘Accessories’.Once you’ve standardized a few more categories you can head over to the Standardization Grid tab to review your data in more detail. In the below image we can see that 18 participants in this study felt that ‘Backpacks’ belong in a category named ‘Bags’ while 5 grouped them under ‘Accessories’. Probably safe to say the backpacks should join the other bags in this case.

‍

‍

So that’s a quick overview of how to interpret the results from your open or hybrid card sorts.Here's a link to Part 2 of this series where we talk about interpreting results from closed card sorts as well as next steps for applying these juicy insights to your IA design process.

‍

‍

Further reading

• Card Sorting 101 – Learn about the differences between open, closed and hybrid card sorts, and how to run your own using OptimalSort.
• Which comes first: card sorting or tree testing? – Tree testing and card sorting can give you insight into the way your users interact with your site, but which comes first?
• How to pick cards for card sorting – What should you put on your cards to get the best results from a card sort? Here are some guidelines.

Reframer is a qualitative research tool that was built to help teams capture and make sense of their research findings quickly and easily. For those of you who have been a long-standing Optimal Workshop customer, you may know that Reframer has been in beta for some time. In fact, it has been in beta for 2 whole years. Truth was that, while we’ve cheerfully responded to your feedback with a number of cool features and improvements, we had some grand plans up our sleeve. So, we took everything we learned and went back to drawing board with the goal to provide the best dang experience we can.We’ll soon be ready to launch Reframer out of beta and let it take its place proudly as a full time member of our suite of user research tools. However, in the spirit of continuous improvement, we want to give you all a chance to use it and give us feedback on the new experience so far.

First-time Reframer user?

Awesome! You’ll get to experience the newer version of Reframer with a fresh set of eyes. To enable Reframer, log in to your Optimal Workshop account. On your dashboard you’ll see a button to join the Reframer beta on your screen at right.

Used Reframer before?

Any new studies you create will automatically use the slick new version. Not quite ready to learn the new awesome? No worries you can toggle back and forth between the old version and the new in the top right corner of your screen.To learn about Reframer’s new look and features, watch the video or read the transcript below to hear more about these changes and why we made them.

When is Reframer actually coming out of beta?

This year. Stay tuned.

Video transcript:  

We’re this close to having our qualitative research tool, Reframer, all set to release from beta.But we just couldn’t wait to share some of the changes we’ve got lined up. So, we’ve gone ahead and launched a fresh version of Reframer to give you a taste of what’s to come.These latest updates include a more streamlined workflow and a cleaner user interface, as well as laying the foundations for some exciting features in the coming months.So let’s take a look at the revamped Reframer.We’ve updated the study screen to help you get started and keep track of where your research is at.

• You now have an area for study objectives to keep your team on the same page

• And an area for reference links, to give you quick access to prototypes and other relevant documents

• Session information is now shown here too, so you can get an overview of all your participants at a glance

• And we’ve created a home for your tags with more guidance around how to use them, like example tags and groups to help you get started.

What’s the most important thing when observing a research session? Collecting insights of course! So we’ve simplified the capture experience to let you focus on taking great notes.

• You can choose to reveal your tags, so they’re at the ready, or hide them so you can save your tagging till later

• We’ve created a whole range of keyboard shortcuts to speed up adding and formatting observations

• The import experience is now more streamlined, so it’s easier to bring observations from other sources into Reframer

• And, with some huge improvements behind the scenes, Reframer is even faster, giving you a more seamless note taking experience.

Now for something totally new — introducing review mode. Here you can see your own observations, as well as those made by anyone else in your team. This makes it easy to tidy up and edit your data after your session is complete. You can filter, search and tag observations, so you’ll be ready to make sense of everything when you move to analysis.We’ve added more guidance throughout Reframer, so you’ll have the confidence you’re on the right track. New users will be up and running in no time with improved help and easy access to resources.You might notice a few changes to our UI as well, but it’s not just about looks.

• We’ve changed the font to make it easier to read on any screen size

• Our updated button colours to provide better contrast and hierarchy

• And we’ve switched our icons from fonts to images to make them more accessible to more users.

And that’s where we’re at.We've got a lot more exciting features to come, so why not jump in, give the new Reframer a try and tell us what you think!Send us your feedback and ideas at research@optimalworkshop.com and keep an eye out for more changes coming soon. Catch you later!