Ready for take-off: Best practices for creating and launching remote user research studies

Card sorting is an invaluable tool for understanding how people organize information in their minds, making websites more intuitive and content easier to navigate. It’s a useful method outside of information architecture and UX research, too. It can be a useful prioritization technique, or used in a more traditional sense. For example, it’s handy in psychology, sociology or anthropology to inform research and deepen our understanding of how people conceptualize information.

‍

The introduction of remote card sorting has provided many advantages, making it easier than ever to conduct your own research. Tools such as our very own OptimalSort allow you to quickly and easily gather findings from a large number of participants from all around the world. Not having to organize moderated, face-to-face sessions gives researchers more time to focus on their work, and easier access to larger data sets.

‍

One of the main disadvantages of remote card sorting is that it eliminates the opportunity to dive deeper into the choices made by your participants. Human conversation is a great thing, and when conducting a remote card sort with users who could potentially be on the other side of the world, opportunities for our participants to provide direct feedback and voice their opinions are severely limited.Your survey design may not be perfect.

‍

The labels you provide your participants may be incorrect, confusing or redundant. Your users may have their own ideas of how you could improve your products or services beyond what you are trying to capture in your card sort. People may be more willing to provide their feedback than you realize, and limiting their insights to a simple card sort may not capture all that they have to offer.So, how can you run an unmoderated, remote card sort, but do your best to mitigate this potential loss of insight?

‍

A quick look into the data

‍

In an effort to evaluate the usefulness of the existing “Leave a comment” feature in OptimalSort, I recently asked our development team to pull out some data.You might be asking “There’s a comment box in OptimalSort?”If you’ve never noticed this feature, I can’t exactly blame you. It’s relatively hidden away as an unassuming hyperlink in the top right corner of your card sort.

‍

‍

‍

‍

Comments left by your participants can be viewed in the “Participants” tab in your results section, and are indicated by a grey speech bubble.

‍

‍

‍

The history of the button is unknown even to long-time Optimal Workshop team members. The purpose of the button is also unspecified. “Why would anyone leave a comment while participating in a card sort?”, I found myself wondering.As it turns out, 133,303 comments have been left by participants. This means 133,303 insights, opinions, critiques or frustrations. Additionally, these numbers only represent the participants who noticed the feature in the first place. Considering the current button can easily be missed when focusing on the task at hand, I can’t help but wonder how this number might change if we drew more attention to the feature.

‍

Breaking down the comments

‍

To avoid having to manually analyze and code 133,303 open text fields, I decided to only spend enough time to decipher any obvious patterns. Luckily for me, this didn’t take very long. After looking at only a hundred or so random entries, four distinct types of comments started to emerge.

‍

1. This card/group doesn’t make sense.Comments related to cards and groups dominate. This is a great thing, as it means that the majority of comments made by participants relate specifically to the task they are completing. For closed and hybrid sorts, comments frequently relate to the predefined categories available, and since the participants most likely to leave a comment are those experiencing issues, the majority of the feedback relates to issues with category names themselves. Many comments are related to card labels and offer suggestions for improving naming conventions, while many others draw attention to some terms being confusing, unclear or jargony. Comments on task length can also be found, along with reasons for why certain cards may be left ungrouped, e.g., “I’ve left behind items I think the site could do without”.
2. Your organization is awesome for doing this/you’re doing it all wrong. A substantial number of participants used the comment box as an opportunity to voice their general feedback on the organization or company running the study. Some of the more positive comments include an appreciation for seeing private companies or public sector organizations conducting research with real users in an effort to improve their services. It’s also nice to see many comments related to general enjoyment in completing the task.On the other hand, some participants used the comment box as an opportunity to comment on what other areas of their services should be improved, or what features they would like to see implemented that may otherwise be missed in a card sort, e.g., “Increased, accurate search functionality is imperative in a new system”.
3. This isn’t working for me. Taking a closer look at some of the comments reveals some useful feedback for us at Optimal Workshop, too. Some of the comments relate specifically to UI and usability issues. The majority of these issues are things we are already working to improve or have dealt with. However, for researchers, comments that relate to challenges in using the tool or completing the survey itself may help explain some instances of data variability.
4. #YOLO, hello, ;) And of course, the unrelated. As you may expect, when you provide people with the opportunity to leave a comment online, you can expect just about anything in return.

‍

How to make the most of your user insights in OptimalSort

‍

If you’re running a card sort, chances are you already place a lot of value in the voice of your users. To ensure you capture any additional insights, it’s best to ensure your participants are aware of the opportunity to do so. Here are two ways you may like to ensure your participants have a space to voice their feedback:

‍

‍Adding more context to the “Leave a comment” feature

‍

One way to encourage your participants to leave comments is to promote the use of the this feature in your card sort instructions. OptimalSort gives you flexibility to customize your instructions every time you run a survey. By making your participants aware of the feature, or offering ideas around what kinds of comments you may be looking for, you not only make them more likely to use the feature, but also open yourself up to a whole range of additional feedback. An advantage of using this feature is that comments can be added in real time during a card sort, so any remarks can be made as soon as they arise.

‍

‍Making use of post-survey questions

‍

‍Adding targeted post-survey questions is the best way to ensure your participants are able to voice any thoughts or concerns that emerged during the activity. Here, you can ask specific questions that touch upon different aspects of your card sort, such as length, labels, categories or any other comments your participants may have. This can not only help you generate useful insights but also inform the design of your surveys in the future.

‍

Make your remote card sorts more human

‍

Card sorts are exploratory by nature. Avoid forcing your participants into choices that may not accurately reflect their thinking by giving them the space to voice their opinions. Providing opportunities to capture feedback opens up the conversation between you and your users, and can lead to surprising insights from unexpected places.

‍

Further reading

‍

• "Moderated card sorts vs online card sorts — why you need both" - An article on our blog that explains some of the differences between these two methods and how to use both for great results.
• "The hidden benefits of remote research" - Kathleen Asjes writes for UX Booth and explains when to use remote research and some of its many benefits.
• "Remote research: real users, real time, real research" - A book by Nate Bolt and Tony Tulathimutte on how to design and conduct remote research studies.

Understanding your customers is central to any organization which wants to deliver an outstanding experience. But how do you understand your customers better? Tailoring their experience with your products and your organization to suit them should include a customer journey map.

‍

It doesn’t mean your organization needs a brightly colored, fully designed infographic that outlines each and every action your user takes within your product. It does mean an effective customer journey map that promotes empathy and provides a clear vision for improving customer interactions. There are no rules around what that visualization looks like and it is up to your team to create one that makes the most of your customer’s journey.

‍

What is customer journey mapping? 🤔

‍

Customer journey mapping (sometimes referred to as a user experience map) is a technique that allows you to visualize your customer’s key touchpoints, sentiments, pain points, and actions. Plotted in sequential order. It’s a map of a customer’s experience with your brand or product, from awareness to purchase and beyond. 

‍

Customer journey mapping helps you look beyond key touchpoints and encourages empathy with your customers. To understand who they are, even a persona to give them a life and demographic. Helping designers and key stakeholders understand where they are coming from. And where you can address their needs, avoid their pain points and encourage them to engage with your product. And even identify opportunities for innovation and improvement across the board.

‍

Why you need a customer journey map 💥

‍

What’s better than a customer that feels seen and understood once? A customer that feels the organization or product really understands their needs (and responds to their frustrations). Like all successful, long-term relationships, keeping customers returning is built on empathy and a solid grasp of their needs and frustrations. 

‍

When you want to get to know your customer, like really get to know them, it’s essential to map their customer journey. Creating a shared understanding of what your customers think, feel, and struggle with as they interact with your organization. Spending the time to establish a customer journey map can help align around identifying known problems, identifying new user pain points, and removing roadblocks for your customers, ensuring their success.

‍

Spending the time to get inside the mind and journeys of your customers through mapping helps your team to:

‍

• Create a visual guide of the end-to-end customer experience
• Get an understanding of multiple customer pathways and unravel complex user experiences
• Create target personas and allow insights to solve problems more effectively
• Increase your organization’s empathy for your current and future customers
• Identify potential pain points and roadblocks for your customers
• Breakdown silos within your organization and improve alignment across teams
• With a clearer understanding of your customers, better insights to achieve stakeholder buy-in

‍

How to create a customer journey map 🛤️

‍

Here are 8 key steps to get the most out of your customer journey map process:

‍

1. Bring key stakeholders together for an initial brainstorming session
2. Identify potential user personas - demographics, pain points, interests, etc
3. Create an empathy map - get a real feel for your customer, who they are, and what they want
4. Flesh out your ideas with user research. Get under your user persona skin with focus groups, interviews, and surveys
5. Identify possible customer touchpoints
6. Choose the information you want to highlight - not everything will be relevant
7. Decide on the best customer journey map tool to answer your questions
8. Start building your map

‍

Customer journey mapping examples 🗺️

‍

Each and every customer’s journey is different. This, of course, means that there is no single best customer journey map example or template. Instead, the best customer journey map for any given situation will depend not only on your customers but also on your product, your team, and the goals you’re hoping to achieve by creating the map in the first place.

‍

We’ve found a few examples of customer journey maps to help inspire your thinking:

‍

Current state customer journey maps help you to visualize a user’s experience as it is right now. These are fact-based journey maps - to create an accurate, current state journey map. A good dose of user research data around your actual customers and interactions will help shape this.

‍

‍

Future state customer journey maps focus on what the customer journey can and should look like in the future. Although UX data is certainly an important aspect of understanding customers, future state journey maps involve a fair amount of creative speculation and interpretation. These customer journey maps focus on customer hopes and wants (future feelings), in addition to experiences and reactions. They can be a little objective and should be developed in balance with both positive and negative interactions.

‍

Day in the life customer journey maps help you visualize your customer’s entire daily routine. Interactions with family, their commute, work meetings, afternoon coffee, etc. Regardless of whether or not the activities are related to your company. This type of journey map should be organized chronologically to give key insights into how and where customers are. What are the distractions, and where could they interact with your brand or product? How can key pain points be eased?

‍

Service blueprints are a useful counterpart to a classic customer journey map. Whereas a customer journey map focuses on the thoughts, needs, and actions of the customer, a service blueprint reflects the perspective of the organization and its employees. What needs to happen behind the scenes to ensure the customer’s experience is the very best it can be.

‍

Circular customer journey maps may be useful to visualize the customer journey as a circle or loop. Recognizing that some customers are recurring and non-transactional. Particularly through subscription-based relationships.

‍

An empathy map is used to create a shared understanding of customers around their wants, needs, thoughts, and actions. This can be a great starting point to getting under your customer’s skin.

‍

‍

Wrap Up 🌯

‍

There is no one size fits all customer journey map. Each customer is unique, each organization is different. Through creating customer journeys, personas and visualizing their key touchpoints, pain points, and understanding who they are, empathy throughout the organization can be generated. With this as a tool to bring key stakeholders on board and to pinpoint where products (and services) can be improved to keep customers or even bring new ones on board, the user experience can be better and more effective.

In 2009, Bob Bailey and Cari Wolfson published published findings that changed how we approach first click testing and usability testing. They analyzed 12 scenario-based user tests and found that if someone gets their first click right, they're about twice as likely to complete their task successfully. This finding was so compelling that we built First Click Testing (formerly Chalkmark) specifically to help teams test this.  But we'd never actually validated their research using our own data, until now.

‍

Turns out, we're sitting on one of the world's largest databases of tree testing results. So we analyzed millions of task responses to see if the "first click predicts success" hypothesis holds up.

‍

It does. Convincingly.

‍

Users who get their first click correct are nearly three times more likely to complete their task successfully (70% vs 24% success rate).

‍

Here's how we validated the original study, what our data shows, and why first clicks matter more than you might think.

‍

‍

‍

Original first click testing study: 87% task success rate

‍

‍

Bob and Cari analyzed data from twelve usability studies on websites and products with varying amounts and types of content, a range of subject matter complexity, and distinct user interfaces. They found that people were about twice as likely to complete a task successfully if they got their first click right, than if they got it wrong:

‍

If the first click was correct, the chances of getting the entire scenario correct was 87% if the first click was incorrect, the chances of eventually getting the scenario correct was only 46%.

‍

‍

‍

Our Tree Testing data: First clicks predict 70% task success rate

‍

‍

We analyzed millions of tree testing responses in our database. We've found that people who get the first click correct are almost three times as likely to complete a task successfully:

‍

If the first click was correct, the chances of getting the entire scenario correct was 70% if the first click was incorrect, the chances of eventually getting the scenario correct was 24%

‍

To give you another perspective on the same data, here's the inverse:

‍

If the first click was correct, the chances of getting the entire scenario incorrect was 30% if the first click was incorrect, the chances of getting the whole scenario incorrect was 76%

‍

‍

‍

How Tree Testing measures first click success and task completion

‍

‍

Bob and Cari proved the usefulness of the methodology by linking two key metrics in scenario-based usability studies: first clicks and task success. First Click Testing doesn't measure task success — it's up to the researcher to determine as they're setting up the study what constitutes 'success', and then to interpret the results accordingly. Tree Testing (formerly Treejack) does measure task success — and first clicks.

‍

In a tree test, participants are asked to complete a task by clicking though a text-only version of a website hierarchy, and then clicking 'I'd find it here' when they've chosen an answer. Each task in a tree test has a pre-determined correct answer — as was the case in Bob and Cari's usability studies — and every click is recorded, so we can see participant paths in detail.

‍

Thus, every single time a person completes an individual tree testing task, we record both their first click and whether they are successful or not. When we came to test the 'correct first click leads to task success' hypothesis, we could therefore mine data from millions of task.

‍

To illustrate this, have a look at the results for one task. The overall Task result, you see a score for success and directness, and a breakdown of whether each Success, Fail, or Skip was direct (they went straight to an answer), or indirect (they went back up the tree before they selected an answer):

‍

‍

‍

‍

In the pie tree for the same task, you can look in more detail at how many people went the wrong way from a label (each label representing one page of your website):

‍

‍

‍

‍

In the First Click tab, you get a percentage breakdown of which label people clicked first to complete the task:

‍

‍

‍

‍

And in the Paths tab, you can view individual participant paths in detail (including first clicks), and can filter the table by direct and indirect success, fails, and skips (this table is only displaying direct success and direct fail paths):

‍

‍

‍

‍

‍

How to run first click tests: Best practices for usability testing

‍

‍

First click analysis is one of the most predictive metrics in usability testing. Whether you're testing wireframes, landing pages, or information architecture, measuring first click success gives you early insight into whether your design will work.

‍

This analysis reinforces something we already knew: first clicks matter. It is worth your time to get that first impression right. You have plenty of options for measuring the link between first clicks and task success in your scenario-based usability tests. From simply noting where your participants go during observations, to gathering quantitative first click data via online tools, you'll win either way. And if you want quantitative first click data, Optimal has you covered. First Click Testing works for wireframes and landing pages, while Tree Testing validates your information architecture.

‍

To finish, here are a few invaluable insights from other researchers on getting the most from first click testing:

‍

• Jeff Sauro details a useful approach to running a first click test, and shares the findings from a test he ran on 13 people.
• An article on Neoinsight describes three common usability problems that first click testing can solve.
• Gianna LaPin describes a first click test she ran on Netflix, VUDU, and Hulu Plus.

‍

‍

About this study

‍

This analysis was conducted in 2015 using millions of task responses from Optimal’s First Click and Tree Testing tools. While the dataset predates recent UI trends, the underlying behavioral principle, that a correct first click strongly predicts task success, remains consistent with modern usability research.