August 8, 2022

—

4 min

Usability Testing Guide: What It Is, How to Run It, and When to Use Each Method

Knowing and understanding why and how your users use your product can be invaluable for getting to the nitty gritty of usability. Where they get stuck and where they fly through. Delving deep with probing questions into motivation or skimming over looking for issues can equally be informative.

‍

Usability testing can be done in several ways, each way has its benefits. Put super simply, usability testing literally is testing how useable your product is for your users. If your product isn't useable users will not stick around or very often complete their task, let alone come back for more.

‍

What is usability testing?

‍

Usability testing is a research method used to evaluate how easy something is to use by testing it with representative users.

‍

These tests typically involve observing a participant as they work through a series of tasks involving the product being tested. Having conducted several usability tests, you can analyze your observations to identify the most common issues.

‍

We go into the three main methods of usability testing:

Moderated and unmoderated
Remote or in person
Explorative, assessment or comparative

‍

1. Moderated or unmoderated usability testing

‍

Moderated usability testing

Moderated usability testing is done in-person or remotely by a researcher who introduces the test to participants, answers their queries, and asks follow-up questions. Often these tests are done in real time with participants and can involve other research stakeholders. Moderated testing usually produces more in-depth results thanks to the direct interaction between researchers and test participants. However, this can be expensive to organize and run.

‍

Top tip: Use moderated testing to investigate the reasoning behind user behavior.

‍

Unmoderated usability testing

‍
Unmoderated usability testing is done without direct supervision; likely participants are in their own homes and/or using their own devices to browse the website that is being tested. And often at their own pace. The cost of unmoderated testing is lower, though participant answers can remain superficial and making follow-up questions can be difficult.

‍

Top tip: Use unmoderated testing to test a very specific question or observe and measure behavior patterns.

‍

2. Research or in-person usability testing

‍

Remote usability testing

‍
‍Remote usability testing is done over the internet or by phone. Allowing the participants to have the time and space to work in their own environment and at their own pace. This however doesn’t give the researcher much in the way of contextual data because you’re unable to ask questions around intention or probe deeper if the participant makes a particular decision. Remote testing doesn’t go as deep into a participant’s reasoning, but it allows you to test large numbers of people in different geographical areas using fewer resources.

‍

Top tip: Use remote testing when a large group of participants are needed and the questions asked can be direct and unambiguous.

‍

In-person usability testing

‍
‍In-person usability testing, as the name suggests, is done in the presence of a researcher. In-person testing does provide contextual data as researchers can observe and analyze body language and facial expressions. You’re also often able to converse with participants and find out more about why they do something. However, in-person testing can be expensive and time-consuming: you have to find a suitable space, block out a specific date, and recruit (and often pay) participants.

‍

Top tip: In-person testing gives researchers more time and insight into motivation for decisions.

‍

3. Explorative, Assessment or comparative testing

‍

These three usability testing methods generate different types of information:

‍

Explorative testing

‍
‍Explorative testing is open-ended. Participants are asked to brainstorm, give opinions, and express emotional impressions about ideas and concepts. The information is typically collected in the early stages of product development and helps researchers pinpoint gaps in the market, identify potential new features, and workshop new ideas.

‍

Assessment research

‍
‍Assessment research is used to test a user's satisfaction with a product and how well they are able to use it. It's used to evaluate general functionality.

‍

Comparative research

‍
‍Comparative research methods involve asking users to choose which of two solutions they prefer, and they may be used to compare a product with its competitors.

‍

Top tip: Depending on what research is being done, and how much qualitative or quantitative data is wanted.

‍

Which method is right for you?

‍

Whether the testing is done in-person, remote, moderated or unmoderated will depend on your purpose, what you want out of the testing, and to some extent your budget.

‍

Depending on what you are testing, each of the usability testing methods we explored here can offer an answer. If you are at the development stage of a product it can be useful to conduct a usability test on the entire product. Checking the intuitive usability of your website, to ensure users can make the best decisions, quickly. Or adding, changing or upgrading a product can also be the moment to check on a specific question around usability. Planning and understanding your objectives are key to selecting the right usability testing option for your project.

‍

Let's take a look at a couple of examples of usability testing.

‍

1. Lab based, in-person moderated testing - mid-life website

‍

Imagine you have a website that sells sports equipment. Over time your site has become cluttered and disorganized, much like a bricks and mortar store may. You’ve noticed a drop in sales in certain areas. How do you find out what is going wrong or where users are getting lost? Having an in-person, lab (or other controlled environment), moderated usability test with users you can set tasks, watch (and record) what they do.

‍

The researcher can literally be standing or sitting next to the participant throughout, recording contextual information such as how they interacted with the mouse, laptop or even the seat. Watching for cues as to the comfort of the participant and asking questions about why they make decisions can provide richer insights. Maybe they wanted purple yoga pants, but couldn’t find the ‘yoga’ section which was listed under gym rather than a clothing section.

‍

Meaning you can look at how your stock is organised, or even investigate undertaking a card sort. This provides robust and fully rounded feedback on users behaviours, expectations and experiences. Providing data that can directly be turned into actionable directives when redeveloping the website.

‍

2. Remote, moderated assessment testing - app product development

‍

You are looking at launching an app for parents to access for information and updates for the school. It’s still in development stage and at this point you want to know how easy the app is to use. Setting some very specific set tasks for participants to complete the app can be sent to them and they can be left to complete (or not). Providing feedback and comments around the usability.

‍

The next step may be to use first click testing to see how and where the interface is clicked and where participants may be spending time, or becoming lost. Whilst the feedback and data gathered from this testing can be light, it will be very direct to the questions asked. And will provide data to back up (or possibly not) what assumptions were made.

‍

3. Moderated, In-person, explorative testing - new product development

‍

You’re right at the start of the development process. The idea is new and fresh and the basics are being considered. What better way to get an understanding of what your users’ truly want than an explorative study.

‍

Open-ended questions with participants in a one-on-one environment (or possibly in groups) can provide rich data and insights for the development team. Imagine you have an exciting new promotional app that you are developing for a client. There are similar apps on the market but none as exciting as what your team has dreamt up. By putting it (and possibly the competitors) to participants they can give direct feedback on what they like, love and loathe.

‍

They can also help brainstorm ideas or better ways to make the app work, or improve the interface. All of this done, before there is money sunk in development.

‍

Usability testing summary: When to use each method (and why)

‍

Key objectives will dictate which usability testing method will deliver the answers to your questions.

‍

Whether it’s in-person, remote, moderated or comparative with a bit of planning you can gather data around your users very real experience of your product. Identify issues, successes and failures. Addressing your user experience with real data, and knowledge can but lead to a more intuitive product.

View all blog articles

Learn more

1 min read

A comprehensive look at usability testing

Usability testing has an important role in UX and if you’re new to it, this article gives you a solid introduction to it with practical tips, a checklist for success and a look at our remote testing tool, Treejack.

‍

Concepts of usability testing

‍

Usability testing is the process of evaluating a product or service with users prior to implementation. The goal of usability testing is to identify any usability issues before the product or service is released into the world for use. Usability testing is a research activity that results in both quantitative and qualitative insights and can be used to gauge user satisfaction.A typical usability testing session is moderated and involves a participant, a facilitator and an observer. The facilitator leads the session and the observer takes notes while the participant completes the task based scenario.

‍

While this is common, usability testing is scalable and the possible approaches are endless giving you the flexibility to work with the resources you have available—sometimes one person performs the role of facilitator and observer!Location also varies for usability testing. For example, you might conduct your testing in a lab environment or you might talk to users in a specific environment. It also worth noting that not all usability testing sessions are moderated—more about this later.Usability testing usually occurs multiple times during the design process and can be conducted anytime you have a design you would like to test.

‍

‍User research activities like a focus group for example, are conducted early in the design process to explore and gain understanding before ideas are generated. Usability testing is about testing those ideas as early and as often as possible. From a fully functioning digital prototype to a simple hand drawn wireframe on paper, nothing is too unrefined or too rough to be tested.

‍

Developing a usability test plan

‍

Before you start a round of usability testing, you need to develop a usability test plan. The usability test plan will keep you organised and is an opportunity to define roles and set clear expectations upfront. The first step in developing this is to hold a meeting with your team and stakeholders to discuss what you are going to do and how you plan to achieve it. Following this meeting, a document outlining the usability test plan as it was discussed is created and shared with the group for review. Any changes suggested by the group are then added to the final document for approval from the relevant stakeholders.

‍

What to include in your usability test plan:

‍

The goal, scope and intent of the usability testing
Constraints impacting upon testing
Details on what will be tested eg wireframes
Schedule and testing location
Associated costs eg participant recruitment
Facilitator and observer details for each session
Session details
Participant recruitment approach
Equipment
Details of any documentation to be produced eg a report

‍

Usability testing questions

‍

Once you have developed your test plan, you need to create a list of questions and task based scenarios for the testing session. These form the structure for your testing and provide a framework of consistency across all testing sessions in the study.The questions serve as a warm up to ease the participant into the session and can also provide insights on the user that you may not have had before. These questions can be a combination of　open and closed questions and are especially useful if you are also developing personas for example. Some examples of what you might ask include:

Tell me about a recent experience you had with this product/service
Do you currently use this product/service?
Do you own a tablet device?

‍

The purpose of the task based scenarios is to simulate a real life experience as closely as possible. They provide a contextual setting for the participant to frame their approach and they need to be realistic—your participant needs an actionable starting point to work from. A good starting point for task based scenario development would be to look at a use case.It is also important that you avoid using language that provides clues to the solution or leads your participant as this can produce inaccurate results. An example of a task based scenario would be:You’re planning a Christmas vacation to New Zealand for your family of two adults and 4 children. Find the lowest priced airfares for your trip.

‍

Usability testing software: Tree testing

‍

Treejack is a remote information architecture (IA) validation tool that shows you exactly where users are getting lost in your content. Knowing this will enable you to design a structure for your website that makes sense to users before moving on to the user interface (UI) design.Treejack works like a card sort in reverse. Imagine you have just completed a card sort with your users to determine your IA and you are now working backwards to test that thinking against real world task based scenarios. Treejack does this using a text-based version of your IA that is free from distracting visual aids such as navigation and colour allowing you to determine if your structure is usable from the ground up. A Treejack study is structured around task based scenarios and comes with the option to include pre and post study questionnaires.

‍

treejack task image — Usability testing with Treejack

‍

As a remote testing tool, Treejack is unmoderated and provides the opportunity to reach a much larger audience because all you have to do is share a link to the study with your participants to gain insights. You also have the option of handing the task of targeted participant recruitment over to Optimal Workshop.Once launched and shared with participants, Treejack takes care of itself by recording the results as they come in giving you the freedom to multitask while you wait for the testing to finish.

‍

The results produced by Treejack are not only detailed and comprehensive but are also quite beautiful. The story of your participants’ journey through your testing activity is told through pietrees. A pietree is a detailed pathway map that shows where your participants went at each fork in the road and their destinations. They allow you to pinpoint exactly where the issues lie and are a powerful way to communicate the results to your team and stakeholders.

‍

bananacom pie tree — Treejack presents your results using pietrees

‍

Treejack also provides insights into where your participants landed their first click and records detailed information on pathways followed by each individual participant.

‍

bananacom paths — Treejack records full details of the paths followed by every participant

‍

Usability testing checklist

‍

The following checklist will help ensure your usability testing process runs smoothly:

‍

Meet with team and stakeholders
Determine goals, scope and intent of usability testing
Decide how many sessions will be conducted
Create usability testing schedule
Select facilitators and observers for each session if applicable
Develop and complete a usability test plan
Determine test questions and scenarios
Recruit participants for testing
Gather equipment required for testing if applicable
Book testing location if applicable
Keep a list of useful contact details close by in case you need to contact anyone during testing
Complete a dry run of a testing session with a team member to ensure everything works before actual testing begins
Organise debrief meetings with observers after each testing session
Set aside time to analyse the findings
Document and present findings to team and relevant stakeholders

‍

Make research a priority for your organization

‍

Before you can do more research, it needs to be prioritized across your entire organization — not just within your design team. To that end, you should:

‍

Know what you’re trying to achieve. By defining specific goals, you can share a clear message with the broader organization about what you’re after, how you can achieve those goals, and how you will measure success. At Viget, we shared our research goals with everyone at the company. In addition, we talked to the business development and project management teams in more depth about specific ways that they could help us achieve our goals, since they have the greatest impact on our ability to do more research.

Track your progress. Once you’ve made research a priority, make sure to review your goals on an ongoing basis to ensure that you’re making progress and share your findings with the organization. Six months after the research group at Viget started working on our goals, we held a retrospective to figure out what was working — and what wasn’t.
Adjust your approach as needed. You won’t achieve your goals overnight. As you put different tactics into action, adjust your approach if something isn’t helping you achieve your goals. Be willing to experiment and don’t feel bad if a specific tactic isn’t successful.

‍

Educate your colleagues and clients

‍

If you want people within your organization to get excited about doing more research, they need to understand what research means. To educate your colleagues and clients, you should:

Explain the fundamentals of research. If someone has not conducted research before, they may not be familiar or feel comfortable with the vernacular. Provide an overview of the fundamental terminology to establish a basic level of understanding. In a blog post, Speaking the Same Language About Research, we outline how we established a common vocabulary at Viget.
Help others understand the landscape of research methods. As designers, we feel comfortable talking about different methodologies and forget that that information will be new to many people. Look for opportunities to increase understanding by sharing your knowledge. At Viget, we make this happen in several ways. Internally, we give presentations to the company, organize group viewing sessions for webinars about user research, and lead focused workshops to help people put new skills into practice. Externally, we talk about our services and share knowledge through our blog posts. We are even hosting a webinar about conducting user interviews in November and we'd love for you to join us.
Incorporate others into the research process. Don't just tell people what research is and why it's important — show them. Look for opportunities to bring more people into the research process. Invite people to observe sessions so they can experience research firsthand or have them take on the role of the notetaker. Another simple way to make people feel involved is to share findings on an ongoing basis rather than providing a report at the end of the process.

‍

Broaden your perspective while refining your skill set

‍

Our commitment to testing assumptions led us to challenge ourselves to do research on every project. While we're dogmatic about this goal, we're decidedly un-dogmatic about the form our research takes from one project to another. To pursue this goal, we seek to:

Expand our understanding. To instill a culture of research at Viget, we've found it necessary to question our assumptions about what research looks like. Books like Erika Hall’s Just Enough Research teach us the range of possible approaches for getting useful user input at any stage of a project, and at any scale. Reflect on any methodological biases that have become well-worn paths in your approach to research. Maybe your organization is meticulous about metrics and quantitative data, and could benefit from a series of qualitative studies. Maybe you have plenty of anecdotal and qualitative evidence about your product that could be better grounded in objective analysis. Aim to establish a balanced perspective on your product through a diverse set of research lenses, filling in gaps as you learn about new approaches.
Adjust our approach to project constraints. We've found that the only way to consistently incorporate research in our work is to adjust our approach to the context and constraints of any given project. Client expectations, project type, business goals, timelines, budget, and access to participants all influence the type, frequency, and output of our research. Iterative prototype testing of an email editor, for example, looks very different than post-launch qualitative studies for an editorial website. While some projects are research-intensive, short studies can also be worthwhile.
Reflect on successes and shortcomings. We have a longstanding practice of holding post-project team retrospectives to reflect on and document lessons for future work. Research has naturally come up in these conversations, and many of the things we've discussed you're reading right now. As an agency with a diverse set of clients, it's been important for us to understand what types of research work for what types of clients, and when. Make sure to take time to ask these questions after projects. Mid-project retrospectives can be beneficial, especially on long engagements, yet it's hard to see the forest when you're in the weeds.

‍

Streamline qualitative research processes 🚄

‍

Learning to be more efficient at planning, conducting, and analyzing research has helped us overturn the idea that some projects merit research while others don't. Remote moderated usability tests are one of our preferred methods, yet, in our experience, the biggest obstacle to incorporating these tests isn't the actual moderating or analyzing, but the overhead of acquiring and scheduling participants. While some agencies contract out the work of recruiting, we've found it less expensive and more reliable to collaborate with our clients to find the right people for our tests. That said, here are some recommendations for holding efficient qualitative tests:

Know your tools ahead of time. We use a number of tools to plan, schedule, annotate, and analyze qualitative tests (we're inveterate spreadsheet users). Learn your tools beforehand, especially if you're trying something new. Tools should fade into the background during tests, which Reframer does nicely.
Establish a recruiting process. When working with clients to find participants, we'll often provide an email template tailored to the project for them to send to existing or potential users of their product. This introductory email will contain a screener that asks a few project-related demographic or usage questions, and provides us with participant email addresses which we use to follow-up with a link to a scheduling tool. Once this process is established, the project manager will ensure that the UX designer on the team has a regular flow of participants. The recruiting process doesn't take care of itself – participants cancel, or reschedule, or sometimes don't respond at all – yet establishing an approach ahead of time allows you, the researcher, to focus on the research in the midst of the project.
Start recruiting early. Don't wait until you've finished writing a testing script to begin recruiting participants. Once you determine the aim and focal points of your study, recruit accordingly. Scripts can be revised and approved in the meantime.

‍

Be proactive about making research happen 🤸

‍

As a generalist design agency, we work with clients whose industries and products vary significantly. While some clients come to us with clear research priorities in mind, others treat it as an afterthought. Rare, however, is the client who is actively opposed to researching their product. More often than not, budget and timelines are the limiting factors. So we try not to make research an ordeal, but instead treat it as part of our normal process even if a client hasn't explicitly asked for it. Common-sense perspectives like Jakob Nielsen’s classic “Discount Usability for the Web” remind us that some research is always better than none, and that some can still be meaningfully pursued. We aren’t pushy about research, of course, but instead try to find a way to make it happen when it isn't a definite priority.

‍

‍World Usability Day is coming up on November 9, so now is a great time to stop and reflect on how you approach research and to brainstorm ways to improve your process. The tips above reflect some of the lessons we’ve learned at Viget as we’ve tried to improve our own process. We’d love to hear about approaches you’ve used as well.

Learn more

1 min read

Quantifying the value of User Research in 2024

Think your company is truly user-centric? Think again. Our groundbreaking report on UX Research (UXR) in 2024 shatters common assumptions about our industry.

We've uncovered a startling gap between what companies say about user-centricity and what they actually do. Prepare to have your perceptions challenged as we reveal the true state of UXR integration and its untapped potential in today's business landscape.

‍

The startling statistics

‍

Here's a striking finding: only 16% of organizations have fully embedded UXR into their processes and culture. This disconnect between intention and implementation underscores the challenges in demonstrating and maximizing the true value of user research.

‍

What's inside the white paper

‍

In this comprehensive white paper, we explore:

‍

How companies use and value UX research
Why it's hard to show how UX research helps businesses
Why having UX champions in the company matters
New ways to measure and show the worth of UX research
How to share UX findings with different people in the company
New trends changing how people see and use UX research

‍

Stats sneak peek

‍

- Only 16% of organizations have fully embedded UX Research (UXR) into their processes and culture. This highlights a significant gap between the perceived importance of user-centricity and its actual implementation in businesses.

‍

- 56% of organizations aren't measuring the impact of UXR at all. This lack of measurement makes it difficult for UX researchers to demonstrate the value of their work to stakeholders.

‍

- 68% of respondents believe that AI will have the greatest impact on the analysis and synthesis phase of UX research projects. This suggests that while AI is expected to play a significant role in UXR, it's seen more as a tool to augment human skills rather than replace researchers entirely.

‍

The UX research crossroads

‍

As our field evolves with AI, automation, and democratized research, we face a critical juncture: how do we articulate and amplify the value of UXR in this rapidly changing landscape? We’d love to know what you think! So DM us in socials and let us know what you’re doing to bridge the gap.

‍

Are you ready to unlock the full potential of UXR in your organization?

‍

Download our white paper for invaluable insights and actionable strategies that will help you showcase and maximize the value of user research. In an era of digital transformation, understanding and leveraging UXR's true worth has never been more crucial.

‍

Download the white paper

‍

What's next?

‍

Keep an eye out for our upcoming blog series, where we'll delve deeper into key findings and strategies from the report. Together, we'll navigate the evolving UX landscape and elevate the value of user insights in driving business success and exceptional user experiences.