Usability Testing 101
Your guide to planning and running effective usability tests with Reframer
Usability testing is a research technique aimed at evaluating how easy and intuitive something is to use. By testing early and often, you and your team can have confidence that what you’re designing and building makes sense to your users and helps them achieve their goals.
Usability testing involves observing participants as they complete a set of tasks. The goal is to reveal moments of frustration or confusion, identifying potential areas for improvement.
It can help you answer questions like:
- Can people find they information they need?
- Can they accomplish their goals?
- Can people understand the content?
- Are people getting lost or confused?
Types of usability testing
You can choose to conduct usability testing in person with your participant in the same room as you, or remotely through video calling software with screen sharing functionality. There’s also unmoderated testing, in which your participant takes part in your test in their own time.
Moderated usability testing
The benefits of conducting your test in person are you’re physically there with your participant to ask follow-up questions, pick up any non-verbal cues, as well as monitor their behavior. Doing usability testing in this way gives you a good level of depth and quality of insights as you’re seeing the sessions being conducted live. This can also be done remotely where you can watch the participant live through a video feed. This is useful for when your participants are in hard-to-reach locations or you don’t have the space or facilities to conduct the session in person.
Unmoderated usability testing
This method is useful for when you need a fast and convenient way to get your research done. In most cases, participants will use dedicated software and will complete a series of tasks in their own time. While it can be faster than traditional in-person usability testing, it doesn’t quite offer the same level of depth and insight, or give you the opportunity to follow up with your participant.
When to do usability testing
Testing a new design
Test as early and frequently as you can. The further along in development you go, the harder and more costly it is to make changes. Work with your team to identify key issues to prioritize and retest in subsequent rounds to ensure the design makes sense to people before launch.
If you’re developing a new design or interaction, you can test this at different levels of fidelity, from rough sketches through to interactive prototypes to developed features before release.
Use what you learn from each round of testing to improve your design, and then test it again. The more frequently you can iterate, the more confidence you’ll have in your design. Even in later stages of the development process, your team should be prepared to address larger usability problems and make relevant tweaks.
Usability testing should be an iterative process. Each round should be made up of several testing sessions, and it pays to do several rounds to get things right.
Testing an existing product or service
Conducting a usability test on an existing product or service helps you to assess the current solution and identify areas for improvement.
It also gives you an opportunity to benchmark the existing experience, making it easier to measure the impact of changes later down the track.
Once you have a new design in place, you can re-test it using the same tasks as your original test and see how things compare.
Testing your competitors
Reviewing the usability of your competitors is a great way to learn how your product or service compares to others in your market. It can also help glean insight into potential opportunities and gaps in your current offering and feed into strategic decision making.
Conducting usability tests on your competitors is no different to testing your own product. Identify a common set of tasks and scenarios to use during your study to easily compare the performance across different competitors.
Plan your research
The first and most important step is figuring out your research objectives with your team.
Perhaps your aim is to uncover current difficulties your users may have and identify opportunities for improvements? Or maybe you’d like to assess whether a current solution is meeting people’s needs?
Start by talking to your stakeholders to clarify the goals of your research and how findings will be used. If there are lots of objectives, prioritize them to ensure you’re focusing on the most important questions during your usability tests. Your stakeholders are people who have a vested interest in your project, for example, designers, product managers or clients.
Once you’ve narrowed down your research questions, it’s a good idea to give stakeholders the opportunity to provide feedback to ensure your research plan meets the needs of everyone involved.
Other things you need to plan:
- Decide who will be involved and what their roles are during the sessions.
- Discuss what the script used during your sessions will contain.
- Decide where sessions will take place and what equipment you’ll need.
- Discuss the kinds of participants you require.
Identify key tasks and scenarios
Usability testing is a task-based activity. This means you need to figure out the flow of actions you want your participants to complete during your session. Tasks should be representative of typical goals your end-users would perform, such as adding a bag onto an existing airline booking.
Scenarios help to guide the context in which the tasks take part. For example, booking the cheapest airline tickets for the upcoming family weekend in Vancouver. Good scenarios are realistic and relatable. Make sure you’re not leading your participants by being too detailed and prescriptive.
Work with your team to identify the key tasks and scenarios you want to cover during your test. Be realistic around how much you can fit within a session and do a run through to make sure that your timing is right before booking in your research participants.
Write your script
A script (or guide) is an outline of all the essential information to cover during your session.
It should include the key questions, scenarios and tasks you want your participant to complete, as well as any additional information you want to communicate to your participant, such as:
- introductions (including any observers present)
- a general overview of the session and the purpose of the research
- privacy and consent information
- recording permissions (if applicable, including information on how these will be used and stored)
- instructions to encourage participants to think aloud and verbalize their thoughts and feelings throughout the session
- a friendly reminder that you’re testing the product or service, not the participant
- any additional information, such as guidance on what to do when the participant gets stuck or has a question.
As a researcher, it’s your job to set clear expectations and make sure the participant feels comfortable. Ensure you cover off all essential information in your script, even if you do usability testing often.
Here’s a great example of a script from renowned usability testing expert Steve Krug.
Coordinate your first session
Before your first session, you’ll want to consider the finer details to help you get your usability tests right:
- Where will your sessions take place?
- Who will be involved in the sessions? What do they need to know?
- What technology do you need?
- What will your incentives be? How will you pay/supply these?
- How will you capture consent for using data and recordings from your sessions?
- Does your research require an NDA?
Work with your team to find the answers to these questions before you conduct your first session to keep your study running smoothly.
When you recruit participants, it’s important to find people who are representative of your end users to ensure the tasks and scenarios make sense to them. For example, if you’re testing the usability of a new baby monitor, you’ll get more reliable results testing with soon-to-be or existing parents, rather than members of the general public who may or may not have children of their own.
You can recruit participants in a variety of different ways, and how you do so will depend on a few different factors such as your budget, company policy, resources and time.
Ask yourself the following questions before recruiting:
- How will you keep track of your participants?
- How will you protect participant privacy?
- How will you handle their personal data?
There’s no correct answer to these questions — it will depend on your company’s policies and how your company handles data. As a rule of thumb, only collect the essential information you need to get your research done.
Decide how many participants to recruit
The number of people you recruit will largely depend on the time and resources available to you, as well as the availability of your target participants.
Aim for a number that will allow you to start seeing patterns in your data. Avoid making any decisions based off what one person said or did, unless they’ve come across a bug or usability issue that blocks people from completing a task successfully.
Try and include at least 5 participants in each round of testing. Add more if you continue to discover issues that are unique to each participant, until you start to see patterns emerge.
If you’re working with personas and have several different types of tasks that each user type may perform, you’ll want to recruit 5 participants from each user group to get reliable results.
Source your own participants
If you have access to a pool of participants (like employees if you’re working on an internal product, or your customer mailing list) then reaching out to potential participants via email can be an easy way to source them.
Similarly, you can recruit participants by advertising on social media channels or by running banners or pop ups on your website.
When sourcing participants for usability tests, you may want to direct them to a screener, depending on your needs. This is to ensure you’re reaching people who meet the specific criteria you’re looking for. What you decide to include in your screener will vary depending on your objectives.
Some useful things to include may be demographic information, previous experiences using your product or service, frequency of use, and anything else that will help you match your candidates against any target-user characteristics.
Source participants through recruiters and panels
You can also make use of recruitment providers, which can be a good option if you don’t have easy access to the people you’d like to target. Dedicated recruiters will generally have access to a wide variety of potential participants, which can make this a fast and easy way to source relevant people with minimal effort.
In some cases, they’ll also oversee logistics, such as scheduling sessions and paying out incentives on your behalf. Keep in mind that recruiters and panels come at a higher price than sourcing participants on your own.
Make sure you provide your recruiter with a clear understanding of the participant characteristics you’re looking for – similar to a screening questionnaire.
An alternative to using recruiting agencies is sourcing participants on your own from online panels. These self-service panels contain databases of potential participants, allowing you to filter, match and submit orders based on your own criteria without relying on a dedicated recruiter.
To encourage potential participants to take part in your research, you’ll want to consider how you incentivize them. While you can get away with little to no incentive for internal participants (for example, existing users within your organization), external participants may be less obligated to participate if they’re not reimbursed for their time.
Incentives can be monetary in the form of cash or gift cards, or take other forms, such as company merchandise or free access to your product or service.
The amount you decide to pay your participants will depend on a few different factors. Consider the length of the session, the complexity of the tasks and whether you need participants with specific experience or skills. For example, if you’re testing a specialist medical interface with doctors, the incentive you offer should be higher than if you’re testing a banking app that has a wider range of users.
An incentive should be high enough to motivate people to participate in your study. A good starting point is to consider how much a participant would usually receive for an hour of their time and add in extra to cover the additional effort required for participating.
Set up your study in Reframer
Set up your study in Reframer in advance so that you’re ready to go for your first usability testing session.
Create a study
You can organize your usability testing sessions and the observations that you capture for each session within a Reframer study.
When you create a new study, you’ll be able to add study objectives and reference links in the overview screen. Use this space to outline the purpose of your research and any other important details.
This is also a place to keep links to prototypes, wireframes and other documents so your team has everything they need on the day of your usability test.
Once you have your first participants scheduled, you can start to create sessions in Reframer. Use the ‘Session information’ field to add any relevant information about your participant during or after your session, such as demographics, their experience using your product, or any other details worth noting.
This is also where you add Segments to each of your sessions. Segments are useful for identifying participant characteristics such as demographics, personas or user types. They come in handy when you’re analyzing your results and wanting to identify patterns in your data across different groups.
For this study, some useful information to capture about our participants could be things such as the device used during the test, their job, age group and how long they’ve been a customer.
Define a tag structure for your study
Tags allow you to code your observations, helping you identify and organize relevant findings later on.
From the study overview, you can create and manage the tags you intend to use for your study. The Tags tab will display all the tags used by you and your team.
Create tags that will make it easy for you to filter and make sense of your data when you move on to the next stage – analysis. For usability testing, you’ll want to create a set of tags dedicated to the tasks, flows or screens you’ll be testing, as well as tags that will help you identify any problems or issues your participants encounter. This could be as simple as a ‘pass’ or ‘fail’ for each of your tasks, or a group of tags that relate to sentiment, such as ‘delighted’, ‘frustrated’ or ‘surprised’.
Whether you defined tags upfront or not, you can still change or create new tags any time.
Once you move on to analysis, you’ll easily be able to filter your data by task and see at a glance where your participants may have run into issues. This will help you to identify potential improvements.
Tips for tagging observations
- If you’re involving others in your research sessions or analysis, it’s a good idea to agree on a consistent set of tags in advance.
- Add tags for keywords you expect to use in your study to help standardize the spelling and style of tags for you and your team.
- Include tags in your script to remind you what tags to use when capturing observations.
- To encourage better quality observations, leave tagging until after your session.
Think about what you need on the day
Consider the environment in which your usability tests will take place so you have what you need for your first session.
Some things to ask yourself:
- Where will the session be held?
- How will you record the session (including any screens)?
- Where should your participant sit to make them most comfortable?
- What technology will your notetakers or observers require?
- Do you need to set up an observation room for anyone else involved?
- If you’re hosting a remote session, how will you set up the call to make it easy for others to observe what’s happening on the screen?
Involve others in your usability tests
Getting team members and stakeholders involved in your research sessions is a great way to expose them to the people actually using your products and services, and makes for a great empathy building exercise.
It also helps to get people on board with the research findings — there’s no better evidence than seeing it for themselves.
Consider including members of your immediate team, such as developers, designers, product owners, as well as those from other areas of the business, such as marketing, support, management.
When to get others involved
Getting other people involved in your research can be done in a myriad of ways and can depend on how your team or company is run. Some ways to include others in your research:
- collaborate on defining objectives and key tasks
- encourage team members and stakeholders to play the role of notetakers or observers during your tests
- conduct debriefs following each research session to pull out key insights and observations from your team members
- after you’ve completed several sessions, involve others when you start to analyze your data to get everyone on the same page.
How many people to involve
While you want to get as many people involved in your research as possible, try to limit the number of people in the same room as your participant when running in-person sessions. This will create a much more relaxed environment for your participant. Keep the number down to the essentials: the facilitator and a notetaker or observer.
If you’d like to invite others to observe the session, consider setting up an observer room with a live feed from the test room. It’s also a good idea to record your sessions so you can share them with those who are unable to attend on the day.
How to prepare your team for the test
If you’ve recruited some coworkers or stakeholders to take part in your sessions, you’ll want to brief them to ensure they’re clear on what they can expect. This is crucial if they have never been part of a usability test before. It’s important to explain what a usability test is, how they’re run, what the value is and what sort of findings it will generate.
People you involve in your study could have a few different roles, such as observing, notetaking, asking questions or even being involved with the analysis later on.
No matter who you’re involving or what their roles are, go over the following details:
- when and where are the sessions taking place
- how many sessions they should attend
- what you expect from them during and after the session
- when they will have an opportunity to ask participants questions (many researchers will specify no questions until the end of the session and/or when prompted)
- let them know not to interject during the test, and ask them to hold off an any additional questions until the end
- how to act during the test and what their role is.
Tips for notetakers
Notetakers have an important job that requires a lot of focus. Make sure you teach your notetakers how to take good quality notes and what to listen and look for during the session. Let your notetakers know what happens to their notes after the session and how you’ll be using them.
If there are any tools you’ll be using during the test, let your notetakers know in advance so they can become familiar with them ahead of time.
For more help with taking effective observations, read this article on UX Mastery’s blog.
Tips for observers
While observing is the easiest way to involve a large number of people, it’s important to ensure everyone stays engaged so they can understand the value. Encourage observers (especially key stakeholders) to write down their top insights following each session, and invite them to participate in a debrief afterwards to identify patterns among everyone involved.
Run your sessions
Ideally, your usability testing sessions will involve:
- your participant
- a facilitator (the person asking questions)
- one or more notetakers
- any number of observers (optional).
Occasionally, you may have to play the role of the facilitator and notetaker at the same time. This can be the case if you’re running testing sessions on your own, or others are unable to make it at the scheduled time. If this is the case, stay focussed on your participant by recording your session and writing up observations afterwards.
Almost anyone can facilitate a usability test; all it really takes is the courage to try it. With a little practice, most people can get quite good at it. Try to choose someone who tends to be patient, calm, empathetic, a good listener, and inherently fair.
How to facilitate the session
At a basic level, the role of the facilitator is to follow the testing script and keep the session on track. This includes introducing others involved, explaining the purpose of the session, keeping the participant comfortable and guiding them as they complete a series of tasks.
In reality, facilitating a usability testing session involves a range of questioning and probing to get to the core of what the participant is thinking and feeling as they interact with the design.
As a facilitator, you should continually encourage your participant to think out loud, and address any hesitations and pauses. Non-verbal cues (such as facial expressions) can provide a lot of insight into what’s going on inside a person’s head, allowing you to reach deeper insights. Make sure you’re not influencing participants in any way by leading them through the test, asking loaded questions or suggesting answers.
While others observing the usability test may have questions they want to ask throughout the session, it’s up to the facilitator to decide on the best time to follow up with any subsequent questions.
It can be tempting for participants to ask the facilitator questions to verify they’re on the right track. To avoid this, it’s best to let the participant know upfront that you may defer answering any questions until the end to avoid influencing their actions.
How to capture observations as a notetaker
Capturing observations as your session is taking place ensures you don’t miss the essential details as they arise. It also takes some of the load off after your session is complete, allowing you to quickly identify key findings.
Observations refer to any relevant things you see or hear during your usability testing sessions. In Reframer, you’ll capture your qualitative data in the form of individual observations.
If you have a dedicated notetaker, they can focus on taking notes while you facilitate the session.
If you’re on your own and don’t have a dedicated notetaker, it’s best to record your session and stay focused on your participant. You can listen back and take notes later.
Information to include in your observations
When capturing observations, it’s important to focus not only on what your participant said but also what they did. There is often a difference between what people say they do, and what they actually end up doing. Capturing behavioral information can give your data an additional level of insight.
The role of a notetaker is not only to ensure key information is being recorded, but also to observe the session and make note of any behaviors and nonverbal cues (pauses, hesitations, looks of frustration or confusion and so on).
Sometimes little things can have a big impact, so be sure to include smaller details in your observations. For example, a participant using an operating system they’re not familiar with may affect their ability to perform tasks.
Tips for capturing observations in Reframer
If you’re a notetaker you can capture your observations directly in Reframer while your session is taking place. Observations can also be added after the session is complete (for example, from a recording). You can also import existing data into Reframer from elsewhere.
Capturing observations directly into Reframer is as easy as writing a note and hitting enter. The observation you’ve written will automatically save and appear in your list of observations. Each of your sessions can have an unlimited number of observations.
You may choose to capture your observations verbatim, or only make note of key information throughout your session. Including quotes is a great way to bring your observations to life.
To help focus on your participant during your session, you may want to refrain from tagging until after your session is over.
You can also:
- use the star to quickly highlight any particularly important observations, or those you want to quickly refer back to later on
- use quotation marks to automatically add a “quote” tag to your observation
- use the hash symbol (#) to quickly tag observations as you capture them, or pick from an existing list.
Debrief after your test
Once your session is complete, it’s a good idea to conduct a debrief, allowing the team to come together and pull out key insights from what everyone saw and heard during the session.
Host this meeting as soon as possible after the session has finished so that the information is still fresh in your minds — booking in time immediately after each session is ideal!
If you had multiple notetakers and observers, encourage each person to list their most significant outtakes from the session to find patterns among what was observed across the team.
The debrief is an opportunity for the team to look through the session and review relevant findings in Reframer. Use the filter to bring up any starred observations so the team can discuss them together, such as why they were starred or if anything in particular made these observations stand out. You can even use this time together to tidy up some of the data and get rid of duplicates and errors before moving on to analysis.
Capture observations from a recording
In some circumstances, you may choose to record your sessions and make notes by reviewing the recordings.
You can use Reframer to make notes from your video or voice recordings in the same way a notetaker would capture observations during live in-person sessions.
This can be the case if you don’t have a dedicated notetaker, in which making notes on your own during the session could be disruptive.
Import observations from elsewhere
If your notetaker has used a different method for taking notes other than Reframer, you can import this information directly into your individual sessions. If you have information on the time your observations were taken, you can include timestamps for individual observations.
Review your data
Focusing on your participant while capturing observations in real-time can be a challenge. The review tab allows you to check over what you’ve captured at the end of your session and tidy things up.
Take the time to get your data in good shape for analysis — fix any typos or incomplete observations, remove accidental duplicates and get rid of incorrectly used tags. This is also a good time to recall any significant observations or tag relevant findings.
Filter your observations
The review stage gives you the opportunity to immerse yourself in the data from your session. Filtering comes in handy when you want to bring up all the observations relating to a particular keyword, making it easier to add tags to multiple observations at once.
For example, you can search for words such as ‘confusing’, ‘hard’ and ‘unsure’ to quickly identify places where your participant experienced issues with your design, or search for a particular moment in your session based on what your participant had said.
Here you can also view any starred observations that may have stood out to you or anyone else during your session.
You can also use the search box to filter observations by tag or author. This allows you to quickly see how a particular tag was used during your session, or review observations made by others in your team.
Tag your observations
Tagging your observations helps you make sense of your data once you move on to analysis. There, you’ll be able to filter your observations based on tags and combine these to uncover themes. For example, observations tagged with #confusion and #check_out may suggest issues with that particular feature.
Use tags to link observations back to your research objectives, identify sentiment or help to mark any particularly significant findings.
You can add tags directly to your observations using hashtags (for example, #frustration), or using the tag list in the sidebar on the right. You can also multi-select your observations and add several tags at once.
Star your observations
The star feature allows you to quickly highlight key observations from your session, making them stand out from the rest of your findings.
Stars function in a similar way to tags, making it easy to filter key findings, quotes, or notes of interest when you begin to analyze your results across all your sessions.
Use this feature to mark any observations that are particularly significant, or those that provide great quotes or outtakes from your session.
Make sense of your data
Before you dive into analyzing your data, it’s important to consider the approach you’ll take throughout to make sure you get started on the right foot.
Since your usability tests will likely already follow tasks in a particular order, approach analysis the same way by looking at your results one task at a time.
Get a high level overview of your data
The Results Overview provides you and your team with a summary of some key statistics for your project, including the total number of observations and frequency of tags used.
Once you start to create themes, these will display here too, giving you an overview of the top themes identified across your study.
Explore your data and create themes
The core of your Reframer analysis lies in the theme builder, which allows you to create themes by grouping observations that all relate to a similar topic or finding.
Using the theme builder, you can filter your observations by combinations of sessions and tags. When you select a tag, you’ll see all observations in your study that have been assigned that specific tag. From here, you can see any other tags that were used in combination with the one you selected, and broaden or narrow your selection to reveal relationships between them.
When you narrow your tag selection you can reveal observations that contain the specific tags you selected (for example, ‘T1_Login’ AND ‘delight’).
Broadening your criteria will show you observations that contain at least one of the tags you have selected (for example, ‘frustration’ OR ‘fail’).
Tips for creating themes
To get the most use out of the theme builder, it’s best to start by exploring themes that relate to your key research objectives. You may want to look at your data from the perspective of:
- major issues for each task, or steps in a workflow
- the most significant observations based on those that have been starred during the session
- parts of the test where participants ran into the most pain points
- moments where participants experienced moments of delight
- any suggestions, requests or ideas your participants had
- patterns in existing behaviors reported by participants at the start of the session
- finding quotes to support the outcomes of your study.
When exploring the relationships between different tags, pay attention to tags that have a strong relationship, as well as those that don’t. These can help you validate any assumptions you had prior to your research. For example, you may have had concerns that a new navigation will confuse users and find that, in fact, no one encountered any issues finding what they needed.
When your tag combinations start to reveal interesting patterns in your data, you can save these as a theme. Themes group together observations that relate to a similar topic. You can name your theme and add a description for additional context and help other team members understand what you’ve discovered.
Explore relationships through the chord diagram
The chord diagram is a visualization that helps you understand the relationships between the tags you’ve used throughout your observations.
You can also exclude any number of tags from the visualization, making it easy to focus on relevant tags.
You can use the chord diagram to explore relationships you may want to delve into in greater detail in the theme builder.
Understand your data through the bubble chart
The bubble chart shows you the relative frequency of all the tags used throughout your project. The larger the bubble, the more frequently a tag was used.
The bubble chart makes it easy to identify tags that were used most often and can help in understanding common findings in your research. It’s also a great way to showcase your findings to stakeholders.
Similar to the chord diagram, you can exclude any number of tags from the visualization. To help you focus on your most important findings, consider removing non-specific tags, such as stars or “quote” tags to help you get a sense of the themes in your data.
Download your results
If you need to export your data for any reason, Reframer makes it easy to download this information in .XLSX format. If you download the spreadsheet you’ll have access to the data from all your sessions, including starred observations, tags, timestamps and any themes you created when analyzing your data.
Share your results with others
Once you’ve identified the key themes from your research, it’s time to share your findings with others.
There’s no right or wrong way of sharing your findings with the rest of your team. While in some cases a simple session with your team and stakeholders to run through the themes you identified will suffice, in other cases you may have to compile supporting evidence such as quotes, audio or video clips to get others onboard.
The most important thing about usability testing is making sure the most significant findings are understood, prioritized and actioned.
Remember — usability testing is an iterative process. Each round should lead to improvements, and with time, lead to more confidence in you and your team, and a better experience for your users.