Tips for recruiting quality research participants

For those new to the qualitative research space, there’s one question that’s usually pretty tough to figure out, and that’s the question of how many participants to include in a study. Regardless of whether it’s research as part of the discovery phase for a new product, or perhaps an in-depth canvas of the users of an existing service, researchers can often find it difficult to agree on the numbers. So is there an easy answer? Let’s find out.

‍

Here, we’ll look into the right number of participants for qualitative research studies. If you want to know about participants for quantitative research, read Nielsen Norman Group’s article.

‍

‍

Getting the numbers right

‍

So you need to run a series of user interviews or usability tests and aren’t sure exactly how many people you should reach out to. It can be a tricky situation – especially for those without much experience. Do you test a small selection of 1 or 2 people to make the recruitment process easier? Or, do you go big and test with a series of 10 people over the course of a month? The answer lies somewhere in between.

‍

It’s often a good idea (for qualitative research methods like interviews and usability tests) to start with 5 participants and then scale up by a further 5 based on how complicated the subject matter is. You may also find it helpful to add additional participants if you’re new to user research or you’re working in a new area.

‍

What you’re actually looking for here is what’s known as saturation.

‍

‍

Understanding saturation

‍

Whether it’s qualitative research as part of a master’s thesis or as research for a new online dating app, saturation is the best metric you can use to identify when you’ve hit the right number of participants.

‍

In a nutshell, saturation is when you’ve reached the point where adding further participants doesn’t give you any further insights. It’s true that you may still pick up on the occasional interesting detail, but all of your big revelations and learnings have come and gone. A good measure is to sit down after each session with a participant and analyze the number of new insights you’ve noted down.

‍

Interestingly, in a paper titled How Many Interviews Are Enough?, authors Greg Guest, Arwen Bunce and Laura Johnson noted that saturation usually occurs with around 12 participants in homogeneous groups (meaning people in the same role at an organization, for example). However, carrying out ethnographic research on a larger domain with a diverse set of participants will almost certainly require a larger sample.

‍

‍

Ensuring you’ve hit the right number of participants

‍

How do you know when you’ve reached saturation point? You have to keep conducting interviews or usability tests until you’re no longer uncovering new insights or concepts.

‍

While this may seem to run counter to the idea of just gathering as much data from as many people as possible, there’s a strong case for focusing on a smaller group of participants. In The logic of small samples in interview-based, authors Mira Crouch and Heather McKenzie note that using fewer than 20 participants during a qualitative research study will result in better data. Why? With a smaller group, it’s easier for you (the researcher) to build strong close relationships with your participants, which in turn leads to more natural conversations and better data.

‍

There's also a school of thought that you should interview 5 or so people per persona. For example, if you're working in a company that has well-defined personas, you might want to use those as a basis for your study, and then you would interview 5 people based on each persona. This maybe worth considering or particularly important when you have a product that has very distinct user groups (e.g. students and staff, teachers and parents etc).

‍

‍

How your domain affects sample size

‍

The scope of the topic you’re researching will change the amount of information you’ll need to gather before you’ve hit the saturation point. Your topic is also commonly referred to as the domain.

‍

If you’re working in quite a confined domain, for example, a single screen of a mobile app or a very specific scenario, you’ll likely find interviews with 5 participants to be perfectly fine. Moving into more complicated domains, like the entire checkout process for an online shopping app, will push up your sample size.

‍

As Mitchel Seaman notes: “Exploring a big issue like young peoples’ opinions about healthcare coverage, a broad emotional issue like postmarital sexuality, or a poorly-understood domain for your team like mobile device use in another country can drastically increase the number of interviews you’ll want to conduct.”

‍

‍

In-person or remote

‍

Does the location of your participants change the number you need for qualitative user research? Well, not really – but there are other factors to consider.

‍

• Budget: If you choose to conduct remote interviews/usability tests, you’ll likely find you’ve got lower costs as you won’t need to travel to your participants or have them travel to you. This also affects…
• Participant access: Remote qualitative research can be a lifesaver when it comes to participant access. No longer are you confined to the people you have physical access to, instead you can reach out to anyone you’d like.
• Quality: On the other hand, remote research does have its downsides. For one, you’ll likely find you’re not able to build the same kinds of relationships over the internet or phone as those in person, which in turn means you never quite get the same level of insights.

‍

‍

Is there value in outsourcing recruitment?

‍

Recruitment is understandably an intensive logistical exercise with many moving parts. If you’ve ever had to recruit people for a study before, you’ll understand the need for long lead times (to ensure you have enough participants for the project) and the countless long email chains as you discuss suitable times.

‍

Outsourcing your participant recruitment is just one way to lighten the logistical load during your research. Instead of having to go out and look for participants, you have them essentially delivered to you in the right number and with the right attributes.

‍

We’ve got one such service at Optimal, which means it’s the perfect accompaniment if you’re also using our platform of UX tools. Read more about that here.

‍

‍

Wrap-up

‍

So that’s really most of what there is to know about participant recruitment in a qualitative research context. As we said at the start, while it can appear quite tricky to figure out exactly how many people you need to recruit, it’s actually not all that difficult in reality.

‍

Overall, the number of participants you need for your qualitative research can depend on your project among other factors. It’s important to keep saturation in mind, as well as the locale of participants. You also need to get the most you can out of what’s available to you. Remember: Some research is better than none!

Summary: The equipment and tools you use to run your user testing sessions can make your life a lot easier. Here’s a quick guide.

‍

It’s that time again. You’ve done the initial scoping, development and internal testing, and now you need to take the prototype of your new design and get some qualitative data on how it works and what needs to be improved before release. It’s time for the user testing to begin.

‍

But the prospect of user testing raises an important question, and it’s one that many new user researchers often deliberate over: What gear or equipment should I take with me? Well, never fear. We’re going to break down everything you need to consider in terms of equipment, from video recording through to qualitative note-taking.

‍

Recording: Audio, screens and video

‍

The ability to easily record usability tests and user interviews means that even if you miss something important during a session, you can go back later and see what you’ve missed. There are 3 types of recording to keep in mind when it comes to user research: audio, video and screen recording. Below, we’ve put together a list of how you can capture each. You shouldn’t have to buy any expensive gear – free alternatives and software you can run on your phone and laptop should suffice.

‍

• Audio – Forget dedicated sound recorders; recording apps for smartphones (iOS and Android) allow you to record user interviews and usability tests with ease and upload the recordings to Google Drive or your computer. Good options include Sony’s recording app for Android and the built-in Apple recording app on iOS.
• Transcription – Once you’ve created a recording, you’ll no doubt want a text copy to work with. For this, you’ll need transcription software to take the audio and turn it into text. There are companies that will make transcriptions for you, but software like Transcribe means you can carry out the process yourself.
• Screen recording – Very useful during remote usability tests, screen recording software can show you exactly how participants react to the tasks you set out for them, even if you’re not in the room. OBS Studio is a good option for both Mac and Windows users. You can also use Quicktime (free) if you’re running the test in person.
• Video – Recording your participants as they make their way through the various tasks in a usability test can provide useful reference material at the end of your testing sessions. You can refer back to specific points in a video to capture any detail you may have missed, and you can share video with stakeholders to demonstrate a point. If you don’t have access to a dedicated camera, consider mounting your smartphone on a tripod and recording that way.

‍

Taking (and making use of) notes

‍

Notetaking and qualitative user testing go hand in hand. For most user researchers, notetaking during a research session means busting out the Post-it notes and Sharpie pens, rushing to take down every observation and insight and then having to arduously transcribe these notes after the session – or spend hours in workshops trying to identify themes and patterns. This approach still has merit, as it’s often one of the best ways to get people who aren’t too familiar with user research involved in the process. With physical notes, you can gather people around a whiteboard and discuss what you’re looking at. What’s more, you can get them to engage with the material directly.

‍

But there are digital alternatives. Qualitative notetaking software (like our very own Reframer) means you can bring a laptop into a user interview and take down observations directly in a secure environment. Even better, you can ask someone else to sit in as your notetaker, freeing you up to focus on running the session. Then, once you’ve run your tests, you can use the software for theme and pattern analysis, instead of having to schedule yet another full day workshop.

‍

Scheduling your user tests

‍

Ah, participant scheduling. Perhaps one of the most time-consuming parts of the user testing process. Thankfully, software can drastically reduce the logistical burden.

Here are some useful pieces of software:

‍

Dedicated scheduling tool Calendly is one of the most popular options for participant scheduling in the UX community. It’s really hands-off, in that you basically let the tool know when you’re available, share the Calendly link with your prospective participants, and then they select a time (from your available slots) that works for them. There are also a host of other useful features that make it a popular option for researchers, like integrations and smart timezones.

‍

If you’re already using the Optimal Workshop platform, you can use our  survey tool Questions as a fairly robust scheduling tool. Simply set up a study and add in prospective time slots. You can then use the multi-choice field option to have people select when they’re available to attend. You can also capture other data and avoid the usual email back and forth.

‍

Storing your findings
‍

One of the biggest challenges for user researchers is effectively storing and cataloging all of the research data that they start to build up. Whether it’s video recordings of usability tests, audio recordings or even transcripts of user interviews, you need to ensure that your data is A) easily accessible after the fact, and B) stored securely to ensure you’re protecting your participants.

‍

Here are some things to ask yourself when you store any piece of customer or user data:

‍

• Who will have access to this data?
• How long do I plan to keep this data?
• Will this data be anonymized?
• If I’m keeping physical data on hand, where will it be stored?

‍

Don’t make the mistake of thinking user data is ‘secure enough’, whether that’s on a company server that anyone can access, or even in an unlocked filing cabinet beneath your desk. Data privacy and security should always be at the top of your list of considerations. We won’t dive into best practices for participant data protection in this article, but instead, just mention that you need to be vigilant. Wherever you end up storing information, make sure you understand who has access.

‍

Wrap up

‍

Hopefully, this guide has given you an overview of some of the tools and software you can use before you start your next user test. We’ve also got a number of other interesting articles that you can read right here on our blog.

Knowing and understanding why and how your users use your product is invaluable for getting to the nitty gritty of usability. Delving deep with probing questions into motivation or skimming over looking for issues can equally be informative. 

‍

Put super simply, usability testing literally is testing how usable your product is for your users. If your product isn’t usable users often won’t complete their task, let alone come back for more. No one wants to lose users before they even get started. Usability testing gets under their skin and really into the how, why and what they want (and equally what they don’t).

‍

As we have been getting used to video calling regularly and using the internet for interactions, usability testing has followed suit. Being able to access participants remotely has allowed us to diversify the participant pool by not being restricted to those that are close enough to be in-person. This has also allowed an increase in the number of participants per test, as it becomes more cost-effective to perform remote usability testing.

‍

But if we’re remote, does this mean it can’t be moderated? No - remote testing, along with modern technology, can mean that remote testing can be facilitated and moderated. But what is the best method - moderated or unmoderated?

‍

What is moderated remote research testing?

‍

In traditional usability testing, moderated research is done in person. With the moderator and the participant in the same physical space. This, of course, allows for conversation and observational behavioral monitoring. Meaning the moderator can note not only what the participant answers but how and even make note of the body language, surroundings, and other influencing factors. 

‍

This has also meant that traditionally, the participant pool has been limited to those that can be available (and close enough) to make it into a facility for testing. And being in person has meant it takes time (and money) to perform these tests.

‍

As technology has moved along and the speed of internet connections and video calling has increased, this has opened up a world of opportunities for usability testing. Allowing usability testing to be done remotely. Moderators can now set up testing remotely and ‘dial in’ to observe participants anywhere they are. And potentially even running focus groups or other testing in a group format across the internet. 

‍

Pros of moderated remote research testing:

‍

- In-depth gathering of insights through a back-and-forth conversation and observing of the participants.

‍

- Follow-up questions don’t underestimate the value of being available to ask questions throughout the testing. And following up in the moment.

‍

- Observational monitoring noticing and noting the environment and how the participants are behaving, can give more insight into how or why they choose to make a decision.

‍

- Quick remote testing can be quicker to start, find participants, and complete than in-person. This is because you only need to set up a time to connect via the internet, rather than coordinating travel times, etc.

‍

- Location (local and/or international) Testing online removes reliance on participants being physically present for the testing. This broadens your ability to broaden the pool, and participants can be either within your country or global. 

‍

Cons of moderated remote research testing:

‍

- Time-consuming having to be present at each test takes time. As does analyzing the data and insights generated. But remember, this is quality data.

‍

- Limited interactions with any remote testing there is only so much you can observe or understand across the window of a computer screen. It can be difficult to have a grasp on all the factors that might be influencing your participants.

‍

What is unmoderated remote research testing?

‍

In its most simple sense, unmoderated user testing removes the ‘moderated’ part of the equation. Instead of having a facilitator guide participants through the test, participants are left to complete the testing by themselves and in their own time. For the most part, everything else stays the same. 

‍

Removing the moderator, means that there isn’t anyone to respond to queries or issues in the moment. This can either delay, influence, or even potentially force participants to not complete or maybe not be as engaged as you may like. Unmoderated research testing suits a very simple and direct type of test. With clear instructions and no room for inference. 

‍

Pros of unmoderated remote research testing:

‍

- Speed and turnaround,  as there is no need to schedule meetings with each and every participant. Unmoderated usability testing is usually much faster to initiate and complete.

‍

- Size of study (participant numbers) unmoderated usability testing allows you to collect feedback from dozens or even hundreds of users at the same time. 

- Location (local and/or international) Testing online removes reliance on participants being physically present for the testing, which broadens your participant pool.  And unmoderated testing means that it literally can be anywhere while participants complete the test in their own time.

‍

Cons of unmoderated remote research testing:

‍

- Follow-up questions as your participants are working on their own and in their own time, you can’t facilitate and ask questions in the moment. You may be able to ask limited follow-up questions.

‍

- Products need to be simple to use unmoderated testing does not allow for prototypes or any product or site that needs guidance. 

‍

- Low participant support without the moderator any issues with the test or the product can’t be picked up immediately and could influence the output of the test.

‍

When should you do moderated vs unmoderated remote usability testing?

‍

Each moderated and unmoderated remote usability testing have its use and place in user research. It really depends on the question you are asking and what you are wanting to know.

‍

Moderated testing allows you to gather in-depth insights, follow up with questions, and engage the participants in the moment. The facilitator has the ability to guide participants to what they want to know, to dig deeper, or even ask why at certain points. This method doesn’t need as much careful setup as the participants aren’t on their own. While this is all done online, it does still allow connection and conversation. This method allows for more investigative research. Looking at why users might prefer one prototype to another. Or possibly tree testing a new website navigation to understand where they might get lost and querying why the participant made certain choices.

‍

Unmoderated testing, on the other hand, is literally leaving the participants to it. This method needs very careful planning and explaining upfront. The test needs to be able to be set and run without a moderator. This lends itself more to wanting to know a direct answer to a query. Such as a card sort on a website to understand how your users might sort information. Or a first click to see how/where users will click on a new website.

‍

Planning your next user test? Here’s how to choose the right method

‍

With the ability to expand our pool of participants across the globe with all of the advances (and acceptance of) technology and video calling etc, the ability to expand our understanding of users’ experiences is growing. Remote usability testing is a great option when you want to gather information from users in the real world. Depending on your query, moderated or unmoderated usability testing will suit your study. As with all user testing, being prepared and planning ahead will allow you to make the most of your test.