November 18, 2022
4 min

Moderated vs unmoderated research: which approach is best?

Knowing and understanding why and how your users use your product is invaluable for getting to the nitty gritty of usability. Delving deep with probing questions into motivation or skimming over looking for issues can equally be informative. 

Put super simply, usability testing literally is testing how usable your product is for your users. If your product isn’t usable users often won’t complete their task, let alone come back for more. No one wants to lose users before they even get started. Usability testing gets under their skin and really into the how, why and what they want (and equally what they don’t).

As we have been getting used to video calling regularly and using the internet for interactions, usability testing has followed suit. Being able to access participants remotely has allowed us to diversify the participant pool by not being restricted to those that are close enough to be in-person. This has also allowed an increase in the number of participants per test, as it becomes more cost-effective to perform remote usability testing.

But if we’re remote, does this mean it can’t be moderated? No - remote testing, along with modern technology, can mean that remote testing can be facilitated and moderated. But what is the best method - moderated or unmoderated?

What is moderated remote research testing? 🙋🏻

In traditional usability testing, moderated research is done in person. With the moderator and the participant in the same physical space. This, of course, allows for conversation and observational behavioral monitoring. Meaning the moderator can note not only what the participant answers but how and even make note of the body language, surroundings, and other influencing factors. 

This has also meant that traditionally, the participant pool has been limited to those that can be available (and close enough) to make it into a facility for testing. And being in person has meant it takes time (and money) to perform these tests.

As technology has moved along and the speed of internet connections and video calling has increased, this has opened up a world of opportunities for usability testing. Allowing usability testing to be done remotely. Moderators can now set up testing remotely and ‘dial in’ to observe participants anywhere they are. And potentially even running focus groups or other testing in a group format across the internet. 

Pros:

- In-depth gathering of insights through a back-and-forth conversation and observing of the participants.

- Follow-up questions don’t underestimate the value of being available to ask questions throughout the testing. And following up in the moment.

- Observational monitoring noticing and noting the environment and how the participants are behaving, can give more insight into how or why they choose to make a decision.

- Quick remote testing can be quicker to start, find participants, and complete than in-person. This is because you only need to set up a time to connect via the internet, rather than coordinating travel times, etc.

- Location (local and/or international) Testing online removes reliance on participants being physically present for the testing. This broadens your ability to broaden the pool, and participants can be either within your country or global. 

Cons:

- Time-consuming having to be present at each test takes time. As does analyzing the data and insights generated. But remember, this is quality data.

- Limited interactions with any remote testing there is only so much you can observe or understand across the window of a computer screen. It can be difficult to have a grasp on all the factors that might be influencing your participants.

What is unmoderated remote research testing? 😵💫

In its most simple sense, unmoderated user testing removes the ‘moderated’ part of the equation. Instead of having a facilitator guide participants through the test, participants are left to complete the testing by themselves and in their own time. For the most part, everything else stays the same. 

Removing the moderator, means that there isn’t anyone to respond to queries or issues in the moment. This can either delay, influence, or even potentially force participants to not complete or maybe not be as engaged as you may like. Unmoderated research testing suits a very simple and direct type of test. With clear instructions and no room for inference. 

Pros:

- Speed and turnaround,  as there is no need to schedule meetings with each and every participant. Unmoderated usability testing is usually much faster to initiate and complete.

- Size of study (participant numbers) unmoderated usability testing allows you to collect feedback from dozens or even hundreds of users at the same time. 


- Location (local and/or international) Testing online removes reliance on participants being physically present for the testing, which broadens your participant pool.  And unmoderated testing means that it literally can be anywhere while participants complete the test in their own time.

Cons:

- Follow-up questions as your participants are working on their own and in their own time, you can’t facilitate and ask questions in the moment. You may be able to ask limited follow-up questions.

- Products need to be simple to use unmoderated testing does not allow for prototypes or any product or site that needs guidance. 

- Low participant support without the moderator any issues with the test or the product can’t be picked up immediately and could influence the output of the test.

When should you do which? 🤔

Each moderated and unmoderated remote usability testing have its use and place in user research. It really depends on the question you are asking and what you are wanting to know.

Moderated testing allows you to gather in-depth insights, follow up with questions, and engage the participants in the moment. The facilitator has the ability to guide participants to what they want to know, to dig deeper, or even ask why at certain points. This method doesn’t need as much careful setup as the participants aren’t on their own. While this is all done online, it does still allow connection and conversation. This method allows for more investigative research. Looking at why users might prefer one prototype to another. Or possibly tree testing a new website navigation to understand where they might get lost and querying why the participant made certain choices.

Unmoderated testing, on the other hand, is literally leaving the participants to it. This method needs very careful planning and explaining upfront. The test needs to be able to be set and run without a moderator. This lends itself more to wanting to know a direct answer to a query. Such as a card sort on a website to understand how your users might sort information. Or a first click to see how/where users will click on a new website.

Wrap Up 🌯

With the ability to expand our pool of participants across the globe with all of the advances (and acceptance of) technology and video calling etc, the ability to expand our understanding of users’ experiences is growing. Remote usability testing is a great option when you want to gather information from users in the real world. Depending on your query, moderated or unmoderated usability testing will suit your study. As with all user testing, being prepared and planning ahead will allow you to make the most of your test.

Share this article
Author
Optimal
Workshop

Related articles

View all blog articles
Learn more
1 min read

5 key areas for effective ResearchOPs

Simply put, ResearchOps is about making sure your research operations are robust, thought through and managed. 

Having systems and processes around your UX research and your team keep everyone (and everything) organized. Making user research projects quicker to get started and more streamlined to run. And robust sharing, socializing, and knowledge storage means that everyone can understand the research insights and findings and put these to use - across the organization. And even better, find these when they need them. 

Using the same tools across the team allows the research team to learn from each other, and previous research projects and be able to compare apples with apples, with everyone included. Bringing the team together across tools, research and results.

We go into more detail in our ebook ResearchOps Checklist about exactly what you can do to make sure your research team is running at its best. Let’s take a quick look at 5 way to ensure you have the grounding for a successful ResearchOps team.

1. Knowledge management 📚

What do you do with all of the insights and findings of a user research project? How do you store them, how do you manage the insights, and how do you share and socialize?

Having processes in place that manage this knowledge is important to the longevity of your research. From filing to sharing across platforms, it all needs to be standardized so everyone can search, find and share.

2. Guidelines and process templates 📝

Providing a framework for how to run research projects is are important. Building on the knowledge base from previous research can improve research efficiencies and cut down on groundwork and administration. Making research projects quicker and more streamlined to get underway.

3. Governance 🏛

User research is all about people, real people. It is incredibly important that any research be legal, safe, and ethical. Having effective governance covered is vital.

4. Tool stack 🛠

Every research team needs a ‘toolbox’ that they can use whenever they need to run card sorts, tree tests, usability tests, user interviews, and more. But which software and tools to use?

Making sure that the team is using the same tools also helps with future research projects, learning from previous projects, and ensuring that the information is owned and run by the organization (rather than whichever individuals prefer). Reduce logins and password shares, and improve security with organization-wide tools and platforms. 

5. Recruitment 👱🏻👩👩🏻👧🏽👧🏾

Key to great UX research is the ability to recruit quality participants - fast! Having strong processes in place for screening, scheduling, sampling, incentivizing, and managing participants needs to be top of the list when organizing the team.

Wrap Up 💥

Each of these ResearchOps processes are not independent of the other. And neither do they flow from one to the other. They are part of a total wrap around for the research team, creating processes, systems and tools that are built to serve the team. Allowing them to focus on the job of doing great research and generating insights and findings that develop the very best user experience. 

Afterall, we are creating user experiences that keep our users engaged and coming back. Why not look at the teams user experience and make the most of that. Freeing time and space to socialize and share the findings with the organization. 

Learn more
1 min read

5 ways to increase user research in your organization

Co-authored by Brandon Dorn, UX designer at Viget.As user experience designers, making sure that websites and tools are usable is a critical component of our work, and conducting user research enables us to assess whether we’re achieving that goal or not. Even if we want to incorporate research, however, certain constraints may stand in our way.

A few years ago, we realized that we were facing this issue at Viget, a digital design agency, and we decided to make an effort to prioritize user research. Almost two years ago, we shared initial thoughts on our progress in this blog post. We’ve continued to learn and grow as researchers since then and hope that what we’ve learned along the way can help your clients and coworkers understand the value of research and become better practitioners. Below are some of those lessons.

Make research a priority for your organization

Before you can do more research, it needs to be prioritized across your entire organization — not just within your design team. To that end, you should:

  • Know what you’re trying to achieve. By defining specific goals, you can share a clear message with the broader organization about what you’re after, how you can achieve those goals, and how you will measure success. At Viget, we shared our research goals with everyone at the company. In addition, we talked to the business development and project management teams in more depth about specific ways that they could help us achieve our goals, since they have the greatest impact on our ability to do more research.
  • Track your progress. Once you’ve made research a priority, make sure to review your goals on an ongoing basis to ensure that you’re making progress and share your findings with the organization. Six months after the research group at Viget started working on our goals, we held a retrospective to figure out what was working — and what wasn’t.
  • Adjust your approach as needed. You won’t achieve your goals overnight. As you put different tactics into action, adjust your approach if something isn’t helping you achieve your goals. Be willing to experiment and don’t feel bad if a specific tactic isn’t successful.

Educate your colleagues and clients

If you want people within your organization to get excited about doing more research, they need to understand what research means. To educate your colleagues and clients, you should:

  • Explain the fundamentals of research. If someone has not conducted research before, they may not be familiar or feel comfortable with the vernacular. Provide an overview of the fundamental terminology to establish a basic level of understanding. In a blog post, Speaking the Same Language About Research, we outline how we established a common vocabulary at Viget.
  • Help others understand the landscape of research methods. As designers, we feel comfortable talking about different methodologies and forget that that information will be new to many people. Look for opportunities to increase understanding by sharing your knowledge. At Viget, we make this happen in several ways. Internally, we give presentations to the company, organize group viewing sessions for webinars about user research, and lead focused workshops to help people put new skills into practice. Externally, we talk about our services and share knowledge through our blog posts. We are even hosting a webinar about conducting user interviews in November and we'd love for you to join us.
  • Incorporate others into the research process. Don't just tell people what research is and why it's important — show them. Look for opportunities to bring more people into the research process. Invite people to observe sessions so they can experience research firsthand or have them take on the role of the notetaker. Another simple way to make people feel involved is to share findings on an ongoing basis rather than providing a report at the end of the process.

Broaden your perspective while refining your skill set

Our commitment to testing assumptions led us to challenge ourselves to do research on every project. While we're dogmatic about this goal, we're decidedly un-dogmatic about the form our research takes from one project to another. To pursue this goal, we seek to:

  • Expand our understanding. To instill a culture of research at Viget, we've found it necessary to question our assumptions about what research looks like. Books like Erika Hall’s Just Enough Research teach us the range of possible approaches for getting useful user input at any stage of a project, and at any scale. Reflect on any methodological biases that have become well-worn paths in your approach to research. Maybe your organization is meticulous about metrics and quantitative data, and could benefit from a series of qualitative studies. Maybe you have plenty of anecdotal and qualitative evidence about your product that could be better grounded in objective analysis. Aim to establish a balanced perspective on your product through a diverse set of research lenses, filling in gaps as you learn about new approaches.
  • Adjust our approach to project constraints. We've found that the only way to consistently incorporate research in our work is to adjust our approach to the context and constraints of any given project. Client expectations, project type, business goals, timelines, budget, and access to participants all influence the type, frequency, and output of our research. Iterative prototype testing of an email editor, for example, looks very different than post-launch qualitative studies for an editorial website. While some projects are research-intensive, short studies can also be worthwhile.
  • Reflect on successes and shortcomings. We have a longstanding practice of holding post-project team retrospectives to reflect on and document lessons for future work. Research has naturally come up in these conversations, and many of the things we've discussed you're reading right now. As an agency with a diverse set of clients, it's been important for us to understand what types of research work for what types of clients, and when. Make sure to take time to ask these questions after projects. Mid-project retrospectives can be beneficial, especially on long engagements, yet it's hard to see the forest when you're in the weeds.

Streamline qualitative research processes 🚄

Learning to be more efficient at planning, conducting, and analyzing research has helped us overturn the idea that some projects merit research while others don't. Remote moderated usability tests are one of our preferred methods, yet, in our experience, the biggest obstacle to incorporating these tests isn't the actual moderating or analyzing, but the overhead of acquiring and scheduling participants. While some agencies contract out the work of recruiting, we've found it less expensive and more reliable to collaborate with our clients to find the right people for our tests. That said, here are some recommendations for holding efficient qualitative tests:

  • Know your tools ahead of time. We use a number of tools to plan, schedule, annotate, and analyze qualitative tests (we're inveterate spreadsheet users). Learn your tools beforehand, especially if you're trying something new. Tools should fade into the background during tests, which Reframer does nicely.
  • Establish a recruiting process. When working with clients to find participants, we'll often provide an email template tailored to the project for them to send to existing or potential users of their product. This introductory email will contain a screener that asks a few project-related demographic or usage questions, and provides us with participant email addresses which we use to follow-up with a link to a scheduling tool. Once this process is established, the project manager will ensure that the UX designer on the team has a regular flow of participants. The recruiting process doesn't take care of itself – participants cancel, or reschedule, or sometimes don't respond at all – yet establishing an approach ahead of time allows you, the researcher, to focus on the research in the midst of the project.
  • Start recruiting early. Don't wait until you've finished writing a testing script to begin recruiting participants. Once you determine the aim and focal points of your study, recruit accordingly. Scripts can be revised and approved in the meantime.

Be proactive about making research happen 🤸

As a generalist design agency, we work with clients whose industries and products vary significantly. While some clients come to us with clear research priorities in mind, others treat it as an afterthought. Rare, however, is the client who is actively opposed to researching their product. More often than not, budget and timelines are the limiting factors. So we try not to make research an ordeal, but instead treat it as part of our normal process even if a client hasn't explicitly asked for it. Common-sense perspectives like Jakob Nielsen’s classic “Discount Usability for the Web” remind us that some research is always better than none, and that some can still be meaningfully pursued. We aren’t pushy about research, of course, but instead try to find a way to make it happen when it isn't a definite priority.

World Usability Day is coming up on November 9, so now is a great time to stop and reflect on how you approach research and to brainstorm ways to improve your process. The tips above reflect some of the lessons we’ve learned at Viget as we’ve tried to improve our own process. We’d love to hear about approaches you’ve used as well.

Learn more
1 min read

How to convince others of the importance of UX research

There’s not much a parent won’t do to ensure their child has the best chance of succeeding in life. Unsurprisingly, things are much the same in product development. Whether it’s a designer, manager, developer or copywriter, everyone wants to see the product reach its full potential.

Key to a product’s success (even though it’s still not widely practiced) is UX research. Without research focused on learning user pain points and behaviors, development basically happens in the dark. Feeding direct insights from customers and users into the development of a product means teams can flick the light on and make more informed design decisions.

While the benefits of user research are obvious to anyone working in the field, it can be a real challenge to convince others of just how important and useful it is. We thought we’d help.

Define user research

If you want to sell the importance of UX research within your organization, you’ve got to ensure stakeholders have a clear understanding of what user research is and what they stand to gain from backing it.

In general, there are a few key things worth focusing on when you’re trying to explain the benefits of research:

  • More informed design decisions: Companies make major design decisions far too often without considering users. User research provides the data needed to make informed decisions.
  • Less uncertainty and risk: Similarly, research reduces risk and uncertainty simply by giving companies more clarity around how a particular product or service is used.
  • Retention and conversion benefits: Research means you’ll be more aligned with the needs of your customers and prospective customers.

Use the language of the people you’re trying to convince. A capable UX research practice will almost always improve key business metrics, namely sales and retention.

The early stages

When embarking on a project, book in some time early in the process to answer questions, explain your research approach and what you hope to gain from it. Here are some of the key things to go over:

  • Your objectives: What are you trying to achieve? This is a good time to cover your research questions.
  • Your research methods: Which methods will you be using to carry out your research? Cover the advantages of these methods and the information you’re likely to get from using them.
  • Constraints: Do you see any major obstacles? Any issues with resources?
  • Provide examples: Nothing shows the value of doing research quite like a case study. If you can’t find an example of research within your own organization, see what you can find online.

Involve others in your research

When trying to convince someone of the validity of what you’re doing, it’s often best to just show them. There are a couple of effective ways you can do this – at a team or individual level and at an organizational level.

We’ll explain the best way to approach this below, but there’s another important reason to bring others into your research. UX research can’t exist in a vacuum – it thrives on integration and collaboration with other teams. Importantly, this also means working with other teams to define the problems they’re trying to solve and the scope of their projects. Once you’ve got an understanding of what they’re trying to achieve, you’ll be in a better position to help them through research.

Educate others on what research is

Education sessions (lunch-and-learns) are one of the best ways to get a particular team or group together and run through the what and why of user research. You can work with them to work out what they’d like to see from you, and how you can help each other.

Tailor what you’re saying to different teams, especially if you’re talking to people with vastly different skill sets. For example, developers and designers are likely to see entirely different value in research.

Collect user insights across the organization

Putting together a comprehensive internal repository focused specifically on user research is another excellent way to grow awareness. It can also help to quantify things that may otherwise fall by the wayside. For example, you can measure the magnitude of certain pain points or observe patterns in feature requests. Using a platform like Notion or Confluence (or even Google Drive if you don’t want a dedicated platform), log all of your study notes, insights and research information that you find useful.

Whenever someone wants to learn more about research within the organization, they’ll be able to find everything easily.

Bring stakeholders along to research sessions

Getting a stakeholder along to a research session (usability tests and user interviews are great starting points) will help to show them the value that face-to-face sessions with users can provide.

To really involve an observer in your UX research, assign them a specific role. Note taker, for example. With a short briefing on best-practices for note taking, they can get a feel for what’s like to do some of the work you do.

You may also want to consider bringing anyone who’s interested along to a research session, even if they’re just there to observe.

Share your findings – consistently

Research is about more than just testing a hypothesis, it’s important to actually take your research back to the people who can action the data.

By sharing your research findings with teams and stakeholders regularly, your organization will start to build up an understanding of the value that ongoing research can provide, meaning getting approval to pursue research in future becomes easier. This is a bit of a chicken and egg situation, but it’s a practice that all researchers need to get into – especially those embedded in large teams or organizations.

Anything else you think is worth mentioning? Let us know in the comments.

Read more

Seeing is believing

Explore our tools and see how Optimal makes gathering insights simple, powerful, and impactful.