November 18, 2022
4 min

Moderated vs unmoderated research: which approach is best?

Knowing and understanding why and how your users use your product is invaluable for getting to the nitty gritty of usability. Delving deep with probing questions into motivation or skimming over looking for issues can equally be informative. 

Put super simply, usability testing literally is testing how usable your product is for your users. If your product isn’t usable users often won’t complete their task, let alone come back for more. No one wants to lose users before they even get started. Usability testing gets under their skin and really into the how, why and what they want (and equally what they don’t).

As we have been getting used to video calling regularly and using the internet for interactions, usability testing has followed suit. Being able to access participants remotely has allowed us to diversify the participant pool by not being restricted to those that are close enough to be in-person. This has also allowed an increase in the number of participants per test, as it becomes more cost-effective to perform remote usability testing.

But if we’re remote, does this mean it can’t be moderated? No - remote testing, along with modern technology, can mean that remote testing can be facilitated and moderated. But what is the best method - moderated or unmoderated?

What is moderated remote research testing? 🙋🏻

In traditional usability testing, moderated research is done in person. With the moderator and the participant in the same physical space. This, of course, allows for conversation and observational behavioral monitoring. Meaning the moderator can note not only what the participant answers but how and even make note of the body language, surroundings, and other influencing factors. 

This has also meant that traditionally, the participant pool has been limited to those that can be available (and close enough) to make it into a facility for testing. And being in person has meant it takes time (and money) to perform these tests.

As technology has moved along and the speed of internet connections and video calling has increased, this has opened up a world of opportunities for usability testing. Allowing usability testing to be done remotely. Moderators can now set up testing remotely and ‘dial in’ to observe participants anywhere they are. And potentially even running focus groups or other testing in a group format across the internet. 

Pros:

- In-depth gathering of insights through a back-and-forth conversation and observing of the participants.

- Follow-up questions don’t underestimate the value of being available to ask questions throughout the testing. And following up in the moment.

- Observational monitoring noticing and noting the environment and how the participants are behaving, can give more insight into how or why they choose to make a decision.

- Quick remote testing can be quicker to start, find participants, and complete than in-person. This is because you only need to set up a time to connect via the internet, rather than coordinating travel times, etc.

- Location (local and/or international) Testing online removes reliance on participants being physically present for the testing. This broadens your ability to broaden the pool, and participants can be either within your country or global. 

Cons:

- Time-consuming having to be present at each test takes time. As does analyzing the data and insights generated. But remember, this is quality data.

- Limited interactions with any remote testing there is only so much you can observe or understand across the window of a computer screen. It can be difficult to have a grasp on all the factors that might be influencing your participants.

What is unmoderated remote research testing? 😵💫

In its most simple sense, unmoderated user testing removes the ‘moderated’ part of the equation. Instead of having a facilitator guide participants through the test, participants are left to complete the testing by themselves and in their own time. For the most part, everything else stays the same. 

Removing the moderator, means that there isn’t anyone to respond to queries or issues in the moment. This can either delay, influence, or even potentially force participants to not complete or maybe not be as engaged as you may like. Unmoderated research testing suits a very simple and direct type of test. With clear instructions and no room for inference. 

Pros:

- Speed and turnaround,  as there is no need to schedule meetings with each and every participant. Unmoderated usability testing is usually much faster to initiate and complete.

- Size of study (participant numbers) unmoderated usability testing allows you to collect feedback from dozens or even hundreds of users at the same time. 


- Location (local and/or international) Testing online removes reliance on participants being physically present for the testing, which broadens your participant pool.  And unmoderated testing means that it literally can be anywhere while participants complete the test in their own time.

Cons:

- Follow-up questions as your participants are working on their own and in their own time, you can’t facilitate and ask questions in the moment. You may be able to ask limited follow-up questions.

- Products need to be simple to use unmoderated testing does not allow for prototypes or any product or site that needs guidance. 

- Low participant support without the moderator any issues with the test or the product can’t be picked up immediately and could influence the output of the test.

When should you do which? 🤔

Each moderated and unmoderated remote usability testing have its use and place in user research. It really depends on the question you are asking and what you are wanting to know.

Moderated testing allows you to gather in-depth insights, follow up with questions, and engage the participants in the moment. The facilitator has the ability to guide participants to what they want to know, to dig deeper, or even ask why at certain points. This method doesn’t need as much careful setup as the participants aren’t on their own. While this is all done online, it does still allow connection and conversation. This method allows for more investigative research. Looking at why users might prefer one prototype to another. Or possibly tree testing a new website navigation to understand where they might get lost and querying why the participant made certain choices.

Unmoderated testing, on the other hand, is literally leaving the participants to it. This method needs very careful planning and explaining upfront. The test needs to be able to be set and run without a moderator. This lends itself more to wanting to know a direct answer to a query. Such as a card sort on a website to understand how your users might sort information. Or a first click to see how/where users will click on a new website.

Wrap Up 🌯

With the ability to expand our pool of participants across the globe with all of the advances (and acceptance of) technology and video calling etc, the ability to expand our understanding of users’ experiences is growing. Remote usability testing is a great option when you want to gather information from users in the real world. Depending on your query, moderated or unmoderated usability testing will suit your study. As with all user testing, being prepared and planning ahead will allow you to make the most of your test.

Share this article
Author
Optimal
Workshop

Related articles

View all blog articles
Learn more
1 min read

The ultimate reading list for new user researchers

Having a library of user research books is invaluable. Whether you’re an old hand in the field of UX research or just dipping your toes in the water, being able to reference detailed information on methods, techniques and tools will make your life much easier.

There’s really no shortage of user research/UX reading lists online, so we wanted to do something a little different. We’ve broken our list up into sections to make finding the right book for a particular topic as easy as possible.

General user research guides

These books cover everything you need to know about a number of UX/user research topics. They’re great to have on your desk to refer back to – we certainly have them on the bookshelf here at Optimal Workshop.

Observing the User Experience: A Practitioner's Guide to User Research

Mike Kuniavsky

Observing the User Experience: A Practitioner’s Guide to User Research

This book covers 13 UX research techniques in a reference format. There’s a lot of detail, making it a useful resource for people new to the field and those who just need more clarification around a certain topic. There’s also a lot of practical information that you’ll find applicable in the real world. For example, information about how to work around research budgets and tight time constraints.

Just Enough Research

Erika Hall

Just Enough Research

In Just Enough Research, author Erika Hall explains that user research is something everyone can and should do. She covers several research methods, as well as things like how to identify your biases and make use of your findings. Designers are also likely to find this one quite useful, as she clearly covers the relationship between research and design.

Research Methods in Human-Computer Interaction

Harry Hochheiser, Jonathan Lazar, Jinjuan Heidi Feng

Research Methods in Human-Computer Interaction

Like Observing the User Experience, this is a dense guide – but it’s another essential one. Here, experts on human-computer interaction and usability explain different qualitative and quantitative research methods in an easily understandable format. There are also plenty of real examples to help frame your thinking around the usefulness of different research methods.

Information architecture

If you’re new to information architecture (IA), understanding why it’s such an important concept is a great place to start. There’s plenty of information online, but there are also several well-regarded books that make great starting points.

Information Architecture for the World Wide Web: Designing Large-Scale Web Sites

Peter Morville, Louis Rosenfeld

Information Architecture for the World Wide Web: Designing Large-Scale Web Site

You’ll probably hear this book referred to as “the polar bear book”, just because the cover features a polar bear. But beyond featuring a nice illustration of a bear, this book clearly covers the process of creating large websites that are both easy to navigate and appealing to use. It’s a useful book for designers, information architects and user researchers.

How to Make Sense of Any Mess

Abby Covert

How to Make Sense of Any Mess

This is a great introduction to information architecture and serves as a nice counter to the polar bear book, being much shorter and more easily digestible. Author Abby Covert explains complex concepts in a way anyone can understand and also includes a set of lessons and exercises with each chapter.

User interviews

For those new to the task, the prospect of interviewing users is always daunting. That makes having a useful guide that much more of a necessity!

Interviewing Users: How to Uncover Compelling Insights

Steve Portigal

Interviewing Users: How to Uncover Compelling Insights

While interviewing users may seem like something that doesn’t require a guide, an understanding of different interview techniques can go a long way. This book is essentially a practical guide to the art of interviewing users. Author Steve Portigal covers how to build rapport with your participants and the art of immersing yourself in how other people see the world – both key skills for interviewers!

Usability testing

Web usability is basically the ease of use of a website. It’s a broad topic, but there are a number of useful books that explain why it’s important and outline some of the key principles.

Don't Make Me Think: A Common Sense Approach to Web Usability

Steve Krug

Don't Make Me Think: A Common Sense Approach to Web Usability

Don’t Make Me Think is the first introduction to the world of UX and usability for many people, and for good reason – it’s a concise introduction to the topics and is easy to digest. Steve Krug explains some of the key principles of intuitive navigation and information architecture clearly and without overly technical language. In the latest edition, he’s updated the book to include mobile usability considerations.

As a testament to just how popular this book is, it was released in 2000 and has since had 2 editions and sold 400,000 copies.

Design

The design–research relationship is an important one, even if it’s often misunderstood. Thankfully, authors like Don Norman and Vijay Kumar are here to explain everything.

The Design of Everyday Things

Don Norman

The Design of Everyday Things

This book, by cognitive scientist and usability engineer Don Norman, explains how design is the communication between an object and its user, and how to improve this communication as a way of improving the user experience. If nothing else, this book will force you to take another look at the design of everyday objects and assess whether or not they’re truly user-friendly.

101 Design Methods: A Structured Approach for Driving Innovation in Your Organization

Vijay Kumar

101 Design Methods: A Structured Approach for Driving Innovation in Your Organization

A guidebook for innovation in the context of product development, this book approaches the subject in a slightly different way to many other books on the same subject. The focus here is that the practice of creating new products is actually a science – not an art. Vijay Kumar outlines practical methods and useful tools that researchers and designers can use to drive innovation, making this book useful for anyone involved in product development.

See our list on Goodreads

We've put together a list of all of the above books on Goodreads, which you can access here.

Further reading

For experienced practitioners and newcomers alike, user research can often seem like a minefield to navigate. It can be tricky to figure out which method to use when, whether you bring a stakeholder into your usability test (you should) and how much you should pay participants. Take a look at some of the other articles on our blog if you’d like to learn more.

Learn more
1 min read

Radical Collaboration: how teamwork really can make the dream work

Natalie and Lulu have forged a unique team culture that focuses on positive outputs (and outcomes) for their app’s growing user base. In doing so, they turned the traditional design approach on its head and created a dynamic and supportive team. 

Natalie, Director of Design at Hatch, and Lulu, UX Design Specialist, recently spoke at UX New Zealand, the leading UX and IA conference in New Zealand hosted by Optimal Workshop, on their concept of “radical collaboration”.

In their talk, Nat and Lulu share their experience of growing a small app into a big player in the finance sector, and their unique approach to teamwork and culture which helped achieve it.

Background on Natalie Ferguson and Lulu Pachuau

Over the last two decades, Lulu and Nat have delivered exceptional customer experiences for too many organizations to count. After Nat co-founded Hatch, she begged Lulu to join her on their audacious mission: To supercharge wealth building in NZ. Together, they created a design and product culture that inspired 180,000 Kiwi investors to join in just 4 years.

Contact Details:

Email: natalie@sixfold.co.nz

LinkedIn: https://www.linkedin.com/in/natalieferguson/ and https://www.linkedin.com/in/lulupach/

Radical Collaboration - How teamwork makes the dream work 💪💪💪

Nat and Lulu discuss how they nurtured a team culture of “radical collaboration” when growing the hugely popular app Hatch, based in New Zealand. Hatch allows everyday New Zealanders to quickly and easily trade in the U.S. share market. 

The beginning of the COVID pandemic spelled huge growth for Hatch and caused significant design challenges for the product. This growth meant that the app had to grow from a baby startup to one that could operate at scale - virtually overnight. 

In navigating this challenge, Nat and Lulu coined the term radical collaboration, which aims to “dismantle organizational walls and supercharge what teams achieve”. Radical collaboration has six key pillars, which they discuss alongside their experience at Hatch.

Pillar #1: When you live and breathe your North star

Listening to hundreds of their customers’ stories, combined with their own personal experiences with money, compelled Lulu and Nat to change how their users view money. And so, “Grow the wealth of New Zealanders” became a powerful mission statement, or North Star, for Hatch. The mission was to give people the confidence and the ability to live their own lives with financial freedom and control. Nat and Lulu express the importance of truly believing in the mission of your product, and how this can become a guiding light for any team. 

Pillar #2: When you trust each other so much, you’re happy to give up control

As Hatch grew rapidly, trusting each other became more and more important. Nat and Lulu state that sometimes you need to take a step back and stop fueling growth for growth’s sake. It was at this point that Nat asked Lulu to join the team, and Nat’s first request was for Lulu to be super critical about the product design to date - no feedback was out of bounds. Letting go, feeling uncomfortable, and trusting your team can be difficult, but sometimes it’s what you need in order to drag yourself out of status quo design. This resulted in a brief hiatus from frantic delivery to take stock and reprioritize what was important - something that can be difficult without heavy doses of trust!

Pillar #3: When everyone wears all the hats

During their journey, the team at Hatch heard lots of stories from their users. Many of these stories were heard during “Hatcheversery Calls”, where team members would call users on their sign-up anniversary to chat about their experience with the app. Some of these calls were inspiring, insightful, and heartwarming.

Everyone at Hatch made these calls – designers, writers, customer support, engineers, and even the CEO. Speaking to strangers in this way was a challenge for some, especially since it was common to field technical questions about the business. Nevertheless, asking staff to wear many hats like this turned the entire team into researchers and analysts. By forcing ourselves and our team outside of our comfort zone, we forced each other to see the whole picture of the business, not just our own little piece.

Pillar #4: When you do what’s right, not what’s glam

In an increasingly competitive industry, designers and developers are often tempted to consistently deliver new and exciting features. In response to rapid growth, rather than adding more features to the app, Lulu and Nat made a conscious effort to really listen to their customers to understand what problems they needed solving. 

As it turned out, filing overseas tax returns was a significant and common problem for their customers - it was difficult and expensive. So, the team at Hatch devised a tax solution. This solution was developed by the entire team, with almost no tax specialists involved until the very end! This process was far from glamorous and it often fell outside of standard job descriptions. However, the team eventually succeeded in simplifying a notoriously difficult process and saved their customers a massive headache.

Pillar #5: When you own the outcome, not your output.

Over time Hatch’s user base changed from being primarily confident, seasoned investors, to being first-time investors. This new user group was typically scared of investing and often felt that it was only a thing wealthy people did.

At this point, Hatch felt it was necessary to take a step back from delivering updates to take stock of their new position. This meant deeply understanding their customers’ journey from signing up, to making their first trade. Once this was intimately understood, the team delivered a comprehensive onboarding process which increased the sign-up conversion rate by 10%!

Pillar #6: When you’re relentlessly committed to making it work

Nat and Lulu describe a moment when Allbirds wanted to work with Hatch to allow ordinary New Zealanders to be involved in their IPO launch on the New York stock exchange. Again, this task faced numerous tax and trade law challenges, and offering the service seemed like yet another insurmountable task. The team at Hatch nearly gave up several times during this project, but everyone was determined to get this feature across the line – and they did. As a result, New Zealanders were some of the few regular investors from outside the U.S that were able to take part in Albirds IPO. 

Why it matters 💥

Over four years, Hatch grew to 180,000 users who collectively invested over $1bn. Nat and Lulu’s success underscores the critical role of teamwork and collaboration in achieving exceptional user experiences. Product teams should remember that in the rapidly evolving tech industry, it's not just about delivering the latest features; it's about fostering a positive and supportive team culture that buys into the bigger picture.

The Hatch team grew to be more than team members and technical experts. They grew in confidence and appreciated every moving part of the business. Product teams can draw inspiration from Hatch's journey, where designers, writers, engineers, and even the CEO actively engaged with users, challenged traditional design decisions, and prioritized solving actual user problems. This approach led to better, more user-centric outcomes and a deep understanding of the end-to-end user experience.

Most importantly, through the good times and tough, the team grew to trust each other. The mission weaved its way through each member of the team, which ultimately manifested in positive outcomes for the user and the business.

Nat and Lulu’s concept of radical collaboration led to several positive outcomes for Hatch:

  • It changed the way they did business. Information was no longer held in the minds of a few individuals – instead, it was shared. People were able to step into other people's roles seamlessly. 
  • Hatch achieved better results faster by focusing on the end-to-end experience of the app, rather than by adding successive features. 
  • The team became more nimble – potential design/development issues were anticipated earlier because everyone knew what the downstream impacts of a decision would be.

Over the next week, Lulu and Nat encourage designers and researchers to get outside of their comfort zone and:

  • Visit customer support team
  • Pick up the phone and call a customer
  • Challenge status quo design decisions. Ask, does this thing solve an end-user problem?

Learn more
1 min read

Efficient Research: Maximizing the ROI of Understanding Your Customers

Introduction

User research is invaluable, but in fast-paced environments, researchers often struggle with tight deadlines, limited resources, and the need to prove their impact. In our recent UX Insider webinar, Weidan Li, Senior UX Researcher at Seek, shared insights on Efficient Research—an approach that optimizes Speed, Quality, and Impact to maximize the return on investment (ROI) of understanding customers.

At the heart of this approach is the Efficient Research Framework, which balances these three critical factors:

  • Speed – Conducting research quickly without sacrificing key insights.
  • Quality – Ensuring rigor and reliability in findings.
  • Impact – Making sure research leads to meaningful business and product changes.

Within this framework, Weidan outlined nine tactics that help UX researchers work more effectively. Let’s dive in.

1. Time Allocation: Invest in What Matters Most

Not all research requires the same level of depth. Efficient researchers prioritize their time by categorizing projects based on urgency and impact:

  • High-stakes decisions (e.g., launching a new product) require deep research.
  • Routine optimizations (e.g., tweaking UI elements) can rely on quick testing methods.
  • Low-impact changes may not need research at all.

By allocating time wisely, researchers can avoid spending weeks on minor issues while ensuring critical decisions are well-informed.

2. Assistance of AI: Let Technology Handle the Heavy Lifting

AI is transforming UX research, enabling faster and more scalable insights. Weidan suggests using AI to:

  • Automate data analysis – AI can quickly analyze survey responses, transcripts, and usability test results.
  • Generate research summaries – Tools like ChatGPT can help synthesize findings into digestible insights.
  • Speed up recruitment – AI-powered platforms can help find and screen participants efficiently.

While AI can’t replace human judgment, it can free up researchers to focus on higher-value tasks like interpreting results and influencing strategy.

3. Collaboration: Make Research a Team Sport

Research has a greater impact when it’s embedded into the product development process. Weidan emphasizes:

  • Co-creating research plans with designers, PMs, and engineers to align on priorities.
  • Involving stakeholders in synthesis sessions so insights don’t sit in a report.
  • Encouraging non-researchers to run lightweight studies, such as A/B tests or quick usability checks.

When research is shared and collaborative, it leads to faster adoption of insights and stronger decision-making.

4. Prioritization: Focus on the Right Questions

With limited resources, researchers must choose their battles wisely. Weidan recommends using a prioritization framework to assess:

  • Business impact – Will this research influence a high-stakes decision?
  • User impact – Does it address a major pain point?
  • Feasibility – Can we conduct this research quickly and effectively?

By filtering out low-priority projects, researchers can avoid research for research’s sake and focus on what truly drives change.

5. Depth of Understanding: Go Beyond Surface-Level Insights

Speed is important, but efficient research isn’t about cutting corners. Weidan stresses that even quick studies should provide a deep understanding of users by:

  • Asking why, not just what – Observing behavior is useful, but uncovering motivations is key.
  • Using triangulation – Combining methods (e.g., usability tests + surveys) to validate findings.
  • Revisiting past research – Leveraging existing insights instead of starting from scratch.

Balancing speed with depth ensures research is not just fast, but meaningful.

6. Anticipation: Stay Ahead of Research Needs

Proactive researchers don’t wait for stakeholders to request studies—they anticipate needs and set up research ahead of time. This means:

  • Building a research roadmap that aligns with upcoming product decisions.
  • Running continuous discovery research so teams have a backlog of insights to pull from.
  • Creating self-serve research repositories where teams can find relevant past studies.

By anticipating research needs, UX teams can reduce last-minute requests and deliver insights exactly when they’re needed.

7. Justification of Methodology: Explain Why Your Approach Works

Stakeholders may question research methods, especially when they seem time-consuming or expensive. Weidan highlights the importance of educating teams on why specific methods are used:

  • Clearly explain why qualitative research is needed when stakeholders push for just numbers.
  • Show real-world examples of how past research has led to business success.
  • Provide a trade-off analysis (e.g., “This method is faster but provides less depth”) to help teams make informed choices.

A well-justified approach ensures research is respected and acted upon.

8. Individual Engagement: Tailor Research Communication to Your Audience

Not all stakeholders consume research the same way. Weidan recommends adapting insights to fit different audiences:

  • Executives – Focus on high-level impact and key takeaways.
  • Product teams – Provide actionable recommendations tied to specific features.
  • Designers & Engineers – Share usability findings with video clips or screenshots.

By delivering insights in the right format, researchers increase the likelihood of stakeholder buy-in and action.

9. Business Actions: Ensure Research Leads to Real Change

The ultimate goal of research is not just understanding users—but driving business decisions. To ensure research leads to action:

  • Follow up on implementation – Track whether teams apply the insights.
  • Tie findings to key metrics – Show how research affects conversion rates, retention, or engagement.
  • Advocate for iterative research – Encourage teams to re-test and refine based on new data.

Research is most valuable when it translates into real business outcomes.

Final Thoughts: Research That Moves the Needle

Efficient research is not just about doing more, faster—it’s about balancing speed, quality, and impact to maximize its influence. Weidan’s nine tactics help UX researchers work smarter by:


✔️  Prioritizing high-impact work
✔️  Leveraging AI and collaboration
✔️  Communicating research in a way that drives action

By adopting these strategies, UX teams can ensure their research is not just insightful, but transformational.

Watch the full webinar here

Seeing is believing

Explore our tools and see how Optimal makes gathering insights simple, powerful, and impactful.