October 3, 2022

Unmoderated usability testing: a checklist

In-person moderated user testing is a valuable part of any research project. Meaning you can see first-hand how your users interact with your prototypes and products. But in-person isn’t always a viable option. What to do if your project needs user testing but it’s just not possible to get in front of your users personally? 

Let’s talk unmoderated user testing. This approach sidesteps the need to meet your participants face-to-face as it’s done entirely remotely, over the internet. By it’s very nature there are also considerable benefits to unmoderated user testing.

What is unmoderated user testing? 💻👀

In the most basic sense, unmoderated user testing removes the ‘moderated’ part of the equation. Instead of having a facilitator guide participants through the test, participants complete the testing activity by themselves and in their own time. For the most part, everything else stays the same.

The key differences are:

  • You can’t ask follow-up questions
  • You can’t use low-fidelity prototypes
  • You can’t support participants (beyond the initial instructions you send them).

Is unmoderated user testing right for your research project?

By nature, unmoderated user research does not include any direct interaction between the researcher and the study participants. This is really the biggest benefit and also the biggest drawback. 

Benefits of unmoderated usability testing 👩🏻💻

  • Speed and turnaround  - As there is no need to schedule meetings with each participant, unmoderated testing is usually much faster to initiate and complete. Depending on the study, it may be possible to launch a study and receive results in just a few hours.
  • Size of study (participant numbers) - Unmoderated user testing also allows you to collect feedback from dozens or even hundreds of users at the same time.
  • Location (local and/or international) -Testing online removes reliance on participants being physically present for the testing which broadens the ability to make contact with participants within your country or across the globe. 

If you’d like to know more about the benefits of unmoderated usability testing, take a look at our article five reasons you should consider unmoderated user testing.

Limitations of unmoderated usability testing 🚧

  • Early-prototype testing is difficult without a moderator to explain and help participants recover from errors or limitations of the prototype.
  • Participant behavior - Without a moderator, participants tend to be less engaged and behave less realistically in tasks that depend on imagination, decision-making, or emotional responses.
  • Inability to ask follow-up questions - by not being in the testing with the participant, the facilitator can’t ask further questions to get a deeper understanding of the participant’s reasoning. As you can’t rely on human judgment through a moderator being in the room with the participants and the ability to adjust the test in the moment, unmoderated usability testing needs thorough up front planning.

Because of these limitations unmoderated usability testing usually works best for evaluating live websites and apps or highly functional prototypes.  It’s great for testing activities that don’t require a lot of imagination or emotion from participants. Such as testing functionality or answering direct queries to do with your product.

What’s involved when setting up unmoderated usability testing? 🤔💭

  1. Define testing goals

With any usability testing, it pays to define your goals before getting underway with setting up the software. What do you want to know from the participants? Goals vary from test to test. Understanding your goals upfront will help you to make the correct tool choice.


  1. Define your demographic

With a clear understanding of your goal, now it’s time to consider which participants are right for your study. Think about who they are, their demographic, and where they live. Are they new users or existing? Are they experts or novices?

  1. Selecting testing software

As unmoderated studies, are done remotely, the software used to faciliate the study plays a key role in ensuring you get useful results. Without a facilitator, the software must guide the participants through the session and record what happens. Take the time to test software and select one that is right for your study.

  1. Write your own tasks and questions

Think through your goals and what you want to achieve from the testing. Many of the unmoderated testing services include study templates with generic example tasks. Remember they are templates, and your tasks and questions should be specific to your particular study. Any task instructions guiding the participants should be clear and directive.

  1. Trial session

You’ve done all of the upfront work, now it’s time to test that it works, the software does what you expect and the instructions you have written can be followed. Doing a test run is crucial, especially with unmoderated usability testing, as there won’t be a facilitator in the testing to fix any problems.

  1. Recruit participants

Having defined your target audience and demographic, now is the time to recruit participants. Ensuring you have some control over the recruitment process is important, either through screening questions or recruiting your own. There are services that  recruit from a pool of willing participants. Thiscan be a great way to get a wide range of users.

  1. Analyze results

You are likely to accumulate a lot of data from your unmoderated testing. You’ll need a way to organize and analyze the data to derive insights that are valuable. Depending on the type of usability testing you do will vary the type of results. Quantitative testing gives data-driven results and direct answers. Whereas qualitative testing through audio or video recordings of participants’ actions or comments will need time to analyze and look at behavioral observations. 

Wrap Up 🌯

Unmoderated usability testing can be a good option for your study. It may not be right for all of your studies all of the time. While it can be quick to implement and often cheaper than moderated usability testing, it still requires time and planning to ensure you get the data insights you are looking for. Following a checklist can be a great way to ensure you approach your research methodically.

Share this article
Author
Optimal
Workshop

Related articles

View all blog articles
Learn more
1 min read

Different ways to test information architecture

We all know that building a robust information architecture (IA) can make or break your product. And getting it right can rely on robust user research. Especially when it comes to creating human-centered, intuitive products that deliver outstanding user experiences.

But what are the best methods to test your information architecture? To make sure that your focus is on building an information architecture that is truly based on what your users want, and need.

What is user research? 🗣️🧑🏻💻

With all the will in the world, your product (or website or mobile app) may work perfectly and be as intuitive as possible. But, if it is only built on information from your internal organizational perspective, it may not measure up in the eyes of your user. Often, organizations make major design decisions without fully considering their users. User research (UX) backs up decisions with data, helping to make sure that design decisions are strategic decisions. 

Testing your information architecture can also help establish the structure for a better product from the ground up. And ultimately, the performance of your product. User experience research focuses your design on understanding your user expectations, behaviors, needs, and motivations. It is an essential part of creating, building, and maintaining great products. 

Taking the time to understand your users through research can be incredibly rewarding with the insights and data-backed information that can alter your product for the better. But what are the key user research methods for your information architecture? Let’s take a look.

Research methods for information architecture ⚒️

There is more than one way to test your IA. And testing with one method is good, but with more than one is even better. And, of course, the more often you test, especially when there are major additions or changes, you can tweak and update your IA to improve and delight your user’s experience.

Card Sorting 🃏

Card sorting is a user research method that allows you to discover how users understand and categorize information. It’s particularly useful when you are starting the planning process of your information architecture or at any stage you notice issues or are making changes. Putting the power into your users’ hands and asking how they would intuitively sort the information. In a card sort, participants sort cards containing different items into labeled groups. You can use the results of a card sort to figure out how to group and label the information in a way that makes the most sense to your audience. 

There are a number of techniques and methods that can be applied to a card sort. Take a look here if you’d like to know more.

Card sorting has many applications. It’s as useful for figuring out how content should be grouped on a website or in an app as it is for figuring out how to arrange the items in a retail store.You can also run a card sort in person, using physical cards, or remotely with online tools such as OptimalSort.

Tree Testing 🌲

Taking a look at your information architecture from the other side can also be valuable. Tree testing is a usability method for evaluating the findability of topics on a product. Testing is done on a simplified text version of your site structure without the influence of navigation aids and visual design.

Tree testing tells you how easily people can find information on your product and exactly where people get lost. Your users rely on your information architecture – how you label and organize your content – to get things done.

Tree testing can answer questions like:

  • Do my labels make sense to people?
  • Is my content grouped logically to people?
  • Can people find the information they want easily and quickly? If not, what’s stopping them?

Treejack is our tree testing tool and is designed to make it easy to test your information architecture. Running a tree test isn’t actually that difficult, especially if you’re using the right tool. You’ll  learn how to set useful objectives, how to build your tree, write your tasks, recruit participants, and measure results.

Combining information architecture research methods 🏗

If you are wanting a fully rounded view of your information architecture, it can be useful to combine your research methods.

Tree testing and card sorting, along with usability testing, can give you insights into your users and audience. How do they think? How do they find their way through your product? And how do they want to see things labeled, organized, and sorted? 

If you want to get fully into the comparison of tree testing and card sorting, take a look at our article here, which compares the options and explains which is best and when. 

Learn more
1 min read

Why information architecture is important for designers

Sitting inside any beautifully crafted and designed digital product, there must be a fully functional and considered information architecture.

As much as information architecture shouldn’t be developed in a vacuum. Neither should the design and look of digital products. In fact, a large proportion of the function of digital designers is devoted to supporting users locating content they need and driving them towards content that the product owners want them to find.

Incorporating visual markers to make sure that certain content is distinct from the rest or creating layers that demonstrate the diverse content on a product.

If you do not have quality content, it is impossible to design a quality digital product. It all comes back to creating a user experience that makes sense and is designed to make task completion simple. And this relates back to designing the product with the content planned for it in mind.

8 Principles of information architecture, according to Dan Brown 🏗️

As a designer, the more you know about information architecture, the better the products you design will meet your user requirements and deliver what they need. If you work with an information architect, even better. If you’re still learning about information architecture the 8 Principles according to Dan Brown is a great place to begin.

If you haven’t come across Dan Brown yet, you have more than likely come across his 8 principles. Dan Brown is one of the UX world's most prolific experts with a career that spans most areas of UX designs. He’s written 3 books on the subject and experience across a multitude of high profile projects. Aiding large organizations to make the most of their user experience.

  1. The principle of objects: Content should be treated as a living, breathing thing. It has lifecycles, behaviors, and attributes.
  2. The principle of choices: Less is more. Keep the number of choices to a minimum.
  3. The principle of disclosure: Show a preview of information that will help users understand what kind of information is hidden if they dig deeper.
  4. The principle of examples: Show examples of content when describing the content of the categories.
  5. The principle of front doors: Assume that at least 50% of users will use a different entry point than the home page.
  6. The principle of multiple classifications: Offer users several different classification schemes to browse the site’s content.
  7. The principle of focused navigation: Keep navigation simple and never mix different things.
  8. The principle of growth: Assume that the content on the website will grow. Make sure the website is scalable.

It’s highly likely that you’ve already used some, or all, of these IA principles in your designs. Don’t be shy about mastering them, or at the very least be familiar. They can only help you become a better user experience designer.

Wrap up 🌯

Mastering the 8 principles, according to IA expert Dan Brown will see you mastering the complex tasks of information architecture. Understanding IA is key to creating digital designs with a content structure that is functional, logical and just what your users need to navigate your product. Design without good IA doesn’t work as well, just as a content structure without a well designed interface will not engage users.

Learn more
1 min read

Information architecture lives here : make IA part of your digital space

There is an enormous amount of information on the internet. Every page you load is crammed with it. With so much information around, it’s incredibly important to remember that for human minds to be able to actually process and make sense of it, considered, strategic, ordered information architecture is needed.

Information architecture is everywhere digitally, but where does it actually live? IA sits in behind all of the digital interfaces we humans use (and even out in the real world). It ideally has been considered, researched, and implemented with humans in mind, though as we all know, this isn’t always the case.  

Information architecture always has a role to play, and here we’ll focus on 4 key digital spaces where information architecture is most prominent, and likely most familiar.

  1. Websites
  2. Apps
  3. Company intranet
  4. Social media software

To find out more about putting information architecture into action check out our blog.

1. Websites 🌐

Information architecture for websites can often be confused with navigation. While site navigation is super important, (think of it as the ‘skin’ that sits above the information), that’s only part of the story. Great website architecture considers and organizes user requirements, organizing the content and structure of the website with labels, search and navigation that simply makes sense and is easy to use. Done well, it will  feel simple and straightforward, guiding users through smoothly.

Hubspot has a great article on IA for websites and how they can be structured. From a complex library system, to travel booking systems. Each has its own needs and can be quite deep in layers of information. The trick is to create a website structure, based on user needs, that makes sense of the piles of information and a simple navigation that sits above, making the journey to task completion as quick as possible. Working together they make any user experience feel quick, simple, and intuitive! 

2. Apps 📱

An organization's app acts as the first point of contact. It needs to be super simple, clean, and quick to interact with. A well thought out, thorough, researched, and organized information architecture plays a big part in this.

Information architecture for mobile use has a different set of rules than websites. The key consideration here is around the ease of use across a smaller screen. Navigation that makes sense for a laptop or desktop computer can be clunky in a mobile app.

An app's interface needs to have fewer options, and fewer clicks to complete the task. Researching and designing an app’s IA with just the right amount of information is key. Some retail apps are simplified websites, whereas other apps exist on their own merits, with no need for a website.

Bringing it back to the humans that will interact with the app is key to creating a product that delivers on user requirements and increases interaction. 

3. Intranet 🖥

An organizational intranet is possibly less of a priority than a website or an app but is vitally important to the success of an organization. And in these times of remote or hybrid working, intranets have proven to be more valuable than ever. An intranet is more than where to stick newsletters for staff, it is an interface that can make or break the productivity or even the wellbeing of an organization. 

Access to files, information, messaging platforms, and corporate requirements wherever and whenever people are working is more important than ever.  A well researched and designed information architecture can build an intranet that meets users’ requirements, increase communication and interaction and ultimately boost productivity. 

Conducting UX research with staff on what they need to access, when, and how will help inform the intranet IA far more intuitively. The information available (and needed) can be huge, keeping it simple and human focused is key.

Have you ever thought about how poor internal information architecture might be hurting your business?

4. Social media 🤳

Social media software is complex in terms of ecosystem and display of information. Each social media platform has developed over time, think back to Facebook and how it looked when it initially launched. And through the uptake of users, gathering of information over time, and continual research and testing, it has evolved into what we see today. And will continue to evolve with users’ needs.

Every user has a different experience based on the individuals, groups, organizations and even retailers that they choose to interact with (or haven’t chosen via advertising). The piles of information that sit behind and are brought to the interface for an individual through their choice, associations, labels, tags, interests, age, etc will present them with a unique feed. 

The interface of a social media platform needs to be considered, tested, tested again, and occasionally tested or changed once launched as the interaction of users is vitally important.

The information architecture sitting behind is huge in order to enable the agility to pull the right information forward in a dynamic and coherent way. Continuously learning, testing and requesting interaction from users through options to ‘hide’ posts that aren’t appropriate or respond to direct queries about what they do and don’t want to see are just some instances of continuous user research.

Social media continues to be a sophisticated information architecture that is constantly updating and changing with user needs.

Wrap Up 🫔

Information architecture lives wherever there is information needing to be found by humans. Successful information architecture is sorted, organized and labeled in a way that is simple, intuitive, and considered. Making interaction and life simple, which in a world where there are an increasing number of websites, apps, and tools to choose from – intuitive information architecture has never been more important for your business and your customers.

Seeing is believing

Explore our tools and see how Optimal makes gathering insights simple, powerful, and impactful.