What gear do I need for qualitative user testing?

"Dear Optimal Workshop

How do I quantify the ROI [return on investment] of investing in user experience?"

— Brian

‍

Dear Brian,

I'm going to answer your question with a resounding 'It depends'. I believe we all differ in what we're willing to invest, and what we expect to receive in return. So to start with, and if  you haven’t already, it's worth grabbing your stationery tools of choice and brainstorming your way to a definition of ROI that works for you, or for the people you work for.

‍

I personally define investment in UX as time given, money spent, and people utilized. And I define return on UX as time saved, money made, and people engaged. Oh, would you look at that — they’re the same! All three (time, money, and humans) exist on both sides of the ROI fence and are intrinsically linked. You can’t engage people if you don’t first devote time and money to utilizing your people in the best possible way! Does that make sense?

‍

That’s just my definition — you might have a completely different way of counting those beans, and the organizations you work for may think differently again.

‍

I'll share my thoughts on the things that are worth quantifying (that you could start measuring today if you were so inclined) and a few tips for doing so. And I'll point you towards useful resources to help with the nitty-gritty, dollars-and-cents calculations.

‍

5 things worth quantifying for digital design projects

‍

Here are five things I think are worthy of your attention when it comes to measuring the ROI of user experience, but there are plenty of others. And different projects will most likely call for different things.

‍

(A quick note: There's a lot more to UX than just digital experiences, but because I don't know your specifics Brian, the ideas I share below apply mainly to digital products.)

‍

1. What’s happening in the call centre?

‍

A surefire way to get a feel for the lay of the land is to look at customer support — and if measuring support metrics isn't on your UX table yet, it's time to invite it to dinner. These general metrics are an important part of an ongoing, iterative design process, but getting specific about the best data to gather for individual projects will give you the most usable data.

‍

Improving an application process on your website? Get hard numbers from the previous month on how many customers are asking for help with it, go away and do your magic, get the same numbers a month after launch, and you've got yourself compelling ROI data.

‍

Are your support teams bombarded with calls and emails? Has the volume of requests increased or decreased since you released that new tool, product, or feature? Are there patterns within those requests — multiple people with the same issues? These are just a few questions you can get answers to.

‍

You'll find a few great resources on this topic online, including this piece by Marko Nemberg that gives you an idea of the effects a big change in your product can have on support activity.

‍

2. Navigation vs. Search

‍

This is a good one: check your analytics to see if your users are searching or navigating. I’ve heard plenty of users say to me upfront that they'll always just type in the search bar and that they’d never ever navigate. Funny thing is, ten minutes later I see the same users naturally navigating their way to those gorgeous red patent leather pumps. Why?

‍

Because as Zoltán Gócza explains in UX Myth #16, people do tend to scan for trigger words to help them navigate, and resort to problem solving behaviour (like searching) when they can’t find what they need. Cue frustration, and the potential for a pretty poor user experience that might just send customers running for the hills — or to your competitors. This research is worth exploring in more depth, so check out this article by Jared Spool, and this one by Jakob Nielsen (you know you can't go wrong with those two).

‍

3. Are people actually completing tasks?

‍

Task completion really is a fundamental UX metric, otherwise why are we sitting here?! We definitely need to find out if people who visit our website are able to do what they came for.

‍

For ideas on measuring this, I've found the Government Service Design Manual by GOV.UK to be an excellent resource regardless of where you are or where you work, and in relation to task completion they say:

‍

"When users are unable to complete a digital transaction, they can be pushed to use other channels. This leads to low levels of digital take-up and customer satisfaction, and a higher cost per transaction."

‍

That 'higher cost per transaction' is your kicker when it comes to ROI.

‍

So, how does GOV.UK suggest we quantify task completion? They offer a simple (ish) recommendation to measure the completion rate of the end-to-end process by going into your analytics and dividing the number of completed processes by the number of started processes.

‍

While you're at it, check the time it takes for people to complete tasks as well. It could help you to uncover a whole host of other issues that may have gone unnoticed. To quantify this, start looking into what Kim Oslob on UXMatters calls 'Effectiveness and Efficiency ratios'. Effectiveness ratios can be determined by looking at success, error, abandonment, and timeout rates. And Efficiency ratios can be determined by looking at average clicks per task, average time taken per task, and unique page views per task.

‍

You do need to be careful not to make assumptions based on this kind of data— it can't tell you what people were intending to do. If a task is taking people too long, it may be because it’s too complicated ... or because a few people made themselves coffee in between clicks. So supplement these metrics with other research that does tell you about intentions.

‍

4. Where are they clicking first?

‍

A good user experience is one that gets out of bed on the right side. First clicks matter for a good user experience.

‍

A 2009 study showed that in task-based user tests, people who got their first click right were around twice as likely to complete the task successfully than if they got their first click wrong. This year, researchers at Optimal Workshop followed this up by analyzing data from millions of completed Treejack tasks, and found that people who got their first click right were around three times as likely to get the task right.

‍

That's data worth paying attention to.

‍

So, how to measure? You can use software that records mouse clicks first clicks from analytics on your page, but it difficult to measure a visitor's intention without asking them outright, so I'd say task-based user tests are your best bet.

‍

For in-person research sessions, make gathering first-click data a priority, and come up with a consistent way to measure it (a column on a spreadsheet, for example). For remote research, check out Chalkmark (a tool devoted exclusively to gathering quantitative, first-click data on screenshots and wireframes of your designs) and UserTesting.com (for videos of people completing tasks on your live website).

‍

5. Resources to help you with the number crunching

‍

Here's a great piece on uxmastery.com about calculating the ROI of UX.

‍

Here's Jakob Nielsen in 1999 with a simple 'Assumptions for Productivity Calculation', and here's an overview of what's in the 4th edition of NN/G's Return on Investment for Usability report (worth the money for sure).

‍

Here's a calculator from Write Limited on measuring the cost of unclear communication within organizations (which could quite easily be applied to UX).

‍

And here's a unique take on what numbers to crunch from Harvard Business Review.

‍

I hope you find this as a helpful starting point Brian, and please do have a think about what I said about defining ROI. I’m curious to know how everyone else defines and measures ROI — let me know!

‍

UX, product and design teams face growing challenges with tool proliferation, relying on different options for surveys, usability testing, and participant recruitment before transferring data into analysis tools like Dovetail. This fragmented workflow creates significant data integration issues and reporting bottlenecks that slow down teams trying to conduct smart, fast UX research. The constant switching between platforms not only wastes time but also increases the risk of data loss and inconsistencies across research projects. Optimal addresses these operational challenges by unifying the entire research workflow within a single platform, enabling teams to recruit participants, run tests and studies, and perform analysis without the complexity of managing multiple tools.

‍

Why Choose Optimal over Dovetail? 

‍

Fragmented Workflow vs. Unified Research Operations

‍

• Dovetail's Tool Chain Complexity: Dovetail requires teams to coordinate multiple platforms—one for recruitment, another for surveys, a third for usability testing—then import everything for analysis, creating workflow bottlenecks and coordination overhead.
• Optimal's Streamlined Workflow: Optimal eliminates tool chain management by providing recruitment, testing, and analysis in one platform, enabling researchers to move seamlessly from study design to actionable insights.
• Context Switching Inefficiency: Dovetail users constantly switch between different tools with different interfaces, learning curves, and data formats, fragmenting focus and slowing research velocity.
• Focused Research Flow: Optimal's unified interface keeps researchers in flow state, moving efficiently through research phases without context switching or tool coordination.

‍

Data Silos vs. Integrated Intelligence

‍

• Fragmented Data Sources: Dovetail aggregates data from multiple external sources, but this fragmentation can create inconsistencies, data quality issues, and gaps in analysis that compromise insight reliability.
• Consistent Data Standards: Optimal's unified platform ensures consistent data collection standards, formatting, and quality controls across all research methods, delivering reliable insights from integrated data sources.
• Manual Data Coordination: Dovetail teams spend significant time importing, formatting, and reconciling data from different tools before analysis can begin, delaying insight delivery and increasing error risk.
• Automated Data Integration: Optimal automatically captures and integrates data across all research activities, enabling real-time analysis and immediate insight generation without manual data management.

‍

Limited Data Collection vs. Global Research Capabilities

‍

• No Native Recruitment: Dovetail's beta participant recruitment add-on lacks the scale and reliability enterprise teams need, forcing dependence on external recruitment services with additional costs and complexity.
• Global Participant Network: Optimal's 200+ million verified participants across 150+ countries provide comprehensive recruitment capabilities with advanced targeting and quality assurance for any research requirement.
• Analysis-Only Value: Dovetail's value depends entirely on research volume from external sources, making ROI uncertain for teams with moderate research needs or budget constraints.
• Complete Research ROI: Optimal delivers immediate value through integrated data collection and analysis capabilities, ensuring consistent ROI regardless of external research dependencies.

‍

Dovetail Challenges: 

‍

Dovetail may slow teams because of challenges with: 

‍

• Multi-tool coordination requiring significant project management overhead
• Data fragmentation creating inconsistencies and quality control challenges
• Context switching between platforms disrupting research flow and focus
• Manual data import and formatting delaying insight delivery
• Complex tool chain management requiring specialized technical knowledge

‍

When Optimal is the Right Choice

‍

Optimal becomes essential for:

‍

• Streamlined Workflows: Teams needing efficient research operations without tool coordination overhead
• Research Velocity: Projects requiring rapid iteration from hypothesis to validated insights
• Data Consistency: Studies where integrated data standards ensure reliable analysis and conclusions
• Focus and Flow: Researchers who need to maintain deep focus without platform switching
• Immediate Insights: Teams requiring real-time analysis and instant insight generation
• Resource Efficiency: Organizations wanting to maximize researcher productivity and minimize tool management

‍

Ready to move beyond basic feedback to strategic research intelligence? Experience how Optimal's analytical depth and comprehensive insights drive product decisions that create competitive advantage.

Summary: The equipment and tools you use to run your user testing sessions can make your life a lot easier. Here’s a quick guide.

‍

It’s that time again. You’ve done the initial scoping, development and internal testing, and now you need to take the prototype of your new design and get some qualitative data on how it works and what needs to be improved before release. It’s time for the user testing to begin.

‍

But the prospect of user testing raises an important question, and it’s one that many new user researchers often deliberate over: What gear or equipment should I take with me? Well, never fear. We’re going to break down everything you need to consider in terms of equipment, from video recording through to qualitative note-taking.

‍

Recording: Audio, screens and video

‍

The ability to easily record usability tests and user interviews means that even if you miss something important during a session, you can go back later and see what you’ve missed. There are 3 types of recording to keep in mind when it comes to user research: audio, video and screen recording. Below, we’ve put together a list of how you can capture each. You shouldn’t have to buy any expensive gear – free alternatives and software you can run on your phone and laptop should suffice.

‍

• Audio – Forget dedicated sound recorders; recording apps for smartphones (iOS and Android) allow you to record user interviews and usability tests with ease and upload the recordings to Google Drive or your computer. Good options include Sony’s recording app for Android and the built-in Apple recording app on iOS.
• Transcription – Once you’ve created a recording, you’ll no doubt want a text copy to work with. For this, you’ll need transcription software to take the audio and turn it into text. There are companies that will make transcriptions for you, but software like Transcribe means you can carry out the process yourself.
• Screen recording – Very useful during remote usability tests, screen recording software can show you exactly how participants react to the tasks you set out for them, even if you’re not in the room. OBS Studio is a good option for both Mac and Windows users. You can also use Quicktime (free) if you’re running the test in person.
• Video – Recording your participants as they make their way through the various tasks in a usability test can provide useful reference material at the end of your testing sessions. You can refer back to specific points in a video to capture any detail you may have missed, and you can share video with stakeholders to demonstrate a point. If you don’t have access to a dedicated camera, consider mounting your smartphone on a tripod and recording that way.

‍

Taking (and making use of) notes

‍

Notetaking and qualitative user testing go hand in hand. For most user researchers, notetaking during a research session means busting out the Post-it notes and Sharpie pens, rushing to take down every observation and insight and then having to arduously transcribe these notes after the session – or spend hours in workshops trying to identify themes and patterns. This approach still has merit, as it’s often one of the best ways to get people who aren’t too familiar with user research involved in the process. With physical notes, you can gather people around a whiteboard and discuss what you’re looking at. What’s more, you can get them to engage with the material directly.

‍

But there are digital alternatives. Qualitative notetaking software (like our very own Reframer) means you can bring a laptop into a user interview and take down observations directly in a secure environment. Even better, you can ask someone else to sit in as your notetaker, freeing you up to focus on running the session. Then, once you’ve run your tests, you can use the software for theme and pattern analysis, instead of having to schedule yet another full day workshop.

‍

Scheduling your user tests

‍

Ah, participant scheduling. Perhaps one of the most time-consuming parts of the user testing process. Thankfully, software can drastically reduce the logistical burden.

Here are some useful pieces of software:

‍

Dedicated scheduling tool Calendly is one of the most popular options for participant scheduling in the UX community. It’s really hands-off, in that you basically let the tool know when you’re available, share the Calendly link with your prospective participants, and then they select a time (from your available slots) that works for them. There are also a host of other useful features that make it a popular option for researchers, like integrations and smart timezones.

‍

If you’re already using the Optimal Workshop platform, you can use our  survey tool Questions as a fairly robust scheduling tool. Simply set up a study and add in prospective time slots. You can then use the multi-choice field option to have people select when they’re available to attend. You can also capture other data and avoid the usual email back and forth.

‍

Storing your findings
‍

One of the biggest challenges for user researchers is effectively storing and cataloging all of the research data that they start to build up. Whether it’s video recordings of usability tests, audio recordings or even transcripts of user interviews, you need to ensure that your data is A) easily accessible after the fact, and B) stored securely to ensure you’re protecting your participants.

‍

Here are some things to ask yourself when you store any piece of customer or user data:

‍

• Who will have access to this data?
• How long do I plan to keep this data?
• Will this data be anonymized?
• If I’m keeping physical data on hand, where will it be stored?

‍

Don’t make the mistake of thinking user data is ‘secure enough’, whether that’s on a company server that anyone can access, or even in an unlocked filing cabinet beneath your desk. Data privacy and security should always be at the top of your list of considerations. We won’t dive into best practices for participant data protection in this article, but instead, just mention that you need to be vigilant. Wherever you end up storing information, make sure you understand who has access.

‍

Wrap up

‍

Hopefully, this guide has given you an overview of some of the tools and software you can use before you start your next user test. We’ve also got a number of other interesting articles that you can read right here on our blog.