Moderated vs unmoderated research: which approach is best?

Return on investment (ROI) is often the term on everyone’s lips when starting a big project or even when reviewing a website. It’s especially popular with those that hold the purse strings.  As UX researchers it is important to consider the ROI of the work we do and understand how to measure this. 

‍

We’ve lined up 5 key ways to measure ROI for UX research to help you get the conversation underway with stakeholders so you can show real and tangible benefits to your organization. 

‍

‍

1. Meet and exceed user expectations

‍

Put simply, a product that meets and exceeds user expectations leads to increased revenue. When potential buyers are able to find and purchase what they’re looking for, easily, they’ll complete their purchase, and are far more likely to come back. The simple fact that users can finish their task will increase sales and improve overall customer satisfaction which has an influence on their loyalty. Repeat business means repeat sales. Means increased revenue.

‍

Creating, developing and maintaining a usable website is more important than you might think. And this is measurable! Tracking and analyzing website performance prior to the UX research and after can be insightful and directly influenced by changes made based on UX research.

‍

Measurable: review the website (product) performance prior to UX research and after changes have been made. The increase in clicks, completed tasks and/or baskets will tell the story.

‍

‍

2. Reduce development time

‍

UX research done at the initial stages of a project can lead to a reduction in development time of by 33% to 50%! And reduced time developing, means reduced costs (people and overheads) and a speedier to market date. What’s not to love? 

‍

Measurable: This one is a little more tricky as you have saved time (and cost) up front. Aiding in speed to market and performance prior to execution. Internal stakeholder research may be of value post the live date to understand how the project went.

‍

‍

3. Ongoing development costs

‍

And the double hitter? Creating a product that has the user in mind up front, reduces the need to rehash or revisit as quickly. Reducing ongoing costs. Early UX research can help with the detection of errors early on in the development process. Fixing errors after development costs a company up to 100 times more than dealing with the same error before development.

‍

Measureable: Again, as UX research has saved time and money up front this one can be difficult to track. Though depending on your organization and previous projects you could conduct internal research to understand how the project compares and the time and cost savings.

‍

‍

4. Meeting user requirements

‍

Did you know that 70% of projects fail due to the lack of user acceptance? This is often because project managers fail to understand the user requirements properly. Thanks to UX research early on, gaining insights into users and only spending time developing the functions users actually want, saving time and reducing development costs. Make sure you get confirmation on those requirements by iterative testing. As always, fail early, fail often. Robust testing up front means that in the end, you’ll have a product that will meet the needs of the user.

‍

Measurable: Where is the product currently? How does it perform? Set a benchmark up front and review post UX research. The deliverables should make the ROI obvious.

‍

‍

5. Investing in UX research leads to an essential competitive advantage.

‍

Thanks to UX research you can find out exactly what your customers want, need and expect from you. This gives you a competitive advantage over other companies in your market. But you should be aware that more and more companies are investing in UX while customers are ever more demanding, their expectations continue to grow and they don’t tolerate bad experiences. And going elsewhere is an easy decision to make.

‍

Measurable: Murky this one, but no less important. Knowing, understanding and responding to competitors can help keep you in the lead, and developing products that meet and exceed those user expectations.

‍

‍

Wrap up

‍

Showing the ROI on the work we do is an essential part of getting key stakeholders on board with our research. It can be challenging to talk the same language, ultimately we all want the same outcome…a product that works well for our users, and delivers additional revenue.

‍

For some continued reading (or watching in this case), Anna Bek, Product and Delivery Manager at Xplor explored the same concept of "How to measure experience" during her UX New Zealand 2020 – watch it here as she shares a perspective on UX ROI.

Introduction

‍

User Experience (UX) research has always been about people. How they think, how they behave, what they need, and—just as importantly—what they don’t yet realise they need. Traditional UX methodologies have long relied on direct human input: interviews, usability testing, surveys, and behavioral observation. The assumption was clear—if you want to understand people, you have to engage with real humans.

‍

But in 2025, that assumption is being challenged.

‍

The emergence of digital twins and synthetic users—AI-powered simulations of human behavior—is changing how researchers approach user insights. These technologies claim to solve persistent UX research problems: slow participant recruitment, small sample sizes, high costs, and research timelines that struggle to keep pace with product development. The promise is enticing: instantly accessible, infinitely scalable users who can test, interact, and generate feedback without the logistical headaches of working with real participants.

‍

Yet, as with any new technology, there are trade-offs. While digital twins may unlock efficiencies, they also raise important questions: Can they truly replicate human complexity? Where do they fit within existing research practices? What risks do they introduce?

‍

This article explores the evolving role of digital twins in UX research—where they excel, where they fall short, and what their rise means for the future of human-centered design.

‍

‍

The Traditional UX Research Model: Why Change?

‍

For decades, UX research has been grounded in methodologies that involve direct human participation. The core methods—usability testing, user interviews, ethnographic research, and behavioral analytics—have been refined to account for the unpredictability of human nature.

‍

This approach works well, but it has challenges:

‍

1. Participant recruitment is time-consuming. Finding the right users—especially niche audiences—can be a logistical hurdle, often requiring specialised panels, incentives, and scheduling gymnastics.
   ‍
2. Research is expensive. Incentives, moderation, analysis, and recruitment all add to the cost. A single usability study can run into tens of thousands of dollars.
   ‍
3. Small sample sizes create risk. Budget and timeline constraints often mean testing with small groups, leaving room for blind spots and bias.
   ‍
4. Long feedback loops slow decision-making. By the time research is completed, product teams may have already moved on, limiting its impact.

‍

In short: traditional UX research provides depth and authenticity, but it’s not always fast or scalable.

‍

Digital twins and synthetic users aim to change that.

‍

‍

What Are Digital Twins and Synthetic Users?

‍

While the terms digital twins and synthetic users are sometimes used interchangeably, they are distinct concepts.

‍

Digital Twins: Simulating Real-World Behavior

‍

A digital twin is a data-driven virtual representation of a real-world entity. Originally developed for industrial applications, digital twins replicate machines, environments, and human behavior in a digital space. They can be updated in real time using live data, allowing organisations to analyse scenarios, predict outcomes, and optimise performance.

‍

In UX research, human digital twins attempt to replicate real users' behavioral patterns, decision-making processes, and interactions. They draw on existing datasets to mirror real-world users dynamically, adapting based on real-time inputs.

‍

‍

Synthetic Users: AI-Generated Research Participants

‍

While a digital twin is a mirror of a real entity, a synthetic user is a fabricated research participant—a simulation that mimics human decision-making, behaviors, and responses. These AI-generated personas can be used in research scenarios to interact with products, answer questions, and simulate user journeys.

‍

Unlike traditional user personas (which are static profiles based on aggregated research), synthetic users are interactive and capable of generating dynamic feedback. They aren’t modeled after a specific real-world person, but rather a combination of user behaviors drawn from large datasets.

‍

Think of it this way:

‍

• A digital twin is a highly detailed, data-driven clone of a specific person, customer segment, or process.
• A synthetic user is a fictional but realistic simulation of a potential user, generated based on behavioral patterns and demographic characteristics.

‍

Both approaches are still evolving, but their potential applications in UX research are already taking shape.

‍

‍

Where Digital Twins and Synthetic Users Fit into UX Research

‍

‍

The appeal of AI-generated users is undeniable. They can:

‍

• Scale instantly – Test designs with thousands of simulated users, rather than just a handful of real participants.
• Eliminate recruitment bottlenecks – No need to chase down participants or schedule interviews.
• Reduce costs – No incentives, no travel, no last-minute no-shows.
• Enable rapid iteration – Get user insights in real time and adjust designs on the fly.
• Generate insights on sensitive topics – Synthetic users can explore scenarios that real participants might find too personal or intrusive.

‍

‍

These capabilities make digital twins particularly useful for:

‍

• Early-stage concept validation – Rapidly test ideas before committing to development.
• Edge case identification – Run simulations to explore rare but critical user scenarios.
• Pre-testing before live usability sessions – Identify glaring issues before investing in human research.

‍

However, digital twins and synthetic users are not a replacement for human research. Their effectiveness is limited in areas where emotional, cultural, and contextual factors play a major role.

‍

‍

The Risks and Limitations of AI-Driven UX Research

‍

For all their promise, digital twins and synthetic users introduce new challenges.

‍

1. They lack genuine emotional responses.
   AI can analyse sentiment, but it doesn’t feel frustration, delight, or confusion the way a human does. UX is often about unexpected moments—the frustrations, workarounds, and “aha” realisations that define real-world use.
   ‍
2. Bias is a real problem.
   AI models are trained on existing datasets, meaning they inherit and amplify biases in those datasets. If synthetic users are based on an incomplete or non-diverse dataset, the research insights they generate will be skewed.
   ‍
3. They struggle with novelty.
   Humans are unpredictable. They find unexpected uses for products, misunderstand instructions, and behave irrationally. AI models, no matter how advanced, can only predict behavior based on past patterns—not the unexpected ways real users might engage with a product.
   ‍
4. They require careful validation.
   How do we know that insights from digital twins align with real-world user behavior? Without rigorous validation against human data, there’s a risk of over-reliance on synthetic feedback that doesn’t reflect reality.

‍

‍

A Hybrid Future: AI + Human UX Research

‍

Rather than viewing digital twins as a replacement for human research, the best UX teams will integrate them as a complementary tool.

‍

Where AI Can Lead:

‍

• Large-scale pattern identification
• Early-stage usability evaluations
• Speeding up research cycles
• Automating repetitive testing

‍

Where Humans Remain Essential:

‍

• Understanding emotion, frustration, and delight
• Detecting unexpected behaviors
• Validating insights with real-world context
• Ethical considerations and cultural nuance

‍

The future of UX research is not about choosing between AI and human research—it’s about blending the strengths of both.

‍

‍

Final Thoughts: Proceeding With Caution and Curiosity

‍

Digital twins and synthetic users are exciting, but they are not a magic bullet. They cannot fully replace human users, and relying on them exclusively could lead to false confidence in flawed insights.

‍

Instead, UX researchers should view these technologies as powerful, but imperfect tools—best used in combination with traditional research methods.

‍

As with any new technology, thoughtful implementation is key. The real opportunity lies in designing research methodologies that harness the speed and scale of AI without losing the depth, nuance, and humanity that make UX research truly valuable.

‍

The challenge ahead isn’t about choosing between human or synthetic research. It’s about finding the right balance—one that keeps user experience truly human-centered, even in an AI-driven world.

‍

This article was researched with the help of Perplexity.ai. 

‍

Think your company is truly user-centric? Think again. Our groundbreaking report on UX Research (UXR) in 2024 shatters common assumptions about our industry.

We've uncovered a startling gap between what companies say about user-centricity and what they actually do. Prepare to have your perceptions challenged as we reveal the true state of UXR integration and its untapped potential in today's business landscape.

‍

The startling statistics 😅

‍

Here's a striking finding: only 16% of organizations have fully embedded UXR into their processes and culture. This disconnect between intention and implementation underscores the challenges in demonstrating and maximizing the true value of user research.

‍

What's inside the white paper 👀

‍

In this comprehensive white paper, we explore:

‍

• How companies use and value UX research
• Why it's hard to show how UX research helps businesses
• Why having UX champions in the company matters
• New ways to measure and show the worth of UX research
• How to share UX findings with different people in the company
• New trends changing how people see and use UX research

‍

Stats sneak peek 🤖

‍

- Only 16% of organizations have fully embedded UX Research (UXR) into their processes and culture. This highlights a significant gap between the perceived importance of user-centricity and its actual implementation in businesses.

‍

- 56% of organizations aren't measuring the impact of UXR at all. This lack of measurement makes it difficult for UX researchers to demonstrate the value of their work to stakeholders.

‍

- 68% of respondents believe that AI will have the greatest impact on the analysis and synthesis phase of UX research projects. This suggests that while AI is expected to play a significant role in UXR, it's seen more as a tool to augment human skills rather than replace researchers entirely.

‍

The UX research crossroads 🛣️

‍

As our field evolves with AI, automation, and democratized research, we face a critical juncture: how do we articulate and amplify the value of UXR in this rapidly changing landscape? We’d love to know what you think! So DM us in socials and let us know what you’re doing to bridge the gap.

‍

Are you ready to unlock the full potential of UXR in your organization? 🔐

‍

Download our white paper for invaluable insights and actionable strategies that will help you showcase and maximize the value of user research. In an era of digital transformation, understanding and leveraging UXR's true worth has never been more crucial.

‍

Download the white paper

‍

What's next?🔮

‍

Keep an eye out for our upcoming blog series, where we'll delve deeper into key findings and strategies from the report. Together, we'll navigate the evolving UX landscape and elevate the value of user insights in driving business success and exceptional user experiences.