Event Recap: Measuring the Value of UX Research at UXDX
Last week Optimal Workshop was delighted to sponsor UXDX USA 2024 in New York. The User Experience event brings together Product, Design, UX, CX, and Engineering professionals and our team had an amazing time meeting with customers, industry experts, and colleagues throughout the conference. This year, we also had the privilege of sharing some of our industry expertise by running an interactive forum on “Measuring the Value of UX Research” - a topic very close to our hearts.
Our forum, hosted by Optimal Workshop CEO Alex Burke and Product Lead Ella Fielding, was focused on exploring the value of User Experience Research (UXR) from both an industry-wide perspective and within the diverse ecosystem of individual companies and teams conducting this type of research today.
The session brought together a global mix of UX professionals for a rich discussion on measuring and demonstrating the effectiveness of and the challenges facing organizations who are trying to tie UXR to tangible business value today.
The main topics for the discuss were:
Metrics that Matter: How do you measure UXR's impact on sales, customer satisfaction, and design influence?
Challenges & Strategies: What are the roadblocks to measuring UXR impact, and how can we overcome them?
Beyond ROI: UXR's value beyond just financial metrics
Some of the key takeaways from our discussions during the session were:
The current state of UX maturity and value
Many UX teams don’t measure the impact of UXR on core business metrics and there were more attendees who are not measuring the impact of their work than those that are measuring it.
Alex & Ella discussed with the attendees the current state of UX research maturity and the ability to prove value across different organizations represented in the room. Most organizations were still early in their UX research maturity with only 5% considering themselves advanced in having research culturally embedded.
Defining and proving the value of UX research
The industry doesn’t have clear alignment or understanding of what good measurement looks like. Many teams don’t know how to accurately measure UXR impact or don’t have the tools or platforms to measure it, which serve as core roadblocks for measuring UXRs’ impact.
Alex and Ella discussed challenges in defining and proving the value of UX research, with common values being getting closer to customers, innovating faster, de-risking product decisions, and saving time and money. However, the value of research is hard to quantify compared to other product metrics like lines of code or features shipped.
Measuring and advocating for UX research
When teams are measuring UXR today there is a strong bias for customer feedback, but little ability or understanding about how to measure impact on business metrics like revenue.
The most commonly used metrics for measuring UXR are quantitative and qualitative feedback from customers as opposed to internal metrics like stakeholder involvement or tieing UXR to business performance metrics (including financial performance).
Attendees felt that in organizations where research is more embedded, researchers spend significant time advocating for research and proving its value to stakeholders rather than just conducting studies. This included tactics like research repositories and pointing to past study impacts as well as ongoing battles to shape decision making processes.
One of our attendees highlighted that engaging stakeholders in the process of defining key research metrics prior to running research was a key for them in proving value internally.
Relating user research to financial impact
Alex and Ella asked the audience if anyone had examples of demonstrating financial impact of research to justify investment in the team and we got some excellent examples from the audience proving that there are tangible ways to tie research outcomes to core business metrics including:
Calculating time savings for employees from internal tools as a financial impact metric.
Measuring a reduction in calls to service desks as a way to quantify financial savings from research.
Most attendees recognise the value in embedding UXR more deeply in all levels of their organization - but feel like they’re not succeeding at this today.
Most attendees feel that UXR is not fully embedded in their orgnaization or culture, but that if it was - they would be more successful in proving its overall value.
Stakeholder buy-in and engagement with UXR, particularly from senior leadership varied enormously across organizations, and wasn’t regularly measured as an indicator of UXR value
In organizations where research was more successfully embedded, researchers had to spend significant time and effort building relationships with internal stakeholders before and after running studies. This took time and effort away from actual research, but ended up making the research more valuable to the business in the long run.
With the large range of UX maturity and the democratization of research across teams, we know there’s a lot of opportunity for our customers to improve their ability to tie their user research to tangible business outcomes and embed UX more deeply in all levels of their organizations. To help fill this gap, Optimal Workshop is currently running a large research project on Measuring the Value of UX which will be released in a few weeks.
If you missed our recent live training on Prototype Testing, don’t worry—we’ve got everything you need right here! You can catch up at your convenience, so grab a cup of tea, put your feet up, and enjoy the show.
In the session, we explored the powerful new features of our Prototype Testing tool, offering a step-by-step guide to setting up, running, and analyzing your tests like a seasoned pro. This tool is a game-changer for your design workflow, helping you identify usability issues and gather real user feedback before committing significant resources to development.
Here’s a quick recap of the highlights:
1. Creating a prototype test from scratch using images
We walked through how to create a prototype test from scratch using static images. This method is perfect for early-stage design concepts, where you want to quickly test user flows without a fully interactive prototype.
2. Preparing your Figma prototype for testing
Figma users, we’ve got you covered! We discussed how to prepare your Figma prototype for the smoothest possible testing experience. From setting up interactions to ensuring proper navigation, these tips ensure participants have an intuitive experience during the test. For more detailed instructions, check out our help article
3. Seamless Figma prototype imports
One of the standout features of the tool is its seamless integration with Figma. We showed how easy it is to import your designs directly from Figma into Optimal, streamlining the setup process. You can bring your working files straight in, and resync when you need to with one click of a button.
4. Understanding usability metrics and analyzing results
We explored how to analyze the usability metrics, and walked through what the results can indicate on click maps and paths. These visual tools allow you to see exactly how participants navigate your design, making it easier to spot pain points, dead ends, or areas of friction. By understanding user behavior, you can rapidly iterate and refine your prototypes for optimal user experience.
It’s a chicken and egg situation when it comes to securing funding for a large transformation program in government. On one hand, you need to submit a business case and, as part of that, you need to make early decisions about how you might approach and deliver the program of work. On the other hand, you need to know enough about the problem you are going to solve to ensure you have sufficient funding to understand the problem better, hire the right people, design the right service, and build it the right way.
Now imagine securing hundreds of millions of dollars to design and build a service, but not feeling confident about what the user needs are. What if you had the opportunity to change this common predicament and influence your leadership team to carry out alignment activities, all while successfully delivering within the committed time frames?
Meera Pankhania, Design Director and Co-founder of Propel Design, recently spoke at UX New Zealand, the leading UX and IA conference in New Zealand hosted by Optimal Workshop, on traceability and her learnings from delivering a $300 million Government program.
In her talk, Meera helps us understand how to use service traceability techniques in our work and apply them to any environment - ensuring we design and build the best service possible, no matter the funding model.
Background on Meera Pankhania
As a design leader, Meera is all about working on complex, purpose-driven challenges. She helps organizations take a human-centric approach to service transformation and helps deliver impactful, pragmatic outcomes while building capability and leading teams through growth and change.
Meera co-founded Propel Design, a strategic research, design, and delivery consultancy in late 2020. She has 15 years of experience in service design, inclusive design, and product management across the private, non-profit, and public sectors in both the UK and Australia.
Meera is particularly interested in policy and social design. After a stint in the Australian Public Service, Meera was appointed as a senior policy adviser to the NSW Minister for Customer Service, Hon. Victor Dominello MP. In this role, she played a part in NSW’s response to the COVID pandemic, flexing her design leadership skills in a new, challenging, and important context.
From funding to delivery: ensuring alignment from start to finish 🏁🎉👏
Meera’s talk explores a fascinating case study within the Department of Employment Services (Australia) where a substantial funding investment of around $300 million set the stage for a transformative journey. This funding supported the delivery of a revamped Employment Services Model, which had the goal of delivering better services to job seekers and employers, and a better system for providers within this system. The project had a focus on aligning teams prior to delivery, which resulted in a huge amount of groundwork for Meera.
Her journey involved engaging various stakeholders within the department, including executives, to understand the program as a whole and what exactly needed to be delivered. “Traceability” became the watchword for this project, which is laid out in three phases.
Phase 1: Aligning key deliverables
Phase 2: Ensuring delivery readiness
Phase 3: Building sustainable work practices
Phase 1: Aligning key deliverables 🧮
Research and discovery (pre-delivery)
Meera’s work initially meant conducting extensive research and engagement with executives, product managers, researchers, designers, and policymakers. Through this process, a common theme was identified – the urgent (and perhaps misguided) need to start delivering! Often, organizations focus on obtaining funding without adequately understanding the complexities involved in delivering the right services to the right users, leading to half-baked delivery.
After this initial research, some general themes started to emerge:
Assumptions were made that still needed validation
Teams weren’t entirely sure that they understood the user’s needs
A lack of holistic understanding of how much research and design was needed
The conclusion of this phase was that “what” needed to be delivered wasn’t clearly defined. The same was true for “how” it would be delivered.
Traceability
Meera’s journey heavily revolved around the concept of "traceability” and sought to ensure that every step taken within the department was aligned with the ultimate goal of improving employment services. Traceability meant having a clear origin and development path for every decision and action taken. This is particularly important when spending taxpayer dollars!
So, over the course of eight weeks (which turned out to be much longer), the team went through a process of combing through documents in an effort to bring everything together to make sense of the program as a whole. This involved some planning, user journey mapping, and testing and refinement.
Documenting Key Artifacts
Numerous artifacts and documents played a crucial role in shaping decisions. Meera and her team gathered and organized these artifacts, including policy requirements, legislation, business cases, product and program roadmaps, service maps, and blueprints. The team also included prior research insights and vision documents which helped to shape a holistic view of the required output.
After an effort of combing through the program documents and laying everything out, it became clear that there were a lot of gaps and a LOT to do.
Prioritising tasks
As a result of these gaps, a process of task prioritization was necessary. Tasks were categorized based on a series of factors and then mapped out based on things like user touch points, pain points, features, business policy, and technical capabilities.
This then enabled Meera and the team to create Product Summary Tiles. These tiles meant that each product team had its own summary ahead of a series of planning sessions. It gave them as much context (provided by the traceability exercise) as possible to help with planning. Essentially, these tiles provided teams with a comprehensive overview of their projects i.e. what their user needs, what certain policies require them to deliver, etc.
Phase 2: Ensuring delivery readiness 🙌🏻
Meera wanted every team to feel confident that we weren’t doing too much or too little in order to design and build the right service, the right way.
Standard design and research check-ins were well adopted, which was a great start, but Meera and the team also built a Delivery Readiness Tool. It was used to assess a team's readiness to move forward with a project. This tool includes questions related to the development phase, user research, alignment with the business case, consideration of policy requirements, and more. Ultimately, it ensures that teams have considered all necessary factors before progressing further.
Phase 3: Building sustainable work practices 🍃
As the program progressed, several sustainable work practices emerged which Government executives were keen to retain going forward.
Some of these included:
ResearchOps Practice: The team established a research operations practice, streamlining research efforts and ensuring that ongoing research was conducted efficiently and effectively.
Consistent Design Artifacts: Templates and consistent design artifacts were created, reducing friction and ensuring that teams going forward started from a common baseline.
Design Authority and Ways of Working: A design authority was established to elevate and share best practices across the program.
Centralized and Decentralized Team Models: The program showcased the effectiveness of a combination of centralized and decentralized team models. A central design team provided guidance and support, while service design leads within specific service lines ensured alignment and consistency.
Why it matters 🔥
Meera's journey serves as a valuable resource for those working on complex design programs, emphasizing the significance of aligning diverse stakeholders and maintaining traceability. Alignment and traceability are critical to ensuring that programs never lose sight of the problem they’re trying to solve, both from the user and organization’s perspective. They’re also critical to delivering on time and within budget!
Traceability key takeaways 🥡
Early Alignment Matters: While early alignment is ideal, it's never too late to embark on a traceability journey. It can uncover gaps, increase confidence in decision-making, and ensure that the right services are delivered.
Identify and audit: You never know what artifacts will shape your journey. Identify everything early, and don’t be afraid to get clarity on things you’re not sure about.
Conducting traceability is always worthwhile: Even if you don’t find many gaps in your program, you will at least gain a high level of confidence that your delivery is focused on the right things.
Delivery readiness key takeaways 🥡
Skills Mix is Vital: Assess and adapt team member roles to match their skills and experiences, ensuring they are positioned optimally.
Not Everyone Shares the Same Passion: Recognize that not everyone will share the same level of passion for design and research. Make the relevance of these practices clear to all team members.
Sustainability key takeaways 🥡
One Size Doesn't Fit All: Tailor methodologies, templates, and practices to the specific needs of your organization.
Collaboration is Key: Foster a sense of community and collective responsibility within teams, encouraging shared ownership of project outcomes.
It’s no secret - New Zealand has a diversity problem in design and technology.
Throughout her career, Sachi often felt like the odd one out - the only woman, the only Pasifika person, the one who laughed too loud, the one who looked different and sounded different. But as a leader, Sachi has been able to create change.
Sachi Taulelei, Head of Design, ANZ, recently spoke at UX New Zealand, the leading UX and IA conference in New Zealand hosted by Optimal Workshop, on how she is building a diverse team of designers at New Zealand’s largest bank.
In her talk, Sachi shares the challenges she’s faced as a Pasifika woman in design and technology; and how this has shaped her approach to leadership and her drive to create inclusive environments where individuals and teams thrive.
Background on Sachi Taulelei
Sachi is a creative strategist, a design leader, and a recovering people pleaser. She has worked in digital and design for over 25 years, spending most of her career creating and designing digital experiences centered on people.
As a proud Pasifika woman, she has a particular interest in diversity, equity, and inclusion. She has spoken out about the need for more diversity within design and technology and the impact it can have on the technology we create.
Sachi is passionate about giving back - when she's not running after her two kids, you'll find her mentoring Pasifika youth, cheering on young leaders through the Young Enterprise Scheme, judging awards for Women in AI, or volunteering at the local hospice.
Odd one out: embracing diversity in design and technology ✨
Looking and sounding different from her peers, Sachi always felt like she was trying to find her place in the office. She always felt like she didn’t belong.
Sachi has experienced all forms of racism and discrimination as a result of her heritage. These experiences aren’t spoken about and often go unnoticed by the majority. She has held equivalent jobs to male counterparts but received lower pay, and was advised to change her name from Sachi to Sacha on her job applications to improve her chances.
Sachi’s response was to work hard and become great at what she does, which was recognized over time. Slowly, she began to rise through the ranks. However, having reached leadership roles, she struggled to be heard and participate, without knowing why. The advice was given freely by managers to “stick at it”, to “grow thicker skin”, and to grow through the “school of hard knocks”. Although this advice worked at face value and she flourished, Sachi began to feel like a fraud and constantly second-guessing herself. She began to “edit” herself to fit into an acceptable mold and, in doing so, felt like she lost part of who she was.
What is success? 🏆🎯💎
Success often comes in the form of our leaders who have already climbed the mountains of achievement. When you see success in this way, as someone who doesn’t fit the mold, there is pressure to conform to get ahead. Using the same tools and advice given to these leaders, she realized, would actually hold her back.
Realizing true value through our uniqueness 🪐🦋
Sachi recounts the treatment of Japanese-American citizens in the U.S. in the years following Pearl Harbour, where Japanese-American citizens were moved to concentration camps. This happened despite an official report finding conclusively that there was no threat from this population. Even though Germany and Italy were also at war with the U.S., for example, citizens with Italian and German heritage were not treated this way. This caused immeasurable pain, shame, and fear for the victims, and fostered a head-down, work-hard mentality in order to try and forget the treatment they received. This attitude, Sachi believes, was passed down to her from her ancestors who experienced that reality. Sachi explains that while there are many things that can hold someone back in life, creating meaningful change starts with introspection. Often, that requires us to work through fear and shame.
Reflecting on her heritage, which is part Samoan and part Japanese, Sachi started to embrace her unique traits. In her case, she embraced the deep empathy and human compassion from her Japanese side and the deep sense of community and connection from her Samoan side. Her uniqueness is something to celebrate, not to hide behind.
Becoming a leader and realizing this, Sachi wanted to create a team culture based on equity, openness, and a sense of belonging – all things that Sachi wished for herself on her journey.
Why it matters 💫
Once she understood herself and what she wanted for her team, Sachi set to work on building a new team culture. Sachi breaks down key learnings from how she turned this vision into reality.
Define
Define what diversity means for your team. You need to clearly understand what it is you want to achieve before you can achieve it. For Sachi’s team, they knew that they wanted to create a team that was representative of New Zealand. Sachi knew, for example, that she had a lack of Māori and Pacific representation within the team. Māori and Pasifika represent 25% of the population. So, an effort was made to increase ranks by hiring talent from these cultures.
Additionally, Sachi focused on creating new role levels - from intern right through to graduates, juniors, and intermediate-level positions. This helped to acknowledge age differences within her team and also helped to manage career progression opportunities.
Effort
It can be difficult to achieve diversity and inclusion and it requires a lot of work. For example, Sachi learned that posting an ad on job boards and expecting to receive hundreds of Māori and Pasifika applicants wasn’t realistic. Instead, partnerships were built with local design schools, and networking events were consistently attended. Job referrals from within the team were also leveraged, as well as establishing a strong direction for recruitment specialists within the organization.
Sachi also recognized that, as a leader, she needed to be more visible and more vocal about sharing her views of the world and what she was trying to achieve. It was important to be clear about the type of culture she was building within her team so that she could promote it.
In less than a year her team grew (from 11 to 40!) which meant a focus on building an inclusive team culture was required. The central theme throughout this time was, “You have to connect to yourself and your strengths first and foremost, before you can connect with others and as a team”. This meant that the team used tools like the Clifton Strength Finder, in order to learn about themselves and each other. Each designer was then encouraged to delve into their own natural working styles and were taught how to amplify their own strengths through various workshops. This approach also becomes handy when recruiting and strengthening potential weak spots.
Integrity
It’s important to have leaders who care - you can’t do it on your own. There can be pain points on the journey to creating diversity and inclusion, so it’s necessary to have leaders who listen, support, and work through some of the challenges that can arise.
Benefits of diversity and inclusion in design teams 👩🏼🤝👨🏿
Why push for diversity and inclusion? Sachi argues that the benefits are evident in the way that her team designs.
For example, her team:
Insist that research is done with diverse customer groups
Advocates for accessibility when no one else will
Understand problems from different perspectives before diving into a project
Most importantly, the benefits show up in the way that each other is treated, and the relationships that are built with key stakeholders. Diversity and inclusion are wins for everyone - the team, the organization, and the customer.
.png)
