It’s been a long day. Scratch that - it’s been a long week! Admit it. You loved every second of it.
Twelve hour days, the mad scramble to get the prototype ready in time, the stakeholders poking their heads in occasionally, dealing with no-show participants and the excitement around the opportunity to speak to real life human beings about product or service XYZ. Your mind is exhausted but you are buzzing with ideas and processing what you just saw. You find yourself sitting in your war room with several pages of handwritten notes and with your fellow observers you start popping open individually wrapped lollies leftover from the day’s sessions. Someone starts a conversation around what their favourite flavour is and then the real fun begins. Sound familiar? Welcome to the post user testing debrief meeting.
How do you turn those scribbled notes and everything rushing through your mind into a meaningful picture of the user experience you just witnessed? And then when you have that picture, what do you do next? Pull up a bean bag, grab another handful of those lollies we feed our participants and get comfy because I’m going to share my idiot-proof, step by step guide for turning your user testing notes into something useful.
Get the ball rolling by holding a post session debrief meeting while it’s all still fresh your collective minds. This can be done as one meeting at the end of the day’s testing or you could have multiple quick debriefs in between testing sessions. Choose whichever options works best for you but keep in mind this needs to be done at least once and before everyone goes home and forgets everything. Get all observers and facilitators together in any meeting space that has a wall like surface that you can stick post its to - you can even use a window! And make sure you use real post its - the fake ones fall off!
Before you put sharpie to post it, it’s essential to agree as a group on how you will tag your observations. Tagging the observations now will make the analysis work much easier and help you to spot patterns and themes. Colour coding the post its is by far the simplest and most effective option and how you assign the colours is entirely up to you. You could have a different colour for each participant or testing session, you could have different colours to denote participant attributes that are relevant to your study eg senior staff and junior staff, or you could use different colours to denote specific testing scenarios that were used. There’s many ways you could carve this up and there’s no right or wrong way. Just choose the option that suits you and your team best because you’re the ones who have to look at it and understand it. If you only have one colour post it eg yellow, you could colour code the pen colours you use to write on the notes or include some kind of symbol to help you track them.
That pile of paper is not going to process itself! Your next job as a group is to work through the task of transposing your observations to post it notes. For now, just stick them to the wall in any old way that suits you. If you’re the organising type, you could group them by screen or testing scenario. The positioning will all change further down the process, so at this stage it’s important to just keep it simple. For issues that occur repeatedly across sessions, just write them down on their own post its- doubles will be useful to see further down the track.In addition to holding a debrief meetings, you also need to round up everything that was used to capture the testing session/s. And I mean EVERYTHING.
Handwritten notes, typed notes, video footage and any audio recordings need to be reviewed just in case something was missed. Any handwritten notes should be typed to assist you with the completion of the report. Don’t feel that you have to wait until the testing is completed before you start typing up your notes because you will find they pile up very quickly and if your handwriting is anything like mine…. Well let’s just say my short term memory is often required to pick up the slack and even that has it’s limits. Type them up in between sessions where possible and save each session as it’s own document. I’ll often use the testing questions or scenario based tasks to structure my typed notes and I find that makes it really easy to refer back to.Now that you’ve processed all the observations, it’s time to start sorting your observations to surface behavioural patterns and make sense of it all.
Affinity diagramming is a fantastic tool for making sense of user testing observations. In fact it’s just about my favourite way to make sense of any large mass of information. It’s an engaging and visual process that grows and evolves like a living creature taking on a life of its own. It also builds on the work you’ve just done which is a real plus!By now, testing is over and all of your observations should all be stuck to a wall somewhere. Get everyone together again as a group and step back and take it all in. Just let it sit with you for a moment before you dive in. Just let it breathe. Have you done that? Ok now as individuals working at the same time, start by grouping things that you think belong together. It’s important to just focus on the content of the labels and try to ignore the colour coded tagging at this stage, so if session one was blue post its don’t group all the blue ones together just because they’re all blue! If you get stuck, try grouping by topic or create two groups eg issues and wins and then chunk the information up from there.
You will find that the groups will change several times over the course of the process and that’s ok because that’s what it needs to do.While you do this, everyone else will be doing the same thing - grouping things that make sense to them. Trust me, it’s nowhere near as chaotic as it sounds! You may start working as individuals but it won’t be long before curiosity kicks in and the room is buzzing with naturally occurring conversation.Make sure you take a step back regularly and observe what everyone else is doing and don’t be afraid to ask questions and move other people’s post its around- no one owns it! No matter how silly something may seem just put it there because it can be moved again. Have a look at where your tagged observations have ended up. Are there clusters of colour? Or is it more spread out? What that means will depend largely on how you decided to tag your findings. For example if you assigned each testing session its own colour and you have groups with lot’s of different colours in them you’ll find that the same issue was experienced by multiple people.Next, start looking at each group and see if you can break them down into smaller groups and at the same time consider the overall picture for bigger groups eg can the wall be split into say three high level groups.Remember, you can still change your groups at anytime.
Once you and your team are happy with the groups, it’s time to start condensing the size of this beast. Look for doubled up findings and stack those post its on top of each other to cut the groups down- just make sure you can still see how many there were. The point of merging is to condense without losing anything so don’t remove something just because it only happened once. That one issue could be incredibly serious. Continue to evaluate and discuss as a group until you are happy. By now clear and distinct groups of your observations should have emerged and at a glance you should be able to identify the key findings from your study.
Scoring relates to how serious the issues are and how bad the consequences of not fixing them are. There are arguments for and against the use of scoring and it’s important to recognise that it is just one way to communicate your findings.I personally rarely use scoring systems. It’s not really something I think about when I’m analysing the observations. I rarely rank one problem or finding over another. Why? Because all data is good data and it all adds to the overall picture.I’ve always been a huge advocate for presenting the whole story and I will never diminish the significance of a finding by boosting another. That said, I do understand the perspective of those who place metrics around their findings. Other designers have told me they feel that it allows them to quantify the seriousness of each issue and help their client/designer/boss make decisions about what to do next.We’ve all got our own way of doing things, so I’ll leave it up to you to choose whether or not you score the issues. If you decide to score your findings there are a number of scoring systems you can use and if I had to choose one, I quite like Jakob Nielsen’s methodology for the simple way it takes into consideration multiple factors. Ultimately you should choose the one that suits your working style best.
Let’s say you did decide to score the issues. Start by writing down each key finding on it’s own post it and move to a clean wall/ window. Leave your affinity diagram where it is. Divide the new wall in half: one side for wins eg findings that indicate things that tested well and the other for issues. You don’t need to score the wins but you do need to acknowledge what went well because knowing what you’re doing well is just as important as knowing where you need to improve. As a group (wow you must be getting sick of each other! Make sure you go out for air from time to time!) score the issues based on your chosen methodology.Once you have completed this entire process you will have everything you need to write a kick ass report.
No process is perfect and there are a few potential dramas to be aware of:
In the middle of the debrief meeting, someone has an epiphany. Shouts of We should move the help button! or We should make the yellow button smaller! ring out and the meeting goes off the rails.I’m not going to point fingers and blame any particular role because we’ve all done it, but it’s important to recognise that’s not why we’re sitting here. The debrief meeting is about digesting and sharing what you and the other observers just saw. Observing and facilitating user testing is a privilege. It’s a precious thing that deserves respect and if you jump into solution mode too soon, you may miss something. Keep the conversation on track by appointing a team member to facilitate the debrief meeting.
Handwritten notes taken by multiple observers over several days of testing adds up to an enormous pile of paper. Not only is it a ridiculous waste of paper but they have to be securely stored for three months following the release of the report. It’s not pretty. Typing them up can solve that issue but it comes with it’s own set of storage related hurdles. Just like the handwritten notes, they need to be stored securely. They don’t belong on SharePoint or in the share drive or any other shared storage environment that can be accessed by people outside your observer group. User testing notes are confidential and are not light reading for anyone and everyone no matter how much they complain. Store any typed notes in a limited access storage solution that only the observers have access to and if anyone who shouldn’t be reading them asks, tell them that they are confidential and the integrity of the research must be preserved and respected.
Before the storage dramas begin, you have to actually pick through the mountain of paper. Not to mention the video footage, and the audio and you have to chase up that sneaky observer who disappeared when the clock struck 5. All of this takes up a lot of time. Another time related issue comes in the form of too much time passing in between testing sessions and debrief meetings. The best way to deal with both of these issues is to be super organised and hold multiple smaller debriefs in between sessions where possible. As a group, work out your time commitments before testing begins and have a clear plan in place for when you will meet. This will prevent everything piling up and overwhelming you at the end.
At the end of that long day/week we’re all tired and discussions around scoring the issues can get a little heated. One person’s showstopper may be another person’s mild issue. Many of the ranking systems use words as well as numbers to measure the level of severity and it’s easy to get caught up in the meaning of the words and ultimately get sidetracked from the task at hand. Be proactive and as a group set ground rules upfront for all discussions. Determine how long you’ll spend discussing an issue and what you will do in the event that agreement cannot be reached. People want to feel heard and they want to feel like their contributions are valued. Given that we are talking about an iterative process, sometimes it’s best just to write everything down to keep people happy and merge and cull the list in the next iteration. By then they’ve likely had time to reevaluate their own thinking.
We all have our own ways of making sense of our user testing observations and there really is no right or wrong way to go about it. The one thing I would like to reiterate is the importance of collaboration and teamwork. You cannot do this alone, so please don’t try. If you’re a UX team of one, you probably already have a trusted person that you bounce ideas off. They would be a fantastic person to do this with. How do you approach this process? What sort of challenges have you faced? Let me know in the comments below.
If you missed our live training, don’t worry, we’ve got you covered! In this session, our product experts Katie and Aidan discuss why, how and when to benchmark an existing structure using Treejack.
They also talk through some benchmarking use cases, demo how to compare tasks between different studies, and which results are most helpful.
There is strong backlash about the perceived failures of Human Centred Design (HCD) and its contribution to contemporary macro problems. There seems to be a straightforward connection: HCD and Design Thinking have been adopted by organizations and are increasingly part of product/experience development, especially in big tech. However the full picture is more complex, and HCD does have some issues.
Dan Dixon, UX and Design Research Director and Stéphan Willemse, Strategy Director/Head of Strategy, both from the Digital Arts Network, recently spoke at UX New Zealand, the leading UX and IA conference in New Zealand hosted by Optimal Workshop, about the evolution and future of HCD.
In their talk, Dan and Stéphan cover the history of HCD, its use today, and its limitations, before presenting a Post HCD future. What could it be, and how should it be different? Dan and Stéphan help us to step outside of ourselves as we meet new problems with new ways of Design Thinking.
Dan is a long-term practitioner of human-centred experience design and has a wealth of experience in discovery and qual research. He’s worked in academic, agency and client-side roles in both the UK and NZ, covering diverse fields such as digital, product design, creative technology and game design. His history has blended a background in the digital industry with creative technology teaching and user experience research. He has taken pragmatic real-world knowledge into a higher education setting as well as bringing deeper research skills from academia into commercial design projects. In higher education, as well as talks and workshops, Dan has been teaching and sharing these skills for the last 16 years.
Stéphan uses creativity, design and strategy to help organizations innovate towards positive, progressive futures. He works across innovation, experience design, emerging technologies, cultural intelligence and futures projects with clients including Starbucks, ANZ, Countdown, TradeMe and the public sector. He holds degrees in PPE, Development Studies, Education and an Executive MBA. However, he doesn’t like wearing a suit and his idea of the perfect board meeting is at a quiet surf break. He thinks ideas are powerful and that his young twins ask the best questions about the world we live in.
Contact Details:
Email: dan.dixon@digitalartsnetwork.com
Dan and Stéphan take us through the evolving landscape of Human Centred Design (HCD) and Design Thinking. Can HCD effectively respond to the challenges of the modern era, and can we get ahead of the unintended consequences of our design? They examine the inputs and processes of design, not just the output, to scrutinize the very essence of design practice.
In the 1950s and 1960s, designers began exploring the application of scientific processes to design, aiming to transform it into a systematic problem-solving approach. Later in the 1960s, design thinkers in Scandinavia initiated the shift towards cooperative and participative design practices. Collaboration and engagement with diverse stakeholders became integral to design processes. Then, the 1970s and 1980s marked a shift in perspective, viewing design as a fundamentally distinct way of approaching problems.
Moving into the late 1980s and 1990s, design thinking expanded to include user-centered design, and the idea of humans and technology becoming intertwined. Then the 2000s witnessed a surge in design thinking, where human-centered design started to make its mark.
Dan and Stéphan discuss the “design squiggle”, a concept that portrays the messy and iterative nature of design, starting chaotically and gradually converging toward a solution. For 20 years, beginning in the early 90s, this was a popular way to explain how the design process feels. However, in the past 10 years or so, efforts to teach and pass down design processes have become common practice. Here enter concepts like the “double diamond” and “pattern problem”, which seek to be repeatable and process-driven. These neat processes, however, demand rigid adherence to specific design methods, which can ultimately stifle innovation.
The critique of such rigid design processes, which developed alongside HCD, highlights the need to acknowledge that humans are just one element in an intricate network of actors. By putting ourselves at the center of our design processes and efforts, we already limit our design. Design is just as much about the ecosystem surrounding any given problem as it is about the user. A limitation of HCD is that we humans are not actually at the center of anything except our own minds. So, how can we address this limitation?
Post-anthropocentric design starts to acknowledge that we are far less rational than we believe ourselves to be. It captures the idea that there are no clear divisions between ‘being human’ and everything else. This concept has become important as we adopt more and more technology into our lives, and we’re getting more enmeshed in it.
Post-human design extends this further by removing ourselves from the center of design and empathizing with “things”, not just humans. This concept embraces the complexity of our world and emphasizes how we need to think about the problem just as much as we think about the solution. In other words, post-human design encourages us to “live” in our design problem(s) and consider multiple interventions.
Finally, Dan and Stéphan discuss the concept of Planetary design, which stresses that everything we create, and everything we do, has the possibility to impact everything else in the world. In fact, our designs do impact everything else, and we need to try and be aware of all possibilities.
To think beyond HCD and to foster innovation in design, we can begin by embracing emerging design practices and philosophies such as "life-centered design," "Society-centered design," and "Humanity-centered design." These emerging practices have toolsets that are readily available online and can be seamlessly integrated into your design approach, helping us to break away from traditional, often linear, methodologies. Or, taking a more proactive stance, we can craft our own unique design tools and frameworks.
To illustrate how design processes can evolve to meet current and future challenges of our time, Dan and Stéphan present their concept of “Post human-centered design” (Post HCD). At its heart, it seeks to take what's great about HCD and build upon it, all while understanding its issues/limitations.
Dan and Stéphan put forward, as a starting point, some challenges for designers to consider as we move our practice to its next phase.
Suggested Post HCD principles:
The talk wraps up by encouraging designers to incorporate some of this thinking into everyday practice. Some key takeaways are:
Summary: A detailed UX research plan helps you keep your overarching research goals in mind as you work through the logistics of a research project.
There’s nothing quite like the feeling of sitting down to interview one of your users, steering the conversation in interesting directions and taking note of valuable comments and insights. But, as every researcher knows, it’s also easy to get carried away. Sometimes, the very process of user research can be so engrossing that you forget the reason you’re there in the first place, or unexpected things that come up that can force you to change course or focus.
This is where a UX research plan comes into play. Taking the time to set up a detailed overview of your high-level research goals, team, budget and timeframe will give your research the best chance of succeeding. It's also a good tool for fostering alignment - it can make sure everyone working on the project is clear on the objectives and timeframes. Over the course of your project, you can refer back to your plan – a single source of truth. After all, as Benjamin Franklin famously said: “By failing to prepare, you are preparing to fail”.
In this article, we’re going to take a look at the best way to put together a research plan.
Any project needs a plan to be successful, and user research is no different. As we pointed out above, a solid plan will help to keep you focused and on track during your research – something that can understandably become quite tricky as you dive further down the research rabbit hole, pursuing interesting conversations during user interviews and running usability tests. Thought of another way, it’s really about accountability. Even if your initial goal is something quite broad like “find out what’s wrong with our website”, it’s important to have a plan that will help you to identify when you’ve actually discovered what’s wrong.
So what does a UX research plan look like? It’s basically a document that outlines the where, why, who, how and what of your research project.
It’s time to create your research plan! Here’s everything you need to consider when putting this plan together.
The first thing you need to do is work out who the stakeholders are on your project. These are the people who have a stake in your research and stand to benefit from the results. In those instances where you’ve been directed to carry out a piece of research you’ll likely know who these people are, but sometimes it can be a little tricky. Stakeholders could be C-level executives, your customer support team, sales people or product teams. If you’re working in an agency or you’re freelancing, these could be your clients.
Make a list of everyone you think needs to be consulted and then start setting up catch-up sessions to get their input. Having a list of stakeholders also makes it easy to deliver insights back to these people at the end of your research project, as well as identify any possible avenues for further research. This also helps you identify who to involve in your research (not just report findings back to).
Action: Make a list of all of your stakeholders.
Before we get into timeframes and budgets you first need to determine your research questions, also known as your research objectives. These are the ‘why’ of your research. Why are you carrying out this research? What do you hope to achieve by doing all of this work? Your objectives should be informed by discussions with your stakeholders, as well as any other previous learnings you can uncover. Think of past customer support discussions and sales conversations with potential customers.
Here are a few examples of basic research questions to get you thinking. These questions should be actionable and specific, like the examples we’ve listed here:
A good research question should be actionable in the sense that you can identify a clear way to attempt to answer it, and specific in that you’ll know when you’ve found the answer you’re looking for. It's also important to keep in mind that your research questions are not the questions you ask during your research sessions - they should be broad enough that they allow you to formulate a list of tasks or questions to help understand the problem space.
Action: Create a list of possible research questions, then prioritize them after speaking with stakeholders.
Your budget will play a role in how you conduct your research, and possibly the amount of data you're able to gather.
Having a large budget will give you flexibility. You’ll be able to attract large numbers of participants, either by running paid recruitment campaigns on social media or using a dedicated participant recruitment service. A larger budget helps you target more people, but also target more specific people through dedicated participant services as well as recruitment agencies.
Note that more money doesn't always equal better access to tools - e.g. if I work for a company that is super strict on security, I might not be able to use any tools at all. But it does make it easier to choose appropriate methods and that allow you to deliver quality insights. E.g. a big budget might allow you to travel, or do more in-person research which is otherwise quite expensive.
With a small budget, you’ll have to think carefully about how you’ll reward participants, as well as the number of participants you can test. You may also find that your budget limits the tools you can use for your testing. That said, you shouldn’t let your budget dictate your research. You just have to get creative!
Action: Work out what the budget is for your research project. It’s also good to map out several cheaper alternatives that you can pursue if required.
How long do you think your user research project will take? This is a necessary consideration, especially if you’ve got people who are expecting to see the results of your research. For example, your organization’s marketing team may be waiting for some of your exploratory research in order to build customer personas. Or, a product team may be waiting to see the results of your first-click test before developing a new signup page on your website.
It’s true that qualitative research often doesn’t have a clear end in the way that quantitative research does, for example as you identify new things to test and research. In this case, you may want to break up your research into different sub-projects and attach deadlines to each of them.
Action: Figure out how long your research project is likely to take. If you’re mixing qualitative and quantitative research, split your project timeframe into sub-projects to make assigning deadlines easier.
Who you recruit for your research comes from your research questions. Who can best give you the answers you need? While you can often find participants by working with your customer support, sales and marketing teams, certain research questions may require you to look further afield.
The methods you use to carry out your research will also have a part to play in your participants, specifically in terms of the numbers required. For qualitative research methods like interviews and usability tests, you may find you’re able to gather enough useful data after speaking with 5 people. For quantitative methods like card sorts and tree tests, it’s best to have at least 30 participants. You can read more about participant numbers in this Nielsen Norman article.
At this stage of the research plan process, you’ll also want to write some screening questions. These are what you’ll use to identify potential participants by asking about their characteristics and experience.
Action: Define the participants you’ll need to include in your research project, and where you plan to source them. This may require going outside of your existing user base.
The research methods you use should be informed by your research questions. Some questions are best answered by quantitative research methods like surveys or A/B tests, with others by qualitative methods like contextual inquiries, user interviews and usability tests. You’ll also find that some questions are best answered by multiple methods, in what’s known as mixed methods research.
If you’re not sure which method to use, carefully consider your question. If we go back to one of our earlier research question examples: “How do our current customers go about tracking their orders?”, we’d want to test the navigation pathways.
If you’re not sure which method to use, it helps to carefully consider your research question. Let’s use one of our earlier examples: “Is it easy for users to check their order history in our iPhone app?” as en example. In this case, because we want to see how users move through our app, we need a method that’s suited to testing navigation pathways – like tree testing.
For the question: “What actions do our customers take when they’re thinking about buying a new TV?”, we’d want to take a different approach. Because this is more of an exploratory question, we’re probably best to carry out a round of user interviews and ask questions about their process for buying a TV.
Action: Before diving in and setting up a card sort, consider which method is best suited to answer your research question.
A protocol is essentially a script for your user research. For the most part, it’s a list of the tasks and questions you want to cover in your in-person sessions. But, it doesn’t apply to all research types. For example, for a tree test, you might write your tasks, but this isn't really a script or protocol.
Writing your protocol should start with actually thinking about what these questions will be and getting feedback on them, as well as:
Action: This is essentially a research plan within a research plan – it’s what you’d take to every session.
Happy researching!