Assurity is a New Zealand based technology delivery consultancy.
Assurity is a New Zealand based technology delivery consultancy.
Consulting B2B
Assurity is New Zealand’s leading technology delivery consultancy that works across user research, testing, design thinking processes, lean business analysis and more.
The Assurity team was recruited by the New Zealand Customs Service to improve the way that passengers move through the customs gates at Christchurch International Airport. What’s interesting about this project is not just that the team was designing a physical space — it’s that the team did the research and testing in virtual reality (VR).
Corvecto, a virtual and augmented reality training platform, handled the VR visual designs, while Assurity handled all of the user research and testing.
Senior Consultants Yvonne Tse and Rupert Burton answered a few questions about how the project went, what it involved, and some advice for others who want to run a VR research project.
Yvonne: The initial scoping exercise of any VR engagement is quite different, and can be difficult in particular if you haven’t seen the environment beforehand. It is harder to work through with a client on something that is conceptual in three dimensions, but maybe that’s just about getting used to the format?
There are many considerations that need to be taken into account, including:
Rupert: We’ve created our own “checklist” of VR things that we use in addition to the standard usability interview checklist.
Rupert: We used the 3D modeling tool SketchUp to create the environment and the specific elements within it, this part is fairly straightforward. The complexity comes with the interactive elements, this is something that we have yet to figure out ourselves as it involves a bit of coding. This is where partnering with a good development team like Corvecto comes in handy. I wouldn’t say it’s beyond us, but we are focusing on using VR as a prototyping tool; we’re not getting into the development space.
Yvonne: We allow for practice/onboarding time for our participants, ideally in a separate environment where they can familiarise themselves with the controls and get over the “wow factor”. We do not proceed without them feeling comfortable.
Rupert: Awareness of the symptoms and causes of cybersickness is essential, we do not want our participants to walk away feeling worse than when they arrived. We need to make sure that the experience is not going to leave people feeling sick when it goes into production, but we also have a responsibility to our participants. Be prepared to raise the risk with the client and be willing to push back and insist on getting causes fixed – you don’t want to be in the situation where you’re greeting your next participant knowing that there’s a 50% chance that they will shortly be feeling nauseous.
Rupert: Allow extra time and over-recruit – we’ve found it easier to recruit for VR sessions since people are keen to give it a go, but some people have a low tolerance for cybersickness so you need to have the capacity that allows you to terminate a session.
Yvonne: When recruiting for VR we are always interested in an individual’s digital literacy, whether they have had experience in gaming and VR before as this usually correlates to how much time it takes for someone to ‘onboard’ successfully and their learning time.
Rupert: You need to be a lot more vigilant when running VR sessions – you’re not just noting what they say and how they say it. You’ve got to keep an eye on what they’re looking at and how they’re interacting with it whilst making sure that they don’t walk into anything or tie themselves up in knots.
Usually we’ve conducted user research on the application or website that was going to go into production. We have done this in VR too, but VR also lends itself well to being a tool to simulate and test real life environments as we did in the Customs project, in this we were testing potential physical environments rather than the application itself.
Yvonne: Virtual reality in user experience blurs the line between a conventional moderated usability test session and a contextual interview. The user is experiencing a physical environment that is entirely constructed and different to the context to the interviewer. Due to the constraints in our projects so far, we weren’t able to be immersed in the environment at the same time as our user. However, we have discussed that this approach would have its advantages and disadvantages – e.g. as having a virtual avatar, or a disembodied ‘interviewer’ voice could potentially ruin the immersive experience, but placing yourself in the environment could allow for the interviewer to understand more of what’s going on. There is plenty of room to improve UX practices as technology advances in this area.
Rupert: Much of our research approach for VR is the same as any other project, but for VR there are difficulties such as: How do you gauge emotional response from a participant that has half of their face covered by a headset or is looking in another direction?
One of the beauties of VR is that it encourages immersion in an experience, but we need to talk to the participant as they are experiencing it – how much is this questioning damaging the immersion, and are the results still going to be valid?
Rupert: VR is highly immersive, the brain is easily tricked into making the user feel like they are really there. Even with basic graphics and no sound we have found people getting emotionally engaged with an experience. It has huge potential in a large range of situations. If you take the Customs project as an example, the alternative to VR would have been to recreate the environment physically by renting a large space and recreating the internal furnishings. I can’t imagine that it could be done for less than the cost of a few hours of development, and I suspect that the level of engagement wouldn’t be the same either.
Rupert: Definitely, VR research is super fun and engaging for both researcher and participants. I would argue that the UX of VR half-hearted design and poor usability, it just won’t be successful.
Rupert: The Customs project was something of a trial for them to see if it was a useful tool. Not only did they get usable results, but I believe they’ve seen the value of VR for future work.
Yvonne: Yes they have! They have increased the space after aviation security and divided the space according to our users’ preferred layout. Signage is a completely separate project for our clients, however, they have taken on board our feedback for future projects.
Yvonne: We did not have to in our virtual reality projects, as they were keen to explore the possibility of VR for the simulation of a physical space and to minimise disruption to BAU.
Rupert: Not so far, we’ve been lucky enough to work on some projects where VR has been an experiment for the client and they needed to see that it could be successful, so testing was probably the only option. I imagine this may become a factor in the future, at which point I will bring up the “UX is more important in VR than ever before” argument that I mentioned earlier.
Yvonne: For our usability sessions, we use Reframer as we find it to be an effective tool to capture observations and allows for easy analysis. We occasionally use Questions too for pre-or post-session questionnaires. Time spent in a VR environment is linked to cyber sickness, so it’s best to limit exposure to the essential elements and ask supplementary questions before or after.
Yvonne: Body language is more important than ever! We haven’t quite worked out the best position for our notetaker to be placed as a user is constantly moving, but will be thinking about better ways to do this next time.
Rupert: We rely on predefined tags to help with note-taking speed and efficiency. I like the way that adding quote marks automatically tags it as a quote and make use of the star option to quickly highlight great material on the fly.
Yvonne: Usually, we have a facilitator and then a junior member of the team as notetaker. During the day we usually upskill the notetaker so they can Facilitate the later sessions if they feel confident enough to lead the sessions. This can be quite an ask as during facilitation you are choreographing. You are making sure the participant is comfortable relying that all important body language and expressions back to the notetaker, especially when the participant is facing away from the notetaker.
Rupert: For some of our user tests we have set-up an extra room for observers.
Yvonne: Again there is no real brief as such but you quickly start to work in sync with the notetaker to ensure nothing is missed. The first thing from a notetaker perspective is have a rolling chair! Simple but effective way for the notetaker to move out of the way or indeed to adjust themselves to get the best view of how the participant is interacting with the environment.
You can also be less neutral than you would normally be, as the participant cannot see outside of the environment it gives you the opportunity for a lot of non-verbal communication with the notetaker, you can express things to the notetaker that give much more information about the session, what is happening with their body language, this can make up somewhat for what you lose with the facial expressions.
Rupert: It doesn’t necessarily involve the relationship between the facilitator and the notetaker, we are also keen to make the experience as easy for the participant as possible, we keep all the demographics and preference questioning until after the participants have finished with the VR aspect, this allowed some of the bias to be removed, for instance we noticed a trend where people preferred the most recent layout they had walked through, so giving them time after showing what they had seen allowed them time to really reflect on what they had experienced. As part of this we also gave the person time to warm up to allow them to feel comfortable with the technology and lose some of the need to treat the simulation as a game and lose the inherent wow factor.
