Why Test?

The first question is why is it important to test. First of all, you want to test in order to evaluate your VR app, to see what doesn’t work as intended and get some insights into how to improve it, find possible bugs, to avoid having to table the whole idea after working on it for months and spending large budgets, to prevent rebuilding the project, at least on a big scale. It helps to be more careful with the budget. User Testing provides an opportunity to identify problems along the way, so there is enough time and resources to iterate. You can also use user testing to generate new ideas. For example, it can be tactical user research, to understand what to build next.

There can be plenty to discover:

  • Do people understand how to use your app and what’s the learning curve for them? It shows how different demographics use your software and if patterns appear (what they do or try to do).
  • Do people enjoy the app or not, would they recommend it or play it again themselves? How valuable it is for them?
  • Was it comfortable?
  • What was their favorite part?

Testing is important to ensure that the design decisions are right and are based on business goals and people who will use this VR experience. It may also help to secure the longevity of the vision and make sure the app idea works in the long run.

Also, building for VR is not the same as building for desktop, so there can be unexpected moments. One of the other goals of User Testing is to uncover people’s mental models. what people think about something, what they assume. They didn’t try, they don’t know much about VR or certain games or apps, but they already assume and expect something. Those are large parts of what may hold some people back from VR at the moment, for example.

And, finally, when you are aware of it all, you can build a better VR experience to make people happy, and you will get more downloads, more sales, more 5-star reviews, higher user retention, and so on because you’ll know what the potential users want and what they like.

When to Test?

Next, what is crucial to consider is when to start testing? It should be done early on and the sooner the better.

When a first scene draft is created, it’s a good time to start testing. There is no need to polish or create a detailed prototype in order to test it. You can even test drafts (not interactive paper prototypes) to preview scenes, paper VR prototypes (skyboxes), storyboards-to-VR, early blocking (objects’ locations), and test some small parts to check the color palette, scales, locations, and approximate distances. It’s easy to test rough volumetric prototypes (grey- or sometimes called white boxing) to see interactions. Traditional media designers create wireframes for web and mobile. To maintain a similar workflow for VR, you can create draft scenes utilizing simple untextured geometry for any object and add mechanics to test if your interaction system is fun, flexible, and reusable.

Don't forget to test regularly throughout the whole VR development process. When a 1-st scene is created, it’s a good time to start testing. Preview it in VR every time you make a change. Test small parts (colors, interactions, scales, proximity, brightness, contrast) to make sure the experience is comfortable, as well as test general ideas and mechanics to see if your approach is fun. Test at all stages of VR product design. And keep testing after launch to uncover new features and expand your community.

The importance of testing the experience in VR is essential because it always feels very different from the desktop. Even very rough small prototypes can confirm the comfort zones, locations, and scales.

What to Test?

Next, it’s important to define what to test, and which part of a VR experience should be tested.

Let’s say you already defined why you want to test an app. For example, you want to find out why people quit after a certain part of the experience or after some time. You want to increase playtime or ensure people continue playing at least until a certain part or level.

To make it happen, depending on the app itself, you may want to test what may cause game interruption. There can be a variety of reasons, e.g., bugs in the code that make the game laggy or just crush it exactly where you think players quit, or poor optimization which makes an app too slow to load or function. One of the other drop-off reasons can as well be an uncomfortable VR environment that causes motion sickness, eye strain, and overall discomfort. It might be a bad user experience or visual design, e.g., too low contrast might cause players to not notice the UI. Or they simply get stuck and don’t know how to proceed further in the game, they get confused, can’t understand what to do next, and eventually quit.

It’s important to know what you want to find out and how it will affect the VR experience after you get the answer to your questions. Will it help? Then you should test it.

Whom to Test With?

The easiest way to test at the very beginning of design and development is to test yourself or with your team. It’s also important to test with people who have domain expertise in the industry type that your product is in. For example, if you are working on some medical-related VR app, you would need to test it with people who have expertise in this specific topic. You can also show your VR app to people who are experts specifically in the VR industry.

And, of course, testing with people from your Target Audience, otherwise, it all might be useless. Test with those whom this app is intended for.

How many testers to select? 5 should be enough to test general usability issues, and some features, but 10 wouldn’t hurt either. To study demographics you would need more people (example - people aged 15-17 y.o, or ppl who play certain types of games). It’s always best to have access to a large pool of testers from your specific Target Audience.

When testing, don’t forget to compare new VR users (people who have never used VR before, or maybe just tried once, but have no knowledge about it) and experienced VR users (those who use VR regularly and own a VR device or even develop for VR). But the most wanted testers for you are those who would be potentially buying and using your app.

Recruiting testers can be challenging, but get creative, offer something valuable for your potential audience, e.g., early access, or do your research to find out what people from your particular Target Audience could be interested in in exchange for their feedback, if monetary compensation isn’t an option for you, and try.

There are definitely, some pros and cons with new and experienced VR users. Experienced users can test your app remotely because they have all the hardware. But they can be already used in various locomotion techniques, so it’s hard to test motion sickness, for example. They may also know how to use some features because they have previous VR experience.

In the case of testing with new VR users, you would need to carry your target device or invite people to your office, cafe, or co-working place in order to conduct testing. On the other hand, they will have a fresh view of your app and most likely their opinion will be unbiased.

Where to Find the Right Testers?

The next challenge is where to find all these people who desire to playtest your game? Well, it might be not so easy, but you can try recruiting people on Reddit or some Facebook groups and events.

You can make a post about an app you want to get tested and list all your criteria including:

  • What are you testing (e.g., VR game, 360 video)
  • Incentive
  • Any important details (devices requested, demographics, the timeframe to complete the test, and any other qualities you are looking for)
  • Link to your build or keys if the app is already launched

Read the group rules to ensure you acknowledge them. Members who are interested in helping you out will comment and share their feedback. Participation is voluntary and results purely rely on your expertise with user testing. The downside is that you do not always know what the response rate will be and if you will be able to complete testing on time, as well as it might not be consistent among all testers since there won’t be any methodology involved. All the members might be absolutely brilliant candidates in various fields but talking to the wrong target audience may bring the wrong results.

You can also attend non-VR-related expos where it will be suitable to showcase your VR app, or partner with a retail shop, school, or University.

It’s also important to consider the context and User Journey - why and where would people want to use your VR software? Is it intended for education, productivity, or leisure and entertainment? Where will people want to use it? Is it at home or an arcade experience? Single or multiplayer? User Journey represents the full end-to-end experience including the emotional part, motivations, actions, interactions, touchpoints, emotions, and feelings. What would trigger potential users to engage with your application?

Another detail is that the way of finding testers may be affected by the type of VR build itself as well as the development stage. WebVR can be more shareable, since there is no need to download it, and you could probably get more responses because it’s very accessible. Or if your build is already in one of the channels but not released yet, you c may want to share it with a selected number of people. If your app is already launched, sharing keys or store links in specific groups can be a solution, but unfortunately, it’s rarely possible to ensure that keys are used properly and won’t be stolen.

Methods and Tools

There are a lot of usability methods and techniques, but the following are the most useful and widely applicable:

  1. Guerrilla User Testing
  2. In-person and Remote VR Testing
  3. VR Analytics, Heatmaps
  4. Professional VR Testing Services

When selecting a test type, it is crucial to select wisely based on:

  • What is being tested
  • Platform (Oculus Quest 1, Meta Quest 2, HTC Vive, Valve Index, etc.)
  • Genre and type of experience (shooter, educational app, training, multiplayer puzzle, etc)
  • Target Audience
  • Development stage
  • The number of users (and is it a social app, a multiplayer, a single player?)

For example, for VR Analytics, having a certain number of users is essential because it won’t make any sense otherwise if an app has too few players yet.

Guerrilla Testing

Guerrilla user testing is usually a relatively simple and low-cost method that involves a facilitator approaching people and asking them to participate in a test, showing them an app, and asking questions. It’s usually very casual, and participants can be found practically anywhere from a cafe to an expo - wherever you can locate your Target Audience. And you can also select a location based on the User Journey.

Definitely, for a VR application to be tested with real users, it should be a 3D VR prototype with some more or less defined environments and objects. It needs to have the main interactions, sound, and some visual design but doesn’t have to be polished yet.

Attending various VR and non-VR expos and events in order to recruit the testers and specifically the first-time VR users can be helpful. It allows to test in a casual setting, and to observe them, observations play a really big role in VR testing. Post-test interviews and questionnaires are very helpful as well.

Guerrilla testing is perfectly suited to:

  • Conduct frequent tests throughout the design and development cycle to identify the usability issues
  • To refine the experience
  • To manage tests when access to the end-users is limited

In-person or Remote Study

Probably the most productive and the most challenging user testing method. To get the best results, the test should be carefully planned.

The test plan should contain all the information such as:

  • Testers’ criteria, recruitment, and screening plan.
  • Stage of your product and milestones with deadlines.
  • What you are testing and why, metrics.
  • Specific outcomes you are looking to get (e.g., why players drop off on level 2).
  • The methodology you are going to use.
  • Scenarios and tasks.
  • Questionnaires and when during the course of the test they would be implemented.
  • Error and severity assessment plan.
  • Logistics.
  • Documentation like consent forms and NDA.

How to conduct it: For in-person user testing, you need to bring the testers into your office or wherever your VR setup is, physically, which adds some serious time and maybe even budget constraints.

For in-person testing, you would need a full setting, an office or a room with a set up that includes:

  • An HMD you are testing with (Oculus Rift, Quest, HTC Vive, etc.)
  • Camera for external recording or your phone with a tripod to film the participants, their behaviors, movements, and reactions, as well as sounds and comments.

Since in VR we can not directly study the participants’ view, we should record what they see. You can use Oculus Mirror or VR View if you are on Steam, to mirror the VR experience from the headset onto the desktop and to record it using OBS, there is also an Open VR plugin for OBS to record an eye of choice and adjust settings. One of the cases, when it makes sense, are games or experiences when you have to aim and want to test right-and left-handed people.

Another option can be built-in video recording and even live streaming which as well should involve in-VR recording for future evaluations. These options are not always useful due to IP considerations. Video recording itself is the whole other topic which we will not cover fully here but there are different options and setups you can use to conduct VR remote user testing.

It’s often recommended to practice a think-aloud protocol. But the cases can vary. Sometimes it’s better to let players relax and just play as they would by themselves while observing them and taking notes. It’s important to not distract testers from the actual test which may happen if they try too hard to not forget to speak aloud instead of just playing the game. Consider all pros and cons and choose wisely.

Always test in the environment and setting that would be natural for your specific software. E.g., if it’s a game it would be most likely played at home, but if it’s a medical training app, it might be used at a University or even Hospital. It can be standing, seated, or room-scale experience. It all will affect the guardian set up, playground areas, and dimensions.

Record both, the gameplay and all actions the user takes in real life, how they move, what they say. It’s really helpful to see the actual players and where they struggle, what they like and don’t like, and when they get tired or feel uncomfortable. You can also include Behavioral and Attitudinal tests - what ppl do vs what they say. E.g., people may say they are fine but by observing them becoming red or pale, or sweating, you can make certain guesses. By observing players’ behavior, you can tell if their feedback is honest. Sometimes people tell the opposite of what they really feel or think because they don’t want to be seen in certain conditions or be judged.

One of the biggest measures of success is that experience is comfortable and easy to navigate, ensuring that a player is able to easily find out what to do inside of VR. Testing learnability is essential.

You may also collect data captured by a wearable device (e.g., Apple Watch, Samsung Gear Fit, Oura ring) to record vitals like heart rate, and blood pressure if needed. It might be useful if you are creating a meditation app, so you can measure if the person is relaxed or stressed.

Before conducting the test don’t forget to gather and document the demographics, as well as info about the hardware set up of each tester such as HMD, GPU, and CPU for the remote studies. Of course, we are talking about more or less experienced user testers in this case.

If you are testing with new users, it’s important to make sure the issue they encountered is not caused by a lack of knowledge about how to use controllers. Make sure you explain it before you test, so you can see that the users are stuck because of in-game settings and behaviors, but not because they don’t know how to push the button inside of VR or how to grab an object. Don’t test the controllers, test your app.

Facilitator

Not only the logistics of the whole test should be planned but also its facilitation, and moderation process if applicable. Create a script with a step-by-step plan and make sure all testers have the same experience, same tasks, spend the same amount of time playtesting and get the same questionnaire. Consistency is really important for user testing.

During in-person tests follow these recommendations:

  • Put participants at ease.
  • Introduce yourself, offer an explanation of the product and any equipment, and make sure they understand.
  • Always treat users as experts.
  • Be impartial, ensure people can be candid with you and are fine to have good and bad comments. You can mention to the participants that you had limited involvement in the design/ dev. if you feel it’s necessary so that they don’t worry about hurting your feelings. Don’t show you are upset if people don’t like something you designed.
  • Look for verbal, and facial cues, and observe. Note everything. Pay attention to health, emotions, and behavior.

Have a script (at least in your head) and be prepared!

Note:

  • Health and emotions: People enjoy expressing positive emotions and excitement over VR, but they are less likely to admit to feeling sick, very often we find out about it later, even the next days after the test was conducted. Sometimes new users think it’s normal if it feels uncomfortable.
    Sometimes users tell what You (as they think) want to hear, or are afraid if they do something wrong. Watch out for signs of discomfort to make sure they are ok.
  • Environment: Pay attention to the surroundings, set up everything having enough space from the walls and other objects, and ensure there are no obstacles or hazards near the participant.
  • Hygiene: Prepare wet napkins, alcohol sheets, and you can also use Surgical Dressings (patches you can get in Walgreens or anywhere, they are cheap) as a protection, it sticks to the foam.

Devices: Cool down headsets and mobile phones between sessions, charge headsets and batteries for controllers, and charge mobile phones if you use them in your testing session.

Documents

Another consideration when testing is to prepare the documents needed for tests such as Tasks. People will forget the task as soon as they put a headset on. So spend some time writing the tasks to make sure they are short, clear, and not biased - the words are chosen carefully.

Don’t forget to ask the participants to sign the consent form before the session, and maybe other documents relative to health disclaimers, or NDA, if needed.

Documents to consider:

  • NDA
  • Consent forms
  • Ethical Guidelines (review if needed) - for example, when testing with children or people with different abilities, check some psychological specifics to make sure you comply with all the possible situations.

    When testing in person, you can use an app like Hello Sign to let people sign electronically from their phone or tablet and receive an e-mail confirmation.
  • Questionnaire after a task or in the end (this is a longer topic as well which we won’t cover in this article but make sure testers fill it out right after completing the test when their memory is still fresh.

And you are all set!

VR Analytics and Heatmaps

Another method you can use is Analytics. Among the tools to measure performance and player engagements are Unity and YouTube, and they both have VR Heatmaps. For YouTube, the video must have 1000 views to enable it, and be 360. So it might be useful to publish some VR scenes on YouTube to receive more statistics. At the time of writing of this material, Unity required to have at least Plus license.

You can measure player engagement, where did people look, how frequently they change position, and use the interaction methods. Discover the most viewed areas of the experience and how long people are looking at a specific part of the video. This method is more suitable for launched applications that already have a user base.

Test report

Once you’ve done testing, don’t forget to document all your findings, even if it’s not what you were actually looking for. If you discover something that you were not actually looking for during your tests and found out a completely new discovery - get happy about it, every new discovery is a win!

Document all the Research Stages to compare and evaluate later. Documents in any medium you’d like - write, create a gif, or a video.

You can also create a UX Report in the end when the analysis is made if you want everything to be accurate and organized.

Professional VR Testing Services

Professional services that specialize in testing would be the most efficient way if you are looking for serious measurable results like sales increase, better reviews, higher user retention, discovering new features to build, and so on. Depending on project goals and complexity one testing round may take from 1 day to 3 weeks.

The deliverables at VR Oxygen are usually:

Comprehensive test report, including testers’ profiles, tasks, playtesting videos, and questionnaire results. Feel free to contact us and learn more.

Analysis and Evaluation

After the tests are done it’s time to study the findings and make some hypotheses. You can create affinity maps or diagrams to help you sort out the discoveries, so you can see if patterns appear.

For example, who and why liked or disliked something. Or connect why they disliked based on what you’ve learned about these participants.

It’s always useful to evaluate app performance (Load Time, Frame Rate, and if there are any drops, lags, bugs), comfort, 3D Art, UI and navigation, gameplay, and game mechanics as well.

When doing tests you can ask test participants to rate the app or rate some features of it. Later you can find out The Net Promoter Score. it’s an index ranging from -100 to 100 that measures the willingness of customers to recommend a company’s products or services to others. It is used to estimate the customer’s overall satisfaction with a company’s product or service and the customer’s loyalty to the brand.

You can rate different components of an app and usability objectives on 1-5 or 1-10 scale depending on what you rate and its complexity. The overall app rating should be on a 1-5 scale as it usually is (like in App Store, Google Play, Oculus Store, Steam, etc.).

Finding both, what people like or dislike is beneficial, because it helps to iterate and move in the right direction.

Findings

Let’s check some of the findings now. These are from VR Oxygen’s research a while ago when the goal was to find out what people do when placed in VR, specifically, first-time users. How they use certain features, their feelings, and thoughts.

People don’t turn around/ look around for cues.

At several expos, VR Oxygen demonstrated Cardboard VR app which was selected as one of the favorites at that time, because of its beautiful graphics. Some of the discoveries were surprising at that time.

Even when people were told that it’s a 360 experience where they can move around and look around, they wouldn’t. They would stay still and wait, some would turn head very slightly to the sides, carefully. And it was because they were not used to it. New Users also didn’t look down, even a little, but looked in front of them.

In this situation the Hypothesis is obvious:

Creating clear visual and audio cues at the very beginning and positioning the UI and important elements in front of the viewer will explain players they can interact with the environment, will guide them. If a scene is engaging, with clear directional cues, more likely people will turn around and explore the full 360. Getting people to understand within the first 5 seconds that they can move around will help to keep them engaged.

People assumed they could perform any gestures, tried to manipulate all the objects.

Some people did that or something similar after encountering that items are interactive in VR, so they tried to manipulate all the objects that were present in the environment. The others assumed that if it’s VR they would see their arms there. They tried to grab things.

In this situation the Hypothesis is obvious:

All the objects should be interactive and useful, and behaviors - are easy to understand in order to support expectations. Creating interactions based on real-life affordances and people’s mental models will provide a smoother learning curve for New Users and may help to predict behaviors.

Tips

  • Always keep in mind the context, and map a User Journey to see an end-to-end experience. Don’t rush or start the experience automatically, make the user control it. Agency is important.
  • Provide users with clear feedback on what’s happening, and how they are doing.
  • Use omnidirectional sound in general at all times.
  • Recruit the right people for tests.
  • Consider ergonomics (Avoid interactions that can cause “Text Neck”)
  • Make sure that labeling is clear for new users

Usability testing is far too important to be neglected or postponed, it’s really important to start testing early, iterate, and test again until the best solution is found.

Join VR Testing community. More materials about VR Design and development can be found on our YouTube channel. Follow and join us on Twitter, Instagram, LinkedIn, and Twitch.

Stay tuned!