VR Playtesting from Alpha to Early Access and Beyond

Case Studies › Crystal Conquest

VR Playtesting from Alpha to Early Access and Beyond

How Mystic Interactive ran 26 test rounds — from alpha to live ops — tested with new and returning players, shipped Crystal Conquest on schedule, and user testing became part of their development process.

26Test Rounds
12+ moAlpha through live ops
New + returningParticipants
Every stageTesting built into how they ship
About the Studio

Mystic Interactive

Mystic Interactive is a small independent studio building Crystal Conquest — a free-to-play VR strategy game where players harness the elements in PvP multiplayer combat. The game supports single-player and multiplayer.

Crystal Conquest

The game runs on both Meta and Steam, supporting standalone Meta Quest as well as PCVR headsets, and features multiple game modes (1v1 Duel mode, 3v3 MOBA mode, and a PvE Gauntlet mode), a soft-currency economy, and a new character pipeline that was actively expanding throughout testing.

Where they started

When Mystic started testing on the platform, Crystal Conquest was in Alpha — not yet public, with no player community to draw from. This is the hardest moment to get fresh-eye feedback, and also the most critical.

They needed to test everything continuously — experience across devices, build distribution flow, new characters, economy systems, multiplayer networking and user experience, and onboarding throughout their entire development cycle as a strategy to de-risk their Early Access release.

The Challenge

Coordinating external VR playtesting is harder than it looks

Mystic needed participants with particular VR headsets and hardware specs, matched to a precise profile, organized into synchronized multiplayer groups — often scheduled and run without the team needing to be present at all. Sessions ran within a defined context, with participants completing tasks, so results were actionable at every stage. That's not something you can coordinate through a Discord community, even if you had one.

1
Multiplayer coordination at scale

Testing multiplayer, especially 3v3 combat, requires the right number of participants online simultaneously — each matching a specific profile, headset, and hardware requirement. The team needed this to happen reliably, without managing it themselves.

2
Precise participant profiles that varied test by test

Requirements varied — sometimes any Meta Quest headset or PCVR, sometimes all Meta models as standalone, sometimes PC specs strict enough to guarantee the game would run. One multiplayer playability test required participants to be physically located on the US East Coast. The platform matched for all of it.

3
Cross-platform complexity

Crystal Conquest runs on Meta Quest headsets and PCVR headsets — distributed via Meta, SideQuest, and Steam — each with a different install and distribution flow. Testing across all of them, sometimes simultaneously, required infrastructure — not just coordination.

4
No public players yet — by design

Studios at this stage can't test with their own community because they don't have one yet. But this is precisely when first-time player perspective matters most: before habits form, before players learn to work around rough edges.

5
New vs. returning players see the game differently

Testing both groups separately — often on the same day with separately matched cohorts — surfaces onboarding gaps that experienced players miss entirely.

"Our testing approach was uncomplicated — because it didn't really exist. We'd make changes, test in-house, and try to recruit community members to play. This seemed to work, until it didn't.

We made changes to certain spells just before our Steam Next Fest demo, ran our usual process, and found no major issues. We had a marketing plan with ads and paid influencer videos leading up to the demo — and this is when we realized our testing was failing.

Contractors making our trailer and marketing videos, and influencers getting familiar with the game, were all hitting networking bugs we'd never caught — because host and client experience the game differently. That was the wake-up call: our testing was insufficient and was wasting time and money in other areas."

— Sean McBeth, Founder, Mystic Interactive
The Journey

One year. Twenty-six test rounds. Every major feature tested

From a small alpha test round to larger synchronized multiplayer sessions — and an Early Access launch six months later. A year in, they were on their 26th test round, following the same principle: small, specific, sequential tests, with new and returning participants tested separately to see what each group experienced differently.

1
Month 1
First Test Round — Alpha

First external participants. UX and interaction design validation across PCVR headsets.

2
~2 months in
Expanding to Meta Standalone

Expanded to Meta standalone and tested build distribution flows — including sideloading via SideQuest — across all Meta headset models simultaneously, with participants matched to the right prior experience profile.

3
Pre-launch sprint — Beta
Multiplayer Validation at Scale

Multiple simultaneous test rounds, including larger synchronized multiplayer sessions. Cross-platform Meta and Steam testing — including a 3v3 session specifically designed to validate whether SteamVR players could play with standalone Meta and SideQuest players in the same game. Test rounds for new and returning participants run in parallel on the same day.

4
Final weeks before launch
Economy, New Characters & Weapon Mechanics

Soft-currency system and purchase UX, new characters, and weapon-grabbing mechanics validated across both platforms. Test rounds for new vs. returning participants. Platform-specific issues surfaced and resolved before launch.

5
~6 months in
Early Access Launch

"We are so grateful to you and your testing platform for helping us get here." — Eniola Suley, Producer, Mystic Interactive

6
Post-launch
Live Ops — New Characters, Economy & In-Game Quest System

Testing continued through live ops. Voice chat, new characters, economy systems, and the in-game quest system tested across Meta and Steam — with new and returning participant cohorts throughout.

7
12+ months in
Ongoing — New Character Pipeline

The testing cadence continues. Small, specific, sequential test rounds tied to each new feature — the same approach that's worked since month one.

The Testing Approach

From small validation test rounds to large synchronized sessions

Test rounds scaled with the question being asked — smaller ones to check if something basically works, larger ones for full multiplayer validation. Each test round had specific goals, with participants completing tasks that gave sessions a defined context, and specific criteria determining who could participate.

The approach was methodologically deliberate. Test rounds were designed to isolate variables — testing the same feature under different conditions simultaneously, combining quantitative data collection with qualitative feedback. New and returning participant cohorts were tested in parallel, sometimes on the same day, including both "happy path" scenarios and cases where participants intentionally stopped short — revealing how the experience held up when things didn't go as planned. New characters and mechanics were tested systematically, one by one, with structured feedback collected on each element independently.

Headset and hardware requirements evolved with the game. Early test rounds used broader criteria. Later test rounds got specific — particular Meta models, standalone or PCVR only, or a mix of both to cover the full device range. PC hardware specs were part of the criteria too, tightening as the game's requirements became clearer.

Single-player and most multiplayer test rounds were fully hands-off: the Mystic team set requirements, participants were matched, test rounds ran, and results were delivered into the dashboard. For sessions where the team wanted to participate alongside players and moderate, the option was there. Either way, the platform handled group synchronization with a managed replacement flow — so sessions ran reliably even when individual participants dropped off.

What the platform enabled

  • Participants matched across a detailed profile — headset, hardware, location, play history, genre preferences, and other requirements — per test round, every test round.
  • Synchronized multiplayer groups with a managed replacement flow, so drop-offs didn't derail sessions.
  • Autonomous single-player and multiplayer test rounds with video recordings and survey results delivered into the dashboard.
  • Meta and Steam builds tested simultaneously, so platform-specific issues surfaced immediately.
  • Test rounds for new and returning participant cohorts, making first-time experience visible at every stage of development.
  • Responsive support wherever needed, always on standby.
What Changed

Testing didn't just find problems — it drove design decisions

Across test rounds, participants flagged issues and patterns that led directly to changes in the game. Here are some examples of what participants experienced and what Mystic changed as a result.

Bow aiming felt unnatural Participants expected the bow to aim in the direction they were physically pointing their controller, as it would in real life. The game had a fixed aiming direction instead.
What changed The bow's physics were overhauled from a projectile spell model to a physics-based system that tracks the player's physical movement. In a subsequent test round, a participant who does archery complimented the bow mechanics for mimicking real-life physics.
Voice chat was confusing in live multiplayer In a multiplayer session with unfamiliar players, participants couldn't tell who they were hearing, whether their microphone was working, or which audio was coming from teammates vs. enemies. There were no clear audio indicators.
What changed The voice chat experience was completely redesigned. Teammates now have mic runes on their wrists — brought to the face to initiate a conversation. The rune also provides a visual indicator of the microphone status being on/off. A functional fix became a VR-native interaction.
Weapon grab was inconsistent under gameplay pressure, with unclear cooldown UI Participants struggled to grab weapons consistently in the heat of combat, and couldn't tell from the UI when a weapon was available again after throwing it.
What changed The weapon grab system was redesigned. "Spawn then grab" was replaced with "spawn with autograb," removing the friction mid-combat. Weapons were also moved to a back holster rather than bound to spell buttons, making the weapon grab interaction during the fight more intuitive. Subsequent test rounds confirmed players loved the change.
Quest completion feedback was unclear Participants completed actions they believed fulfilled quest objectives — healing damage, winning matches, completing tutorial steps — but received no confirmation that the quest had registered. Some completed the correct actions, but the system didn't track them.
What changed A more prominent progress bar was added to give real-time feedback on quest completion. Registration inconsistencies and remaining edge cases are in the backlog, tied to the upcoming Daily Quests release.
The Fountain Rune scroll triggered the settings menu accidentally Participants trying to grab the fountain icon on the map were accidentally hitting the settings icon instead because the two UI elements were too close together physically.
What's planned While not yet implemented, the finding directly informed the redesign plan. Building on the wrist rune concept introduced for voice chat, the solution will be a "transport rune" on the right wrist, eliminating the map interaction entirely and replacing it with a more intuitive physical gesture.

What the video recordings revealed

Survey responses capture what participants notice and choose to report. Video recordings capture what they actually do, including things they don't think to mention or don't even realize, which can be even more revealing.

Two memorable examples of how observed player behavior uncovered insights that looked innocuous at first, but ended up tied directly to patterns the team had seen in their reviews — and couldn't make sense of, because reviews lacked specificity to act on — playtest videos showed the "why".

The map visibility problem

Player reviews consistently mentioned in the MOBA that the map felt small and didn't have a lot of lanes — feedback the team acknowledged but couldn't understand what players meant, because, for a 3v3, based on best game practice, they expected it to be sufficient.

Watching playtest videos across multiple test rounds related to other features, a pattern emerged: every participant ran straight toward the center of the MOBA map. Nobody ever tried to take the first crystal near the fountain, even though it's closer and advantageous early in the match. Nobody explored the enemy camps either.

After watching several videos, the reason became clear: large boulders were blocking sightlines. Players literally couldn't see those areas of the map. The team changed the environment, lowered and repositioned certain rocks. In the next test rounds, roughly 20% of participants started navigating toward that crystal — and telemetry confirmed that both playtesters and real players began moving through different parts of the map more.

The playtesting videos also revealed that Aeromancers could essentially fly across the whole map undisturbed, even over the invisible walls designed to keep the game controlled. The team fixed this right away before too many players exploited it to dominate matches.

Players didn't know they could block spells

Videos showed that most participants never used the spell blocking mechanic — meaning they took more damage, died faster, and found the game harder than intended. This connected directly to recurring reviews consistently mentioning people's dissatisfaction with enemies being too difficult and players dying too quickly. The root cause wasn't game balance — it was that a core mechanic was invisible to new players.

The team added a dedicated blocking tutorial and introduced new player quests specifically designed to teach spell blocking. The blocking mechanic is central to the game's design, and playtesting videos made the gap visible in a way that reviews and surveys alone never had.

Shaped by Real Studio Needs

Infrastructure that adapts as the game evolves

"You were always willing to answer questions, to offer best practices and even added features quickly in responses to some of our asks. We really valued this and it was very helpful to have such amazing partners. It made all of us at Mystic feel so confident and relaxed about our testing strategy. We're fairly confident that without VRO2, we wouldn't have achieved our goal of getting to early access this year with such an ambitious game."

— Sean McBeth, Founder, Mystic Interactive

VR Oxygen continues to evolve based on real studio needs. One recent addition: the ability to run a single test round across multiple builds, platforms, and headset models — e.g., Meta Quest standalone, Meta Quest as PCVR, and PC VR headsets — simultaneously, without needing separate test rounds for each.

Results

Launch was the midpoint

Mystic Interactive launched Crystal Conquest on Early Access roughly six months after their first test round on the platform. What came after launch is the clearest proof it worked: testing didn't stop. They continued running test rounds through live ops — new characters, economy systems, voice chat, the in-game quest system — with new and returning participant cohorts throughout.

Across 26 test rounds on the platform, the approach scaled but the logic stayed the same: small, sequential, purpose-specific test rounds tied to each new feature. Testing became part of how they ship.

26Test rounds over 12+ months
Alpha → live opsTesting built into every stage
New + returningParticipant cohorts throughout
Is This You?

Does Mystic's starting point sound familiar?

If any of this resonates with where you are right now, you're ready to start.

1
You have a prototype, vertical slice, or early build and you're curious what real players make of it

Fresh-eye external feedback is hard to come by at this stage, especially without a player community to draw from.

2
The game is complex enough that internal testing alone isn't enough

Especially if it's multiplayer, has complex mechanics, an economy system, or onboarding that needs to work for someone who's never seen it before.

3
You're iterating fast

Shipping new builds, new characters, new systems — and each one needs validation before it goes live.

4
You're building for VR

And searching for the right target audience, coordinating headsets, builds, and access methods across platforms, and organizing tests is a real burden you'd rather not manage manually.

How Mystic Interactive ran 26 test rounds — from alpha to live ops — tested with new and returning players, shipped Crystal Conquest on schedule, and user testing became part of their development process.

‍

About Mystic Interactive and Crystal Conquest

Mystic Interactive is a small independent studio building Crystal Conquest — a free-to-play VR strategy game where players harness the elements in PvP multiplayer combat. The game supports single-player and multiplayer. 

The game runs on both Meta and Steam, supporting standalone Meta Quest as well as PCVR headsets, and features multiple game modes (1v1 Duel mode, 3v3 MOBA mode, and a PvE Gauntlet mode), a soft-currency economy, and a new character pipeline that was actively expanding throughout testing.

‍

Where they started

When Mystic started testing on the platform, Crystal Conquest was in Alpha — not yet public, with no player community to draw from. This is the hardest moment to get fresh-eye feedback, and also the most critical.

They needed to test everything continuously — experience across devices, build distribution flow, new characters, economy systems, multiplayer networking and user experience, and onboarding throughout their entire development cycle as a strategy to de-risk their Early Access release.

‍

The Challenge

Coordinating external VR playtesting is harder than it looks.

Mystic needed participants with particular VR headsets and hardware specs, matched to a precise profile, organized into synchronized multiplayer groups — often scheduled and run without the team needing to be present at all. Sessions ran within a defined context, with participants completing tasks, so results were actionable at every stage. That's not something you can coordinate through a Discord community, even if you had one.

1 Multiplayer coordination at scale

Testing multiplayer, especially 3v3 combat, requires the right number of participants online simultaneously — each matching a specific profile, headset, and hardware requirement. The team needed this to happen reliably, without managing it themselves.

2 Precise participant profiles that varied test by test

Requirements varied — sometimes any Meta Quest headset or PCVR, sometimes all Meta models as standalone, sometimes PC specs strict enough to guarantee the game would run. One multiplayer playability test required participants to be physically located on the US East Coast. The platform matched for all of it.

3 Cross-platform complexity

Crystal Conquest runs on Meta Quest headsets and PCVR headsets — distributed via Meta, SideQuest, and Steam — each with a different install and distribution flow. Testing across all of them, sometimes simultaneously, required infrastructure — not just coordination.

4 No public players yet — by design

Studios at this stage can't test with their own community because they don't have one yet. But this is precisely when first-time player perspective matters most: before habits form, before players learn to work around rough edges.

5 New vs. returning players see the game differently

Testing both groups separately — often on the same day with separately matched cohorts — surfaces onboarding gaps that experienced players miss entirely.

‍

“Opening testing to remote players from anywhere in the world allowed us to get a wide range of perspectives on the project, with feedback coming from people with varying levels of vision.”
- Jazmin Cano, Accessibility Product Manager

‍

Observations and feedback received from the playtesters with blindness and low vision helped Owlchemy to prioritize iterations, validate hypotheses, and uncover new VR accessibility features to integrate into Cosmonious

‍

‍

Image credits: Mystic Interactive and Crystal Conquest

‍

‍

‍

Get in TouchGet in Touch