VR, AR + MR: Designing the New Reality Experience

VR, AR + MR: Designing the New Reality Experience

UPDATE: this post was refined and published on InVision’s blog on March 6, 2017. You can read that version here

Seven days after the release of Pokémon Go, more than 65 million people were playing it. SIXY-FIVE MILLION PEOPLE.

How many people live in your whole town? County? State?

Here’s some perspective: in the whole of New York City: 8.4 million.

Whether we care or not, the success of Pokémon Go is a big statement about how humans are interacting with machines. Not as in the future but as in RIGHT NOW.

Even with the popularity of so many virtual platforms, many of us still struggle with the distinction between Virtual Reality (VR), Augmented Reality (AR) and Mixed Reality (MR).

Let’s define these:

VR (Virtual Reality): VR transports the audience to another world, isolating their sense of sight and sound to immerse them in an artificial environment generated exclusively by technology.

AR (Augmented Reality): AR overlays digital content on top of the real world.

MR (Mixed Reality): MR seamlessly combines virtual and real world objects.

Here’s a quick sample:

OculusRift? VR.

Have a display in your car that projects onto the windshield? AR.

HoloLens? MR.

What does this mean for UX Design?

Usability led us to User Experience which now leads us onward to Experience Design.

the hidden machinery behind how people think about their needs and wantsIn the time I have been paying attention, over the past twenty years or so, being engaged in the application of usability across many contexts and the design principles that optimize it, here’s my take:

The first 10 years –> Usability Design

In the beginning, it was all about reducing friction between the user and technology. Designing anything during this time typically meant the user was an afterthought until the thing was released into the wild and someone (like me) would say, “Oh, it’s not at all user-friendly. What can we do to improve that?” As long as the product met the basic business requirements, it was considered just fine.

The past 10 years –> User Experience Design

On June 29, 2007, iPhones changed the rules completely. In no time at all, a single, agnostic (in the sense that it had no limitations UI-wise from a hardware perspective) piece of technology was built with more focus on the user than had ever been seen before. This was unprecedented but has quickly become the norm. If we are serious about building products and services that are successful, designing devices for users now comes first (also called User Advocacy) and everything else (business requirements) comes second.

Without this level of attention to usability, any device, Website or app is now immediately excluded from the market by the very audiences they were intended for. In other words, people ignore technology that isn’t friendly to use. And they should. Steve Jobs’ vision of the iPhone set an entirely new precedent for what is expected today.

The overall experience of using a product or a service has become the new normal in the context of discussion, design and expectation. Suddenly, on top of being simply usable for whatever task they are intended for, products also had an expectation to be engaging, delightful and aligned with a company’s brand and larger, long-term strategy. Users continue to consistently demand more and come back only when there is a great user experience for them.

The next 10 years –> Experience Design

The best is yet to come. While most companies are still working hard to get user experience right, it won’t be enough in the years to come.

Wearables, voice recognition, suggestive approaches, chatbots, VR, AR, MR, robots, AI all make it clear to see that a lot of them aren’t used in the traditional sense anymore.

For example:

Nest’s self-learning thermostat saves you energy without you having to do a thing.

Siri can send that text for you while you’re driving.

Fitbit’s SmartTrack recognizes and records your exercises for you. You can keep track of your workouts without ever pushing a button.

Based on your calendar and current traffic, Alexa notifies you that it’s time to hop in your car if you’re going to make it to that meeting on time.

The balance is tipping over to the experience side of User Experience. VR, AR and MR are raising expectations to a whole new level. In a year, things will have changed even more.

The future of UX Design

The trend is less User and more Experience. Designers are already shifting focus in this way. So, as a UX designer, how do you prepare for what’s coming? How do you stay relevant?

definition of empathyHow much psychological insight into people do you have? The increasing focus on experience will require a sixth sense into human sensibilities and those who are gifted with deep empathy and social insight will succeed. Being able to anticipate the needs of our audiences will determine our level of success.

Are you a good communicator or do you just think you are? Invest in talking to people, listen, really listen to what they want, what they need and, arguably most important, what they fear.

These are the kinds of skills that will nurture keen anticipatory design methods that deliver the most value to the broadest audiences. Technology should anticipate our audience’s actions and needs. Coupled to improved algorithms, cleaner data and more powerful hidden machinery to process it all, we can bet that anticipatory design is the future.

Get competent in the world of analytics. The tools are improving constantly and they are friendlier than ever to use.

Follow the leaders in the space, outfits like Magic Leap, who are expected to reveal their “Cinematic Reality” technology this year. Good things ahead. It makes for interesting reading, even if you’re not a science fiction fan. This science fiction is real.

Human computer interaction used to be of great interest to me. I hardly care, anymore. This is a field that is changing faster than it can be adequately studied. Look around you. Watch how others use technology. Pay attention to how you use it. These observations and conversations are far more valuable than an academic study of a paper that was published long before the latest hotness came out. Remember, studies that are 6-months old are already at their half-life.

If you are interested in diving into the intelligentsia of it all, follow the work of people like Donald Norman, Susan Weinschenk and Jakob Nielsen. In addition, my finger-on-the-pulse-pal, Sean Scott, also recommends following Mike Enger at Google VR, keeping an eye peeled on what Leap Motion does is also key as they are leaders right now in hand tracking (a key component to get to natural interactions), along with the titans of AR over at Metavision.com. Last, but not least, visit The UX of VR once in a while for a finely curated list of other resources.

The work of these people inspires me to do my best work and I have no doubt they will inspire you, too, into a future that is very exciting and very real, whether it is virtual, augmented or mixed.