What's the reality of an Immersive Designer?
A starter interaction for those who don't know me, so you can figure out who, why, and how I am.
Greetings from the crappy drizzle-filled skies of Manchester UK, famous for being crappy, drizzly, and having some music bands and football teams. I’ve been living here for 26 years now and it’s always been this way.
Before that I grew up, lived and worked in Sheffield, which is a steel town on the other side of the Pennines, the spinal ridge of northern England which comprises a bunch of picturesque Wallace and Gromit -style villages, hills and water features linked by snaking, dangerous winding country roads that frequently can’t offer any overtaking opportunities. A fairy tale commute that takes fucking forever to traverse. As the crow flies, I only relocated by 33 miles. But as the car drives, I moved around two hours away.
I commuted back and forth across that pleasantly accursed range of bijou-mountain-ettes every day for years and I suspect that the long, drawn out but often enjoyable journey heavily influenced how I tell a story, make a point, or write a newsletter. Apologies then that we’re three paragraphs into our journey and you still know nothing about me or what this newsletter is about. But at least now you know that it’s a really crap idea to commute every day over English Pennines. Knowledge is power. I’m here to help.
So - Hi, I’m Jed. Hey there. In my work I’m an Immersive Design Specialist who runs a consultancy working with private clients to help them ‘game up’ their XR abilities, where I try to help them improve their immersive storytelling, create more immersive interactions, intelligently apply the right gameplay elements to enhance immersion and attachment, plan and build smart demos, create compelling concepts and pitches based around what works well in XR, and teach them loads of different ways they can do that smartly, efficiently and effectively. There isn’t really an umbrella term for what I do so the best I’ve ever managed is ‘an XR consultant who’s an immersive design specialist’ and then to reel off the above. If you know a more concise descriptor please get in touch, because currently all my business cards need to be printed on A3 sheets.
How do I know about this MR and VR stuff? Another long and winding tale but I’ll take the bypass, as we’ll get there quicker. Before I went independent back in 2017, I worked for 12 years at Sony PlayStation as a Game Director, and was involved in a whole bunch of fun things including the MotorStorm series and Drive Club. Sometimes I was airdropped in to rescue projects that were delayed or disarrayed. I supported external developers, and mentored teams and students at different universities. And I was also part of the internal Green Light team who helped creators develop new game ideas and pitches to turn them into viable proposals. If you were a PlayStation 3 or 4 developer, or even player, it’s pretty possible that my hand touched your game at some point in it’s journey. Don’t look so worried, I used my clean hand.
In late 2010, while I was finishing up Game Director duties on MotorStorm Apocalypse and looking about for my next challenge, I had the good fortune to encounter the very earliest version of Project Morpheus, a skunk-works VR headset that was being created by the 3D gaming team that would eventually morph into PlayStation®VR. It was built from a head massager, a move controller, some solder-heavy circuit boards, two suspended iPhones for screens, and some lenses from Specsavers opticians (because they did, and still do, offer two-for-one on lenses. Result.) It looked for all the world like Doc Brown’s crazy helmet from Back to the Future, but inside it was something that worked even less reliably.
It was the crappy ol’ Unity ‘Island’ free demo - but in VR! It was controlled with a PS3 controller, and it pretty quickly made me feel giddy running up and down the pathways and hills, and then pretty quickly made me feel hugely nauseous shortly after. This wasn’t my first time using VR, but it was the first time in around 6-7 years since I’d visited Disney Quest in Florida or the Trocadero arcade in London, and this didn’t even need me to queue. But it was basic and not like the VR experiences of today (unless you really go digging for crap in the depths of the Steam or Meta stores). There was no sense of deep immersion, A complete absence of anything that we’d now regard as standard techniques to improve the user’s comfort. And it ran at a variable frame rate between 25 and 55 fps depending on which direction you turned your head. In a lot of ways, it really was the most basic and uncomfortable raw-dog incarnation of a modern VR experience, like if your first experience of driving a car was by rolling in an out-of-control brake-less fireball down a steep hill. It was rough but it was a siren call to me. An ugly mermaid on a bird-crap covered rock, sure, but with a haunting voice like Annie Lennox, singing a song of promises that seduced and lured me. I wasn’t pulled in by the immediate ‘WOW!’ that today’s (lucky) VR first-timers get to experience. While kind of impressive it was pretty grotty to actually play. I was fascinated, though, by the sizeable list of things that were clearly wrong with it, that didn’t work, didn’t feel natural, didn’t feel nice, and would never happen in a 2D game. It was full of new kinds of unpleasantness.
We had a good chat. I asked what the team - PlayStation developers, remember - had done to tune it all to make it feel more ‘delightful’, they admitted that they had tried a few things but really didn’t have a clue about how to manage the … experiential side. I sensed we were getting up to 88mph here. What they really needed, they told me, was to find a game designer who was up to try and figure it all out and see if there was any promise in pursuing the idea further for PlayStation gaming. It would be, as they put it, pioneering steps in a new entertainment medium. It might be long, hard work, but hopefully - and they used this phrase - I could define a route between the real and virtual worlds for PlayStation gamers. Woah.
Of course, I couldn’t resist. And - this is heavy - It was important to me to make sure that route was as pleasant, straightforward, and direct as possible. No meandering. I’m a UX freak who thinks the aim of every software journey should be to make all users feel like they’re making the journey by water slide. And I’ve long been frustrated that gamers live in a coded world where previous experience and gaming history are a must. Those of us who grew up as videogamers while videogames themselves matured have built a lifetime of expectations and understanding about the medium, and out of that has come a tendency for developers and hardware manufacturers to rely on players having, y’know, played plenty of other videogames before. But everyone starts somewhere. We don’t remember our first steps as babies but that doesn’t mean they’re not something important and to be provided for and even celebrated. Videogames have become an exclusive, elusive club unless you’re already a member. If you’re coming from the outside and want in, you need to have been either born into a gaming family, raised by an iPad, or prepared to bumble your way in blindly through brave, blundering force-of-will when you’re older. I believe that lots more people would enjoy videogames, and lots more people would play videogames, if there were easier, standardized ways to onboard players into that world, and clear pathways to learning the necessary hardware and software knowledge that many people feel is such a big barrier from the outside looking in.
Put it this way. I’ve been trying to learn piano and guitar my whole life and find them hard, but both of those instruments are easy to figure out in terms of how they work and what they do. It’s not hard to make them ‘do noises’, and I am living testament to this truth. But it is hard to play them well. The onboarding is easy, and the skill ceiling high. By contrast, a videogame controller is infinitely more complex to understand because it’s got twiddly sticks, many buttons with symbols on, touch pads, bumpers, triggers, a dpad, lights, sounds, a built in microphone, and bloody tilt controls too, all of which have no obvious physical or mechanical context attached to them - unlike strings, frets, or keys, you can’t see and hear what the thing does unless the software lets you. And how any of those things work entirely depends on the singular context of each game you’re playing, and probably will even be different depending on what you’re specifically doing at that moment in the gameplay in that specific title. With videogames, the onboarding is always set to ‘hard mode’ even if you’re playing your game on ‘beginner’.
Really, the essential knowledge for any new video game wannabe to even start to feel they can engage at a basic level is to understand how to interface with the device itself, and that’s something that platform holders like Sony, Microsoft and Meta have historically struggled with. Nintendo, not so much; they consider this stuff more deeply from limiting the number of controller buttons, designing for both left and right handed players as default, and adopting the design ethos that any one of their in-house titles might be somebody’s first ever videogame, and should behave as such. One of the featheriest feathers in my cap from my Sony days was getting the Japanese PS5 team to consider adding a deep help feature as a standard to the console, and surfacing it for new users, integrating it smartly to provide help for many aspects of system operation and game progression. It got met with plenty of laughs when I suggested it as a potential headline feature, but I pointed to Nintendo’s many patents in this area and the number of YouTube videos for both game help and, more crucially, onboarding new gamers and teaching them the basics of the controller and the console. Their arguments that gaming help and game walkthroughs are all over the internet already was also proof, to my mind, that lots of people are going elsewhere, seeking help for various questions that the console itself isn’t answering authoritatively. Why make players have a second device open on a Wiki page when they’re trying to play a game or perform an admin function, how does that help us immerse them? I think I framed it okay but figured it was a no-go. I’d been away from Sony for a number of years before I finally got to try out the retail PS5, and was surprised and delighted to see loads of the stuff I’d talked about had made it in, even though they’d taken a hard pass on my suggestion of having a big red flashing ‘help me’ button on the controller shaped like a confused Crash Bandicoot face. Big win, bigger loss - yes, but I suspect maybe they’re saving it for PS6.
Meanwhile, back in the VR demo on that shitty island (see how hard it is for me to keep a straight line on a journey?). Right there and then I saw the potential that consumer VR really could bring on an entirely new gaming paradigm (such a fun word to type, and a hilarious one to hear people say out loud sometimes), and as such, one that had no choice but to somehow educate and train new users in it’s usage and conventions because there had never been anything else like it before in the consumer space. I felt a strong calling from that, I was interested in how we could make sure that this new user experience was handled properly, just as much as the curiosity to explore the potential of VR more fully.
A week later I was on the team. A month later we were building all sorts of internal demos and exploring possibilities, and I was reading a lot of literature and papers from the past 60 years of military and academic research and trying to figure out how much of it was still relevant (it turned out to be some, but not that much, because the tech had moved on so far in the meantime). 18 months later Palmer Lucky released the Oculus DK1 (Sony had offered him a job when he started his kickstarter, he said no) and we finally got to see content that other people were building. I’d written a list of about 200 things that might be cool to experience in VR and building them was taking a long time. Thanks to the DK1 we got to experience a wide variety of cool things in VR much, much more quickly, and see a lot more of other people’s ideas, what worked well for them, and what didn’t.
Plenty of moments of wonder, plenty of learnings, but also plenty of friction. As Dickens would put it today, the besty times, but also the worsty times too. Getting to the point where you could experience anything was a pain. Getting head and controller tracking working consistently was a pain. Getting hardware to play nicely with it was a pain. Getting software to launch was a pain. Every demo and experience was a different journey and demanded a different route, fiddling with these settings here, disconnecting that device there from Windows, making sure the magnetic-tracking for the controllers was set up right, making sure there was room to swing your arms or duck out of the way of an errant polygon that might fly at your head. And every experience worked differently, and had different ways of doing things - setting your height, moving, interacting with objects, even starting the game (there was a lot of coming and going between VR and looking at the PC screen back in the real world — there were no virtual desktops in those days). So there we were, as I feared. Amazing sights to be seen and experienced, but clearly an unnecessary, long and winding road getting there. Not fast, not straight, not uniform, not pleasant.
So that was where I kind of started. We crawled our way out of VR clueless-ness and over the next few years we figured out how our developers could and should be creating immersive experiences, imbuing the PlayStation values and player-first ethos into these new VR experiences. I developed the internal best-practice documents (which were smuggled out to Oculus when they poached one of our coders - but that’s another story!) and gave many talks and presentations to our internal and external development teams on how to build great VR experiences, and to do it in a ‘PlayStation’ way. I covered techniques, design approaches, development strategies, and everything we could figure out about how players could avoid discomfort and sickness in VR. I traveled around the US, UK and Europe meeting and working with developers, attending conferences and occasionally advising about developments in the medium at a senior boardroom level. My best practice teachings for Comfort and Gameplay in VR were propagated through lectures far and wide, and the written versions came as part of the standard documentation for new PSVR developers to introduce and explain the concepts and complexities required in developing for VR. That’s something I’m very proud of -- just about everything I advised people to do has since become just standard operating procedures for VR. And to my endless delight I got to play and feed back my thoughts, advice and opinions on (just about) every PlayStation®VR title developed until 2017 at different points in their development cycles all the way through to launch, which was like a dream job for me.
I still do largely the same thing now. I’ve been fortunate enough to have helped somewhere over 140 XR projects (including MR, AR, Location-based experiences, theatrical and storytelling projects) to ‘git gud’ in a variety of ways. I’ve experienced probably thousands of Immersive works at this point, and yet VR still gets me hot in all the right ways. I still get those ‘WOW’ feelings regularly, I still get excited by the new advancements in the tech, the possibilities they open up, and the excitement of my clients who are taking advantage of these new opportunities. I still count myself as a passionate traditional videogamer and always will - represent! - but VR and MR’s trajectory and possibility space in particular feel much more vital and exciting these days. Where lots of videogames feel like I’m re-treading familiar paths to see shinier versions of sights I’ve seen before, VR and MR feel like they’ve got a lot of unknown territory to explore, and lots of secrets waiting to be discovered. As Mr Dickens would say, ‘Excitey!’
I love having seen the way this current wave of consumer VR has been exploring this new frontier, pushing out in every direction and still not finding the limits. How far has XR come? Maybe only about 30-odd miles as the crow flies, up and down the spine of existing ideas around interaction and immersion, but it’s an exciting and fascinating journey and every time it’s different and unique. Not the same pretty roads and the same journeys, but exploring new routes and new ideas. I love that. And so I’m someone who happily spends hours meandering through this landscape to see the sights and discover the discoveries, mapping out the new ideas and things people are trying, and always looking at the ways they’re doing things brilliantly, and the ways they could be doing them better. It’s fun and fascinating to me, and I love to talk about it with others.
So that’s the who, the how and the why of it. I’m an XR guy who gets excited about the possibilities of Immersive technologies but remains realistic about the realities of them. If you want to join me as I meander all over the XR map, encounter crazy new experiences and, y’know, talk about them probably a little too much, then please subscribe and stick around – I’m happy to be your driver and tour guide, but you’re in charge of bringing the snacks.
Jed Ashforth has been working in the games industry since 2002, and working with VR, MR and AR apps since 2010. He has too many headsets and not enough time. If you’re interested to find out more (or even bring him on board to help you) you can reach him, and check out more of his work and articles, at www.realisedrealities.com.
This here Realised Realities newsletter hopes to one day grow up to be a responsible reader-supported publication, but for now put your money back in your pocket / purse / sock / Swiss holdings account, it’s no good here.
To receive new posts and support my work, make sure you’re subscribed, and please feel free to share this newsletter with anyone you think might dig it. Seriously — that would be awesome.
Great! You are the designer and I am the developer on the immersive side ;)