Emily Cross and I organized the Bangor Social Robotics Workshop on the Emerging Social Neuroscience of Human-Robot Interaction. It took place on the 17th and 18th of August. We aimed for a small-scale meeting of diverse minds with enough room for discussion, inspiration, reflection, and ultimately collaboration. Social roboticists, computer scientists, psychologists, neuroscientists, lawers, a diverse group of people attended the workshop. This post is a very, very, very short summary using a collection of tweets. So contemporary!
Ish, am I stuck in Virtual Reality? What happened? Did I lose contact with reality? Did I became an automaton? Is it as grim as Black Mirror? Not at all! Only good news. In the last couple of months, I moved from from South Africa via The Netherlands to Wales! New year, new position, new adventure! I started as a postdoc in the Social Brain in Action Lab at Bangor University.
This is Cozmo, my new colleague and office mate. Together with him/her/it, I try to investigate how human social cognition shapes the interaction with his fellow artificial agents and vice versa. So it might well be that the future blogposts will be a little bit more adventurous. Virtual Reality, Social Robotics, very exciting new developments in neuroscience and psychology. ‘The future is here’ as all the trend forecasters will say.
Just don’t stand there, be part of the future! Two events will showcase these grand words.
If you want to know more about contemporary Virtual Reality research in social and cognitive neuroscience, please join us at the International Convention for Psychological Research on Friday 24 March. Together with Sofía Seinfeld (University of Barcelona), I organized the symposium ‘Real Worlds for Virtual Humans: Changing Perspectives’.
This symposium explores how we as psychologists can study, manipulate and enhance social cognition and interactions with Virtual Reality. The talks will discuss Virtual Reality studies on mimicry in individuals with and without autism, the malleability of threat perception, prosocial behavior in life-threatening situations, and the enhancement of empathy.
It will include talks by Antonia Hamilton (University College London), Giorgia Silani (University of Vienna), Sofía and yours truly, and Beatrice de Gelder (Maastricht University) will integrate the four talks and discuss the future of VR for psychology and neuroscience.
If you want to know more about Social Robotics, please join us in scenic North Wales for a social robotics workshop on ‘the emerging social neuroscience of human-robot interaction’. Emily Cross and I organized this exciting meeting that will take place in Bangor from August 17-18.
As interest grows among social neuroscientists, psychologists, and roboticists to understand how humans interact with artificial agents, it will become increasingly more important for constructive dialogue and collaboration across these domains. The aim of this workshop is to bring together individuals working within and across a diverse range of disciplines to discuss the latest findings, current challenges and exciting possibilities in human-robot interaction. Together, we will explore social robotics questions using an interdisciplinary lens and discuss several of the major themes attracting increased research attention in this emerging field, from the technical and theoretical foundations of social robotics to the neural mechanisms and developmental and clinical implications of human-robot interactions. We invite students and researchers from social and developmental psychology, social neuroscience, artificial intelligence, virtual reality and robotics to attend this small-scale meeting of diverse minds
The registration is now open, please visit the workshop website for more information or the webshop of Bangor University to register. This will be a small scale meeting, with only 55 attendees, so act quick and be part of this meeting of diverse minds.
Brilliant times, na?! Stay tuned for more.
 Is Cozmo a he, she or it? Excellent question, difficult to answer. It depends on many factors which I hope to map in the near future.
Right. VR is hip and happening. There is a hype, there are promises, and there is a huge interest. You can see that by eye-balling the Google Trends graph below.
Of course, this is largely reflecting consumers’ eagerness to get their hands on the Oculus Rift or HTC Vive. But there is also an increasing interest in the use of VR in psychology and neuroscience. The number of scientific publications increases every year, scientists talk in the media on the importance of VR (1, 2, 3), and one of the hippest scientific journals, Nature, published two articles on the potential impact of VR (1, 2). Hell, even philosophers are enthusiastic. So are we on the forefront of VR research in psychology and neuroscience?
Yes, no, maybe. In this post I will provide a brief history on the use of VR in psychology and neuroscience. As part of my adventures in Virtual Reality I dived into the history of VR. To use an music metaphor, one can say that we are in the midst of the third wave of VR, with the first and second wave occurring in the nineties and zeroes. In this post I will provide a brief history of VR research in psychology and neuroscience. To do so I compiled a short summary based on several themes. This is of course totally subjective and incomplete, but a nice way to tell history. To summarize, VR has been around for a long time and at the end of the post you’ll know whom among VR scientists the original hipsters are.
The first-wave of VR
When I was writing this blog, Anthony Steed tweeted this paper by Steve Ellis published in 1991. It is a paper on the history of VR. The history of VR. In 1991. C’mon. Just when you thought that VR was solely for millennials. While VR applications were mostly confined around teleoperation, education, training, and scientific visualization, The Lancet published an editorial on the ethics of VR applications in psychology in the same year. Like the Ellis paper, it’s an interesting read. The editorial states that “[a]t one level VR becomes a terrifying instrument of torture, at another a powerful means of education”. The former because “[c]ontinuous exposure to VR will impoverish those aspects of life that determine social development, interpersonal insight, and emotional judgment”. So there you go. Nice starting point. While it was not based on empirical evidence per se, it was a timely editorial that raises important ethical issues for patient studies and can be viewed as the beginning of psychological VR applications.
Treatment of phobias
Mid-nineties, Barbara Rothbaum and colleagues published two case studies, one on the treatment of fear of heights and one of the treatment of fear of flying. These case studies were followed by more formally controlled experiments (1, 2), and extended to other anxiety disorders (e.g., claustrophobia, posttraumatic stress disorder). VR is well suitable to treat fear of heights as I can attest. Standing in the CAVE in a glass elevator speeding to the top of a skyscraper did induce all the ‘right’ physiological and behavioral responses. During Virtual Reality Exposure Therapy (VRET), as it is formally called, an individual is gradually exposed to the object of fear. For example, for the treatment of fear of height the individual is standing on a real, slightly elevated, platform surrounded by hand rails while in virtual reality the person is standing in a glass elevator, on an outdoor balcony or foot bridge (all with virtual railings). The individual explores the virtual environment and if anxiety levels drop, the height increases and so on. Currently, VRET is widely used in the treatment of anxiety disorders. Another related application is in the treatment of erectile dysfunctions, something that I have never thought of. One group of researchers investigated if VR could counteract premature ejaculation and other erectile dysfunctions. They even combined it with neuroimaging (PET). In contrast to your fantasy, the VR scenario did not involve any erotic material. It was only used as a psychoanalytic tool (think a couch and Sigmund Freud).
The nineties also saw the release of several big VR projects on neurological assessment and rehabilitation. For example, stroke rehabilitation or the treatment of movement disorders. As described in these early reviews (1, 2), VR offers several unique features that enrich training and rehabilitation. In VR you can slow down the movement of a ball in order for the patient to catch it, easily and fully manipulate attention as the patient is wearing a head-mounted display, and move a virtual hand based on imagined movement of a paralyzed patient. These projects were the precursors to the ‘paralyzed man walks again’ headlines we’re seeing today. The work in the nineties has also let to NeuroVirtual, a free VR platform for neurorehabilitation and psychological treatment.
Another early VR application in psychology of which I was unaware, is the use of VR in autism research. Since autism is characterized by deficits in social communication and interaction, VR can be used to immersive children in a virtual social setting. In a case study VR was used as a learning aid for children with autism (navigate a street). Around the same time, researchers investigated if it can be used for diagnosis purposes. They created a virtual sandbox in which children were allowed to create virtual landscapes, which were later analyzed to determine psychological and psychiatric problems. While these were feasibility studies, there is a reemerging interest in the use of VR in autism research. Interaction with virtual avatars allow not only the measurement but also the training of aspects of social communication.
In 2009 while studying for my MSc at Utrecht University, Neil Burgess gave a workshop on the neuroscience of spatial memory. In his talk he showed how he used the Unreal Engine to program a VR fMRI experiment. I, as a p > .1 gamer, thought that was pretty cool. VR and cognitive neuroscience go way back. Mid nineties, Jean Decety, of empathy and morality fame, and the late Marc Jeannerod published two papers (1,2) that used VR to study the execution of actions. With VR you can manipulate the movement of the hand to study things that are impossible to dissociate between in reality (the perceptual and kinesthetic aspect of a movement). Besides these and other studies on the human motor system, early studies incorporated VR into the study of spatial memory. In VR it is easy to construct a building or a neighborhood and have individuals find their way in these places. After participants learn to navigate these virtual worlds, they are tested on their spatial memory. Geoffrey Aguirre and Mark D’Esposito published the first fMRI and VR study in 1997, shortly followed by publications (1,2) from an all-star team from University College London, including John O’Keefe (the Nobel Prize winner). These VR studies were not published in some obscure journal. In contrary, the hippest journals in the field published these papers. The use of VR in spatial memory research is pretty much standard these days. While the first studies did not use MRI-compatible HMD, thus VR was used outside of the scanner, recent studies use VR inside the scanner. Complete immersion, complete control.
The second-wave of VR
In the zeroes, the power-houses of VR research in psychology and neuroscience, Mel Slater together with Mavi Sánchez-Vives, and Jeremy Bailenson together with Jim Blascovich, published two high-impact reviews. In 2001, Jim Blascovich, Jeremy Bailenson and colleagues, wrote an article called ‘Immersive Virtual Environment Technology as a Methodological Tool for Social Psychology’ that was published in Psychological Inquiry. In 2005, ‘From presence to consciousness through virtual reality’ by Mel Slater and Mavi Sánchez-Vives was published in Nature Reviews Neuroscience. While I save the details for another time, these papers are highly influential and cited (603 and 583 times, respectively), and have successfully argued for further integration of VR research in psychology and neuroscience. These articles provide a positive starting point for the second wave of VR research, in contrast to the skeptical editorial in the Lancet. During this period the virtual environment used in research became increasingly more complex. Several research topics progressively continued in the zeroes, such as spatial memory and anxiety disorders. One interesting development with regard to the latter is the study of fear of public speaking. A virtual crowd is a very effective tool to map and potentially decrease anxious responses to public speaking (1, 2). Just look at this video of a bored crowd. In the same period, VR was used to study persecutory delusions. Besides these developments, several other topics can be distinguished, namely embodiment and social cognition and behaviour, and high-impact situations.
Embodiment is the poster child of VR research. Here, the power of VR really shows off. Embodiment refers to substitution of a real body for a virtual body. This body swap creates the brilliant illusion of ownership over a virtual body and the causation of the actions in the virtual body. Again, several precursors can be found in the literature. Already in 1994, Mel Slater and Martin Usoh showed that a basic virtual body increases the sense of presence, the belief that you are in the virtual world and not in the real world. Another nineties precursor is the work on VR and eating disorders that actively distorted virtual body image to reconstruct the body image in reality. Ok, my point being that, again, some research has been done in the nineties. However, embodiment was thoroughly investigated in the zeroes. In two-years time several crucial papers were published by Mel Slater, Mavi Sánchez-Vives and colleagues. In started out by only a virtual arm and ended up in complete embodiment of males in a female body. How is that for perspective taking? Also, with VR you can make people fat. Of course these studies are fun, but they also provide the crucial foundation for manipulation of social cognition with VR. This is where things get really interesting
Embodiment and Social Cognition and Behavior
Embodiment is a brilliant tool to study perspective taking, empathy, perception and the like. Surely, being embodied in a different body will influence how you perceive and react to the world? Indeed, over the last ten years, several groups have shown the positive effects of embodiment. These studies use a very straightforward and basic setup and procedure. Participants rocking a HMD, see their virtual body from a first-person perspective, but also reflected in a virtual mirror. Any movement they make is also visible in the virtual world. Most of the time, that’s it. To measure the effect of embodiment, a social cognition test is completed before and after the VR procedure. The video below provides an example of embodiment.
So can we use VR to change the world? Well, I have to say, the results are very promising. Embodiment of white participants in a black body for less then 12 minutes reduced their implicit racial biases. Similar changes in associations to children, and older adults have been observed. Even prosocial behavior can be increased. The embodiment in a virtual superhero with the ability to fly increased helping behavior afterwards. Besides the possible prosocial consequences of embodiment, VR and the interaction with virtual humans has been used to study empathy and mimicry. Definitely much more to come in future years!
Remember me saying in a previous blogpost that VR gives the opportunity to measure ‘responses in well-controlled situations that cannot be created in reality because of costs, danger or ethics’. Well, we have come to the danger and unethical part. Two studies from the second wave illustrate this. First, a VR version of Milgram’s obedience experiment has been created. Similar as in the original study, participants were test leaders during a series of memory tasks and were instructed to administer an electrical shock when the response of the test subject was incorrect. Second, the reactions of bystanders to a violent bar fight have been measured. Participants observed a verbal fight between two football supporters that escalated into a physical fight. The video below shows a 2D rendering of this high-impact VR situation. Favorites!
While the participants know that these situations are not real, their behavioral and subjective responses were as if it were. These VR studies do not suffer the shortcomings often found in contemporary studies in psychology and neuroscience. Seeing a violent scene on a computer screen really doesn’t elicit the same emotional response as during a bar fight, and during a staged emergency situation complete experimental control of the situation is lost. VR overcomes these shortcomings and allows to study high-impact situation even inside an MRI environment. From Milgram’s obedience experiment to a life-threatening emergency to defensive reactions to threat. Again, the power of VR.
The third-wave of VR
What does the future hold? What is released during the third-wave of VR research? More embodiment manipulations of social cognition for sure. Also, more high-impact situations to study empathy, emotion, and even morality. Combine this with neuroimaging or brain stimulation and you will have my dream study. Another thing that will likely be used more often is Collaborative Virtual Environments. In these environments multiple people interact and collaborate as virtual humans. But I could be completely wrong of course. We will see what the virtual world offers.
“Despite the excessive hype for virtual reality, […] the technology has genuine potential in a wide range of applications, and is engaging the serious research interests of scientists, social scientists and engineers”, as written in Nature in 2016 1994. So are we on the forefront of VR research in psychology and neuroscience? No, because the topics studied have been around since the nineties. Yes, because in the nineties a VR lab would cost between $60.000 and $1.000.000, in the zeroes between $20.000 and $100.000, and right now around $4000. So maybe.
———  Wait shouldn’t that be post-VR, c.f. post-punk, post-hardcore, or indie?
 For those skeptical about the computer technology at that time, remember that Pixar’s Toy Story was released in 1995. However, I will not discuss the technical evolution in this blogpost.
 I once read about this www.sci-hub.cc website that gives you access to many scientific articles behind evil paywalls. Apparently, you only have to copy the url of the article in the Sci-Hub website. Sounds very straightforward…
 Apparently, VR and psychoanalytic or psychodynamic approaches went well together in the nineties.
 As well as hard to distinguish the second from third wave. Well it was subjective anyway.
Aweh, I opened the package and stepped into the virtual world. It was virtuous.
In this bonus blogpost I will describe how I travelled with the VR setup to Cape Town and what VR promises for travelling in the near future.
Travelling with VR – the practical now
Before I could further immerse myself and others in a virtual world, I had to travel in the real world. I had to bring the VR setup from The Netherlands to South Africa. So how do you travel with a VR setup? How do you ship two giant boxes of delicate hardware from a little country in Europe to the southern point of Africa? The answer is fairly simple and contains a grocery bag, styrofoam, a carry-on suitcase, some sweat, and an annoyed wife. You just bring the entire setup (PC included) on the plane as carry-on luggage.
Friends and family thought I was crazy and it would be impossible. However, people bring all kinds of weird stuff on the plane. Before ordering, I made sure that the size and weight of the VR setup (especially the PC) was compatible with the restrictions of the airline (<12KG, 55 x 35 x 25 cm). After doing some research online, I decided to wrap the PC in a blanket, put stryofoam on all sides, and put it in a grocery bag. The Vive was straightforward as well. The Vive comes in a neat box, with little compartments for the different components of the setup. I just put all the necessary compartments (head-mounted display, motion trackers, controllers) in a carry-on suitcase, made sure it was tight, cables in the checked luggage and off we went. The downside of the approach I took is that you need a companion that is willing to travel altruistically, i.e. to bring your carry-on luggage while leaving her own. Also, it looks ridiculous.
An alien pc
The Vive cramped like a passenger
Amazingly in these times of airport restrictions, secondary inspections, and general travel angst, I did not have any problem at all bringing a VR setup onboard. The carry-on suitcase with the HTC Vive went through inspection without any hassle, or questions asked, and the PC was checked on drugs, but was left in the “protective case of blanket+stryofoam”. That’s it. A bit of an anticlimax (especially for friends and family that hoped for some trouble). This is how you travel with VR at the moment, but how do we travel in the future?
Travelling with VR – the future space
During the flight to Cape Town I thought about how VR could change travelling. The state-of-the-art in air travel is hundreds of people in a confined space with the sole inflight entertainment coming from a small movie screen and the elbow of your neighbor and the knees of the person behind you. What if instead of watching movies on a small screen, you could be immersed in a full 360 degrees movie. Or you could travel the world while travelling the world. You could walk through nature, already experience the city you’re travelling to, or simply enjoy the comfort of you home, while being in the lower regions of the stratosphere.
This isn’t just daydreaming. My thoughts were grounded in reality, and inspired by the work of Mel Slater and his lab at the University of Barcelona. In a recent article in Scientific Reports, they showed that it is possible to create the illusion that people are walking outdoors while sitting on a chair indoors. In short, seated participants (n = 28), fitted with a head-mounted display, entered a virtual world in which they saw a standing virtual body. Importantly, participants were only able to look around the virtual environment with head movements. At one point the virtual body started to walk across the virtual environment and eventually climbed a hill (see the video for more details).
When they saw the world through the eyes of the virtual body, the participants felt not only as if they were in the virtual world, but also had the illusion that they themselves were walking and that the leg movements were their own. Walking while sitting. Does it also result in objective changes? Yes, when the virtual body climbed the hill, skin conductance (a measure of arousal) and heart rate increased. Thus, with VR you can create the illusion of walking while in fact you are sitting still with no leg movements whatsoever.
Of course, this study has much more depth than I currently acknowledge. It is relevant to the study of agency, predictive coding, action perception, and paraplegia. But for the travel edition of the blog, I highlight the practical implications. You can sit in a chair in a Boeing Dreamliner or Airbus-A380, put on your head-mounted display and enter and move through a completely different environment. Imagine, two hundred people side-by-side in an airplane all in a different or shared environment.
To impact of VR on travelling is an active research field. The highlighted work has been part of a larger body of work, the VR-HYPERSPACE project. In this multidisciplinary project researchers investigated ways to improve passenger comfort (see the video). Besides VR, the project investigated some exciting new approaches and technologies. In the future you will travel in an invisible plane and with inflight entertainment shown on a multi-user 3D display. Again this is not me daydreaming. Last year, Qantas tried out the Samsung Gear in their first class. VR, going places.
The next blogpost: a brief history of Virtual Reality
The next journal club post: Blascovich et al. (2002). Immersive virtual environment technology as a methodological tool for social psychology. Psychological Inquiry, 13:103-124.
 My favorite junk TV-show: “Border Security: Australia’s Front Line”
The package on the picture contains a portal to a complete new world. With it I can step into a world beyond the physical domain. A world that is only bound by imagination . The package contains a HTC Vive, a Virtual Reality (VR) headset. Thanks to a grant from the Limburg University Professors’ Fund, I can take this step and acquire a VR setup. However, I have to wait several days to open it, as the package and I are currently 13.295 km apart. To control my excitement, I started this blog on my scientific adventures in Virtual Reality.
In my research I study the neural mechanisms of positive and negative social interactions. I mainly use a combination of neuroimaging, brain stimulation, and behavioral measures. But in the recent years I have become increasingly interested in the unique possibilities VR offers to psychology and neuroscience. While most people will immediately think of gaming, movies and other form of entertainment, VR has a lot to offer to us scientists. From animal research to human studies on memory, bodily consciousness, implicit biases, empathy, and morality. It gives us the possibility to measure genuine physiological, behavioral, and phenomenological responses in well-controlled situations that cannot be created in reality because of costs, danger or ethics. Or to put it in a less formal way: you can measure the reactions of participants during a violent conflict, make people old, or get IRB approval to recreate the obedience experiment by Stanley Milgram.
After attending several talks by eminent researchers in the field, and with my colleagues of the Brain and Emotion lab having incorporated VR in their research (find a primer in this recent review), I decided to follow the crowd as a *late* adopter. Luckily, I had the opportunity to visit the EVENT lab of Mel Slater as part of a research project. These days I’m completely sold on the promises of VR for social neuroscience. I am very happy to further integrate VR in my research with the acquisition of a VR setup. This blog will be a testimony to that.
Of course, I do not claim authority. This blog will be an online collection of my journey into VR research. The joys and the struggles. Some posts will focus on the VR literature (e.g., the concept of presence, embodiment), others will be more practical in nature (e.g., VR setup, programming, motion tracking), and some fall in between (e.g., motion sickness, ethics). Blog posts will be intertwined with journal club posts in which I will discuss a paper from the VR literature.
Before I conclude this introductory post, these are the specs of the VR setup I will be working with;
Headset: HTC Vive with an OLED display, 2160 x 1200 resolution, 90Hz refresh rate, 110 degrees field of view, 4.5 x 4.5m tracking area,
PC: Alienware Aurora PC, with a Intel(R) Core(TM) i5-6400 Processor (4-Cores, 6MB Cache, Turbo Boost 2.0, up to 3.3GHz), 8GB Dual Channel DDR4 2133MHz (4GBx2) RAM, and NVIDIA GeForce GTX 1070 Founders Edition with 8GB GDDR5.
Sorry, I will get into the details once I can physically hold the content of the package. Let’s get virtual.
The next blogpost: a brief history of Virtual Reality
The next journal club post: Blascovich et al. (2002). Immersive virtual environment technology as a methodological tool for social psychology. Psychological Inquiry, 13:103-124.
 and the graphics card. I will have a Nvidia GeForce GTX 1070 Founders Edition so I will be fine.
 This fund is just rad. In 2012, two professors at Maastricht University asked their colleagues to use their privilege and give an annual donation to support starting researchers. It was a great success, with a large number of professors donating from different resources (etc. book royalties, speaker fees). Today, this is used to fund small projects by Maastricht University researchers that just obtained their PhD.
 The prequel to this adventure was funded by an Academy Van der Gaag Grant from the Royal Netherlands Academy of Arts and Sciences