Professor Rob Lindeman and PhD student Dilshani Kumarapeli are part of a research team exploring how virtual reality could be used as a tool during remote medical assessments.
In a recent study PhD student Dilshani Kumarapeli, Postdoctoral Fellow Dr Sungchul Jung, Research Associate Dr Yuanjie Wu, and Professor Rob Lindeman, Director of the Human Interface Technology (HIT) Lab at Te Whare Wānanga o Waitaha | University of Canterbury (UC), investigated the potential for Virtual Reality (VR) headsets in medical consultations where access or risk factors may prevent doctor and patient from being in the same room.
The team’s research was designed to provide the patient and doctor with a personalised experience via VR headset. Kumarapeli and Professor Lindeman call this an “asymmetric system” and see it as an important point of difference when compared with more common applications for VR technology — like video games — where both users employ the same system and receive identical information.
The researchers believe the highly customised nature of their system could enhance the connection between doctor and patient, even when they’re not in the same room, by providing a sensory experience designed to best address their individual needs. The system could be used for patients based in remote locations, patients with highly contagious illnesses, or patients with mental health conditions that may cause them to behave violently towards others.
“The patient needs to feel trust in their clinician, so for them, the system is focused on relaying eye-contact and facial expressions that make them feel connected to the doctor despite being in a different physical location,” Professor Lindeman says. “For the clinician, it can provide detailed physiological information that the patient might struggle to convey verbally during a remote assessment.”
Data conveyed to the doctor via the headset could include eye-tracking, facial expressions, pulse and breathing rate. This data would also be interpreted by a connected software programme, enabling the clinician to make a more accurate diagnosis.
While the research team has trialled the system with a group of UC students, and both Professor Lindeman and Kumarapeli see great potential to use similar “asymmetric” VR technology in a diverse range of scenarios, including educational frameworks, the system will be further developed for use in real-world scenarios.
For Kumarapeli, the most rewarding aspect of the research was developing the emotion recognition software.
“Because the participants are wearing the headset, we weren’t able to capture all their facial expressions with a camera, so we had to find an alternative to solve this issue,” she says.
“We used a facial capture device and trained a neural network to identify various key emotions. These emotions were then relayed to the clinician, to help in managing the diagnosis session. I really enjoyed the journey of developing the neural network and applying the trained modules to this real-life scenario.”
The researchers received support from the Science for Technological Innovation National Science Challenge. Their work is being prepared for publication.
Dr Jung was the principal investigator on this work, and has since taken a continuing Assistant Professor position in the United States.
Sustainable Development Goal (SDG) 3 - Good health and wellbeing.
