The role of HoloLens 2 in healthcare
In recent years, artificial environments such as virtual, augmented, and mixed reality have been investigated to see if these technologies can support the healthcare industry. Virtual reality (VR) is a completely computer-generated 3D environment, the user has no contact with the real world. Augmented reality (AR) is an environment that extends the real world of the user with virtual elements. In the case of mixed reality (MR), the user exists in the real-world environment and can also interact with virtual objects (Guo and Prabhakaran, 2022).
Microsoft released HoloLens 2 in 2019, which is in many aspects an improvement from its previous model, HoloLens 1. As the price is quite high ($ 3.500 – $ 5.199 (HoloLens 2 – Pricing and Options | Microsoft HoloLens, 2022), the target audience of the goggles is not individuals, but large enterprise companies. Microsoft HoloLens 2 is an MR head-mounted display (HMD) that is equipped with various sensors, and a holographic processing unit (HPU) with an optimized neural network (Newcomb A., 2019). These are only a few of the many reasons why this device can have a serious impact on the efficiency and safety of medical practices.
Technical overview of HoloLens 2
HoloLens 2 is equipped with 4 visible light cameras for the head and 2 Infrared (IR) cameras for real-time eye tracking (Microsoft Learn, 2022). The display is optimized for the captured eye position, and since one can see through holographic lenses, the view results in an extended reality with holographic objects. A depth sensor is responsible for capturing movements of both hands and fingers so users can directly manipulate the environment.
Spatial mapping uses the real-world environment to construct the virtual one, where holograms can be visualized (Microsoft Learn, 2022). HoloLens 2 shows a serious improvement from the first edition of the device, however, for better visualization, enough brightness should be provided (Guo and Prabhakaran, 2022). The available light and the size of the object affect the construction efficiency of the environment and the holograms. Spatial mapping is also responsible for reducing the overlapping ratio, which indicates how well the holograms overlay on the real-world environment and its objects. The research performed by (Guo and Prabhakaran, 2022) shows that HoloLens 2 overlapping ratio results in a 53,69% accuracy deviation, which is a serious advancement since HoloLens 1.
Controlling the environment and holographical objects is also possible with voice commands next to hand gestures thanks to the built-in 5-channel microphone. On top of system-defined commands, one can create custom commands. The (Guo and Prabhakaran, 2022) research shows a small improvement in HoloLens 2 from HoloLens 1 in user-defined commands, while a serious advancement can be seen when applying system-defined commands.
The Azure Spatial Anchor enables positioning world-scale anchors on the environment (Microsoft Learn, 2022), which can be shared with others, hence it is a great feature for guiding a user while performing a task.
The greatest improvement in HoloLens 2 from the previous edition is that one can obtain data from the Inertial Measurement Unit (IMU) sensors, more specifically from the accelerometer, the gyroscope, and the magnetometer (Guo and Prabhakaran, 2022). This information can be used to better detect the movement and rotation of the device.
The spatial sound built into the speakers allows for a lifelike experience and a more natural sound, as it includes a vertical element in addition to the horizontal sense of surround sound (Painter, 2021).
Applying MR in healthcare
Medical practices have a great impact on people’s lives, as a single mistake can be fatal. Various complications can develop due to incorrect treatment, which can cause severe pain or even serious internal organ problems. Patients often need a lot of help and support, but unfortunately, sometimes there are not enough nurses to take care of them.
Dementia is not caused by aging, but because of brain cell damage which reduces the ability to think, reason, and remember (Dementia vs. Alzheimer’s Disease: What is the Difference? | alz.org, 2022). People with dementia often find it difficult to carry out everyday tasks and, especially in the later stages, need to rely on a care person. However, a caregiver cannot always be around, so the My Daily Routine (MDR) application was developed for HoloLens (Hamilton et al., 2021) to support patients while they are alone, also giving them a little sense of independence. While the device is in use, it can show video reminders at a specified time, and navigate patients indoors, which sets it aside from previous solutions. Additionally, it offers relaxing music and videos to make the patient feel comfortable. Cognitive stimulation with visual and audio elements is used to help the user remember, for example, with real voice and video recordings of loved ones.
The service consists of a web service and an application installed on HoloLens. The website is used by the caregiver to easily register and customize the content presented to the patient. To enable the patient’s lower cognitive presence, the caregiver can configure feedback for custom voice commands from the patient and objects perceived by the patient. Thanks to the device’s head and eye tracking capabilities, the reminders follow the user’s movements, so they don’t forget about the notifications if something suddenly disturbs them.
The (Hamilton et al., 2021) study reveals that HoloLens can present various kinds of content with hardly noticeable reduction (20-25%) in performance by the human eye. However, because of the high price of the device, it is not accessible and affordable to most people with dementia.
A 3D view of human anatomy is crucial for dental students, and this is greatly supported by teaching with cadavers, which allows students to gain more accurate and in-depth knowledge (Zafar et al., 2020). In this way, they can also gain experience with different human bodies. However, making the necessary number of cadavers available involves high costs and ethical issues. By relying only on cadavers, students have less time to improve their hands-on knowledge.
To provide a more extensive learning opportunity to dental students, HoloHuman (Zafar et al., 2020) for HoloLens 2 gives a complete 3D experience of cadaver dissection without a real corpse. It allows the user to explore 3D anatomy models in depth, analyze the specific body parts layer by layer, and understand the relationships between organs by manipulating the holographic models with hand gestures. Lecturers can create activities for a session by preparing the view of the cadaver that is to be used in the classroom (3D4Medical from Elsevier, 2019).
The (Zafar et al., 2020) study compares the effectiveness of HoloHuman and cadaver anatomy sessions, observing verisimilitude, availability, and controllability. Dental students enjoy learning with HoloLens 2 and find it beneficial for expanding their knowledge, but more than 70% of them would not replace traditional classes, but supplement those with virtual education. The holographic models are not as realistic as real corpses and, in some cases, manipulation is difficult, because users have to learn hand gesture skills as well. HoloHuman is still a great option for gaining deeper knowledge when a real corpse is not available.
The (Grad et al., 2022) study examines the efficiency of the occlusion based on a 3D printed or holographic „anatomical tooth reference model”, by observing the difference between the reference model and the one after the recreation of the tooth. The use of HoloLens 2 to execute the intervention resulted in a shorter operating time than the same method on a 3D-printed model. Some participants noted that HoloLens cannot be properly used with dental magnifiers, which are essential for dentists’ daily tasks. As HoloLens 2 is lighter and more balanced than the previous edition, it is more comfortable and less harmful to the muscles and the spine.
Based on the above studies it can be concluded that HoloLens 2 is excellent for learning dental anatomy, but it is not satisfactory for replacing physical exercises.
Pose-Díez-De-la-lastra et al. (2022) examined how well HoloLens 2 can navigate the surgeon during oncological operations. The solution discussed is specialized on „sarcoma on the distal leg” and a tumor on the shoulder. The holographic model was based on a 3D-printed computed tomography (CT) scan of an existing patient’s bone structure along with an AR marker and surgical guide. Instead of spatial mapping, the Vuforia SDK (Vuforia Developer Portal, 2022) was used to register the holographic model on the surface of the given body part by detecting the AR marker because of its higher tracking accuracy. This surface is constantly changing during the operation if the body moves or the surgeon removes soft parts, so accurate registration is important since the success or failure of the operation can depend on millimeters.
In the virtual environment, there is also a board on which the surgeon can monitor the progress of the operation, together with its detailed conditions. Wearing the HoloLens 2 HMD during an oncological operation, the surgeon’s hands are free and can see the holographic model fixed on the patient’s bone, so gazing away time is reduced. With the use of hand gestures, transparency, and hologram location can be manipulated for convenience.
HoloLens 2 shows 25% better hologram registration than its predecessor and surgeons preferred the new edition due to its ergonomic improvements that make the device lighter and more comfortable (Pose-Díez-De-la-lastra et al., 2022).
Another treatment for cancer is radiation therapy, which reduces the size of the tumor or completely destroys the cancer cells (NCI, 2019). There are many ways of delivering the treatment depending on the type, size, location of the tumor, and demographic factors of the patient. The temporary procedure involves using a needle filled with radioactive seeds and injecting them into the tumor (Brachytherapy (Internal radiation therapy) (radiologyinfo.org), 2022). Some lesions are difficult to target exactly because of their location which can lead to longer intervention time, a higher dose of radiation, and several often serious complications that require further operations. To improve the efficiency of the treatment and reduce the radiation dose, the (Park et al., 2020) study presents a CT-guided needle trajectory simulation on an abdominal tumor supported by HoloLens 2.
The CT scan results in a segmented CT grid, after which the threshold between the meshes is reduced so that the abdominal lesion is more visible. For the 3D model used by HoloLens 2, a lesion is selected as a treatment target then its exact location and ideal needle trajectory are marked on the CT grid.
Using computer vision and the Vuforia SDK, the grid lines of the holographic model are fully aligned with the physical one. The (Park et al., 2020) study presents the serious efficiency of holographic guiding needles with HoloLens 2; the user can seamlessly follow the ideal trajectory when using the real needle. Procedure time and radiation dose were reduced by fewer needle insertions, that is, fewer corrections were needed in hologram-guided simulation than in conventional interventions.
HoloLens 2 in the healthcare industry appears to improve the efficiency and safety of medical activities, however, further research is needed before it can be introduced into everyday medical practice. Using the device as a guide during oncological surgeries shows promising results. Since the surgeons’ hands remain free during use, and the panel showing the surgical results follows their gaze, it provides great comfort, so they can perform the intervention more safely and efficiently. It also provides students with support in dental education, since even in the absence of cadavers, they can improve their knowledge of 3D holographical models, which is cost- and time-efficient. Based on the literature, while it is possible to develop its realism, the technology is still far from completely replacing education on corpses and intervention on living people. HoloLens 2 plays a big role in patient care, although due to the high price, it may be difficult for private individuals to access it. If the device becomes more accessible, it could be of great help to the elderly. By observing the data from the IMU sensors, the care person can notice in time if the patient suddenly falls for some reason. Then the care person can immediately alert the emergency services or rush to the scene to check the situation and even start lifesaving (Salvetti et al., 2018).
It would be very interesting to examine how HoloLens 2 supports doctors in establishing a diagnosis. As described in the (Sicht J., 2020) article, the device can be used perfectly if an expert opinion is needed when investigating a special case. At that time, both the on-site doctor and the specialist put on the glasses, and both can see and hear the patient, thus forming the diagnosis together.
We can see amazing advancements not only in healthcare but also in other fields using HoloLens 2, such as manufacturing (Buń, Grajewski and Górski, 2021; Gallala et al., 2022), architecture (Vermandere, Bassier and Vergauwen, 2022), law enforcement (Dzermansky et al., 2021), education (Leonard and Fitzgerald, 2018; Lavric et al., 2022), war and defense (Phys.org, 2019).
However, for its practical application in everyday life, it is necessary to evaluate its battery life and the display width for the specific use case. It is important to consider cybersickness as well because, after a certain period of time, several persons who took part in examinations reported headaches, blurred vision, and dizziness.
Brachytherapy (Internal radiation therapy) (radiologyinfo.org) (2022) Available at: https://www.radiologyinfo.org/en/info/brachy (Accessed: 18 December 2022)
Dementia vs. Alzheimer’s Disease: What is the Difference? | alz.org (2022) Available at: https://www.alz.org/alzheimers-dementia/difference-between-dementia-and-alzheimer-s (Accessed: 18 December 2022)
Buń, P., Grajewski, D. and Górski, F. (2021) ‘Using augmented reality devices for remote support in manufacturing: A case study and analysis’, Advances in Production Engineering And Management, 16(4), pp. 418–430. Available at: https://doi.org/10.14743/APEM2021.4.410.
Dzermansky, M. et al. (2021) ‘USE OF AUGMENTED REALITY TECHNOLOGY IN POPULATION PROTECTION AND CRISIS MANAGEMENT’, in Annals of DAAAM and Proceedings of the International DAAAM Symposium. DAAAM International Vienna, pp. 408–414. Available at: https://doi.org/10.2507/32nd.daaam.proceedings.060.
Gallala, A. et al. (2022) ‘Digital Twin for Human–Robot Interactions by Means of Industry 4.0 Enabling Technologies’, Sensors, 22(13). Available at: https://doi.org/10.3390/s22134950.
Grad, P. et al. (2022) ‘Application of HoloLens ‐based Augmented Reality and three‐dimensional printed anatomical tooth reference models in dental education’, Anatomical Sciences Education [Preprint]. Available at: https://doi.org/10.1002/ase.2241.
Guo, H.-J. and Prabhakaran, B. (2022) ‘HoloLens 2 Technical Evaluation as Mixed Reality Guide’. Available at: http://arxiv.org/abs/2207.09554.
Hamilton, M.A. et al. (2021) ‘Augmented reality technology for people living with dementia and their care partners’, in ACM International Conference Proceeding Series. Association for Computing Machinery, pp. 21–30. Available at: https://doi.org/10.1145/3463914.3463918.
HoloLens 2 – Overview, Features, and Specs | Microsoft HoloLens (2022) Available at: https://www.microsoft.com/en-us/hololens/hardware (Accessed: 18 December 2022)
HoloLens 2 – Pricing and Options | Microsoft HoloLens (2022) Available at: https://www.microsoft.com/en-us/hololens/buy (Accessed: 18 December 2022)
Lavric, T. et al. (2022) ‘A Low-Cost AR Training System for Manual Assembly Operations’, Computer Science and Information Systems, 19(2), pp. 1047–1073. Available at: https://doi.org/10.2298/CSIS211123013L.
Leonard, S.N. and Fitzgerald, R.N. (2018) ‘Holographic learning: A mixed reality trial of Microsoft Hololens in an Australian secondary school’, Research in Learning Technology, 26. Available at: https://doi.org/10.25304/rlt.v26.2160.
Microsoft Learn (2022) HoloLens 2 hardware. Available at: https://learn.microsoft.com/en-us/hololens/hololens2-hardware (Accessed: 18 December 2022)
NCI (2019) Radiation Therapy for Cancer. Available at: https://www.cancer.gov/about-cancer/treatment/types/radiation-therapy (Accessed: 18 December 2022)
Newcomb A. (2019) ’Microsoft Keeps HoloLens 2 Focused on Work’, Fortune, 26 February. Available at: https://fortune.com/2019/02/25/microsoft-hololens-2-glasses/ (Accessed: 15 December 2022)
Painter, L. (2021) ‘What is Spatial Audio? An Immersive Sound Guide’, Tech Advisor, 20 August. Available at: https://www.techadvisor.com/article/744441/what-is-spatial-audio-a-guide-to-immersive-sound.html (Accessed: 18 December 2022)
Park, B.J. et al. (2020) ‘Augmented reality improves procedural efficiency and reduces radiation dose for CT-guided lesion targeting: a phantom study using HoloLens 2’, Scientific Reports, 10(1). Available at: https://doi.org/10.1038/s41598-020-75676-4.
Phys.org (2019) Microsoft workers protest use of HoloLens headset for war. Available at: https://phys.org/news/2019-02-microsoft-workers-protest-hololens-headsets.html (Accessed: 15 December 2022)
Pose-Díez-De-la-lastra, A. et al. (2022) ‘HoloLens 1 vs. HoloLens 2: Improvements in the New Model for Orthopedic Oncological Interventions’, Sensors, 22(13). Available at: https://doi.org/10.3390/s22134915.
Salvetti, F. et al. (2018) ‘HoloLens, Augmented Reality and Teamwork: Merging Virtual and Real Workplaces’, International Journal of Advanced Corporate Learning (iJAC), 11(1), p. 44. Available at: https://doi.org/10.3991/ijac.v11i1.9228.
Sicht J. (2020) ’HoloLens Connects Techs With Remote Experts’, Vehicle Service Pros, 30 October. Available at: https://www.vehicleservicepros.com/shop-operations/collision-repair/business-and-finance/article/21201667/mercedesbenz-mixedreality-hololens-connects-techs-with-remote-experts (Accessed: 15 December 2022)
Vermandere, J., Bassier, M. and Vergauwen, M. (2022) ‘Two-Step Alignment of Mixed Reality Devices to Existing Building Data’, Remote Sensing, 14(11). Available at: https://doi.org/10.3390/rs14112680.
Vuforia Developer Portal (2022) Available at: https://developer.vuforia.com/ (Accessed: 18 December 2022)
Zafar S, Zachar JJ. Evaluation of HoloHuman augmented reality application as a novel educational tool in dentistry. Eur J Dent Educ. 2020;24:259–265. https://doi.org/10.1111/eje.12492
3D4Medical from Elsevier [@3D4Medical] (2019) Introducing HoloHuman by Pearson, powered by 3D4Medical. 7 March. Available at: https://www.facebook.com/3D4Medical/videos/2197484543898306/ (Accessed: 18 December 2022).