Multimodal emotion recognition in the metaverse era: New needs and transformation in mental health work

Abstract

This editorial comments on an article recently published by López del Hoyo et al. The metaverse, hailed as "the successor to the mobile Internet", is undoubtedly one of the most fashionable terms in recent years. Although metaverse development is a complex and multifaceted evolutionary process influenced by many factors, it is almost certain that it will significantly impact our lives, including mental health services. Like any other technological advancements, the metaverse era presents a double-edged sword for mental health work, which must clearly understand the needs and transformations of its target audience. In this editorial, our primary focus is to contemplate potential new needs and transformation in mental health work during the metaverse era from the perspective of multimodal emotion recognition.

Key Words: Multimodal emotion recognition; Metaverse; Needs; Transformation; Mental health

Core Tip: As the successor to the mobile Internet, metaverse refers to a shared, persistent, three-dimensional virtual space that serves as a digital extension of the real world. The future development of the metaverse is fraught with potential variables and uncertainties; however, we can discuss its possible directions and challenges. This editorial focuses on contemplating potential new needs and transformation in mental health work during the metaverse era from the perspective of multimodal emotion recognition.

INTRODUCTION

The metaverse, a virtual world driven by artificial intelligence (AI), is hailed as "the successor to the mobile Internet"[1]. The metaverse enables seamless interaction between users, other users, and the environment across both the virtual and physical worlds. The metaverse can transform various sectors, including healthcare, by integrating emerging technologies, including cloud edge computing, AI, blockchain, Internet of Things, and semantic communication[2]. Despite the hype surrounding the metaverse in recent years, there is little empirical research on who uses the service, for what specific purposes, and with what consequences, and the same goes for the Healthcare Metaverse[3].

Since the COVID-19 pandemic, there has been a growing interest and demand for online platforms that replace traditional face-to-face activities. The metaverse development has also attracted widespread attention in the mental health field, including contemplation on how the metaverse can expand new psychological therapy strategies and methods in the emerging therapeutic space[4]. Within the field of mental health work in the metaverse, we were particularly interested in reading the original article by López del Hoyo et al[5]. The authors pondered how the metaverse could reshape how we deliver mental health treatments through opportunities and challenges[5]. We find this to be a qualified and enlightening article and agree with the authors' view that, although the metaverse is still in its early stages of development, it has the potential to change how we provide mental health care while introducing new challenges.

Multimodal emotion recognition (MER) refers to the technology that identifies and interprets an individual's emotional state by combining and analyzing two or more different types of input data, including text, voice, images, physiological signals, etc.[6]. Compared to unimodal emotion recognition, MER allows for a more comprehensive and accurate understanding and assessment of emotional states[7]. MER will play a significant role in mental health work during the metaverse era, enabling more precise emotion identification and evaluation and promoting personalized human-computer interaction. Furthermore, it opens new virtual spaces and technological pathways for delivering effective mental health services. In this editorial, we aim to further explore potential new needs and transformations in mental health work during the metaverse era from the perspective of MER.

NEW NEEDS FOR MENTAL HEALTH

The term "metaverse" was first coined by Neal Stephenson in his 1992 science fiction novel "Snow Crash" where characters use their digital avatars to escape a dystopian reality. The ever-evolving metaverse is currently a digital world created using various technologies such as virtual reality (VR), augmented reality (AR), mixed reality (MR), AI, and the Internet. The emergence of the metaverse is set to transform how we socialize, work, and interact and may even change how we receive treatment[8]. As López del Hoyo et al[5] mentioned in their article, one key focus in psychotherapy research is determining "what works for whom". The same applies in the metaverse era, where future mental health work must ascertain which needs of which users should be targeted with novel psychological strategies.

As the direction in which the mmetaverse will evolve is not yet clear, there is also no definite answer to what mental health needs users might have in the metaverse era. However, starting from the characteristics of the metaverse, mental health work in the new era may need to pay attention to the following issues. First, the immersive experience of the metaverse could increase the risk of digital addiction, as depicted in science fiction, where individuals seek to escape the physical world by indulging in the digital realm. Mental health professionals will need to identify and address metaverse-related addictive behaviors. Second, the metaverse platform for virtual identities and diverse life experiences could confuse users over their identity. Therefore, maintaining a balance between virtual and real life and how to handle the relationship between virtual and real-life identities will become a new focus for mental health work. Following this, individuals' social skills and real-world relationships could be affected due to changes in interpersonal interaction in the metaverse era. Mental health workers need to help users understand and adapt to this new mode of social interaction while maintaining healthy real-life relationships. Moreover, the metaverse is a global platform with users from different countries and cultural backgrounds. Mental health workers must have cross-cultural competencies to provide services suitable for users from diverse cultures. Another point worth noting is that mental health work in the metaverse era, based on VR, AR, MR, and internet technologies, significantly differs from the current mainstream, which predominantly involves face-to-face interactions. How can mental health professionals interact in real-time with clients who may be thousands of miles away and inaccessible regarding physical expressions, postures, and movements and understand the underlying messages that they convey? Emerging technologies like MER may become an essential auxiliary method of mental health work in the future metaverse era to address these challenges.

MER has been one of the key research directions in emotion recognition in recent years. It may offer multiple potential benefits to mental health work in the metaverse era[6,9]. First, it promotes personalized human-computer interaction. In the metaverse, where interactions between humans and machines are more frequent and complex, MER can help machines better understand and respond to users' emotional states, leading to a more natural and humane interactive experience, thereby improving users' mental health. Second, MER enables more accurate emotion identification and assessment. In the metaverse, users interact with digital identities that may exhibit different emotional states from those in the real world. MER technology can analyze various data, including user behavior, language, and physiological signals in the virtual environment, to identify and assess their emotional states more accurately, supporting mental health diagnosis[10]. Based on this, MER helps provide remote, personalized mental health interventions. The metaverse offers new possibilities for remote mental health services and counseling, and MER can enhance the efficiency and effectiveness of these remote consultations. MER also provides detailed data on individual emotional responses, allowing mental health professionals to tailor personalized treatment plans to specific user needs. For example, cognitive-behavioral therapy or mindfulness meditation training can be designed to better meet individual requirements by analyzing users' emotional response patterns. Additionally, MER can promote self-awareness and self-management among users, contributing to mental health education and disease prevention. Users can better understand their emotional response patterns through MER technology in the metaverse. This self-awareness can foster self-emotion management and enhance emotional intelligence. Furthermore, the metaverse and MER technologies provide new tools and methods for mental health research. Researchers can observe and record emotional expressions more naturally in this virtual environment, thus advancing research in mental health. In summary, as the metaverse and related technologies continue to develop, MER will make the above work more intelligent and efficient, providing strong support for maintaining and promoting users' mental health.

TRANSFORMATION IN THE METAVERSE ERA

Reference Citation Analysis (RCA, https://www.referencecitationanalysis.com/) is a unique AI system for citation evaluation of biomedical literature. RCA has been employed to analyze previous studies of needs and transformations in mental health work during the metaverse era from the perspective of MER up to March 2024.

The demand and intervention methods for mental health work evolve continuously with human society and technology development. The earliest recorded evidence of emotional disorders appears in the ancient Egyptian manuscript, The Ebers Papyrus, dating back to 1500 BCE[11]. We leap forward through millennia to the present day, and the mental health field has undergone significant transformations. What was once a collection of observations, conjectures, and empirical methods inscribed on papyrus has burgeoned into a multifaceted discipline encompassing various therapeutic approaches, ranging from psychological to pharmacological treatments. As we stand on the brink of the metaverse era, we are presented with unprecedented opportunities to advance our understanding and practices in mental health care. Integrating virtual reality technologies within this new digital universe offers a chance to reimagine therapeutic paradigms, creating the potential for more immersive, personalized, and accessible forms of support and intervention.

First and foremost, one of the most apparent transformations in mental health work during the metaverse era is the change in therapeutic settings and the move towards immersive experiences. In this metaverse era, the traditional boundaries of counseling rooms and hospital wards begin to blur as emerging technologies like VR, AR, and MR enable the creation of immersive environments that can mimic, modify, or completely reimagine the settings in which therapy takes place. The immersive nature of virtual worlds allows for innovative therapeutic experiences. For instance, VR therapy can be used in exposure therapy, treating phobias, post-traumatic stress disorder, and anxiety disorders by gradually exposing patients to their fears within a controlled virtual environment[8,12]. Through the use of VR headsets and other equipment, individuals with anxiety disorders can be exposed to their fears within a controlled environment, enabling them to learn coping mechanisms at their own pace. Similarly, trauma survivors can revisit and process their experiences within a safe virtual space. Therefore, technologies like MER have become increasingly indispensable to achieving an immersive therapeutic experience, ensuring engagement, and maintaining treatment efficacy within virtual therapeutic spaces.

Second, mental health work in the metaverse era will transition towards utilizing emerging technologies such as MER for real-time diagnostic and therapeutic assessments, contributing to improved quality and objectivity in evaluating mental health services[13]. While mental health interventions historically relied on the experience of practitioners and face-to-face interaction for recognition and treatment, the metaverse era envisions a transformation towards wearable devices and sensors. Leveraging technologies such as MER, these devices can provide real-time physiological data and accurate emotional status during therapy, enabling therapists to understand better and adjust personalized interventions based on patients' real-time emotional states, thereby improving the quality and efficacy of mental health care.

Third, the metaverse era promises a paradigm transformation in providing mental health support[4,14]. Some researchers believe that long-term and repeated counseling or psychotherapy may become outdated in the metaverse era, with rapid, direct, and engaging psychological services for younger generations of metaverse users emerging as a new trend[15]. Bolstered by technologies like MER, there is potential to broaden the scope of mental health assistance. With its 24/7 availability, the virtual world also paves the way for continuous emotional support in daily life. Furthermore, the metaverse offers opportunities to enhance focus on social and collective experiences, facilitating innovations in group therapy. For instance, virtual world group therapy sessions could connect individuals from around the globe to share their experiences, fostering a sense of community and support that may be lacking in their physical environments.

Fourth, the advent of the metaverse era changes the target audience for mental health work. The younger generations, born into the digital age, are more accustomed to using technology and virtual platforms for communication and interaction. For these individuals, the metaverse offers a familiar and engaging environment, one where they may be more inclined to seek out mental health counseling and therapy[15,16]. Additionally, those residing in remote or underserved areas with limited mental health professionals, people with social anxiety or autism spectrum disorders, people with a high demand for privacy and control over their identity, those experiencing stigma related to their conditions, as well as individuals with mobility challenges or sensory impairments, can all benefit from the accessible digital communication provided by the metaverse to receive better mental health services[17]. While the primary audience for mental health work has not fundamentally changed, the metaverse offers new modalities of service delivery that make mental health care more accessible and personalized for diverse groups. Meanwhile, this requires mental health professionals to adapt to the new technological landscape to ensure the effectiveness and safety of their services.

Fifth, the advent of the metaverse and MER technology presents various ethical challenges that demand rigorous examination. Privacy protection, for instance, is a paramount concern, as the technology has the potential to collect and analyze vast amounts of personal data. Privacy concerns arise from the potential for continuous surveillance and multimodal data collection, which can lead to the erosion of personal autonomy and the commodification of user data. Additionally, the availability of low-cost internet and other data transmission media allows for rapidly sharing large volumes of data[18]. These raise questions about consent, data security, and the potential for misuse of information. Additionally, managing virtual identities in the metaverse raises ethical issues related to authenticity, self-expression, blurring lines between the real and the virtual life, and concerns regarding "Human Digital Twins"[19]. These issues necessitate careful navigation to ensure that technological advancements align with ethical standards and societal values. The following explorations and recommendations warrant attention, including implementing stringent data protection laws and accountability systems, technological innovations centered on user privacy protection, and enhancing user awareness of privacy risks associated with the virtual world[18,20,21].

Furthermore, one noteworthy yet potentially overlooked aspect is that mental health practitioners' training content and skill requirements will also undergo a corresponding transformation in the metaverse era. Mental health professionals of the new age will need to master skills for conducting assessments and treatments in virtual spaces, adhere to ethics and privacy protections in the digital world, develop cross-cultural communication abilities, possess the capacity to interpret multimodal data, and become familiar with methods for crisis intervention and risk management within virtual environments to meet the new challenges posed to mental health work in the metaverse era. Since cultivating medical professionals is not an overnight process, early planning is required, which should also draw the attention of health administration departments and medical educational institutions.

CONCLUSION

The metaverse will undoubtedly change how we provide mental health care. It is also sure that MER will play a significant role in mental health work in the metaverse era. MER will facilitate the construction of virtual psychological treatment spaces, help improve diagnostic accuracy, personalize treatment, enhance teletherapy, improve patient engagement, support data-driven decision-making, and foster interdisciplinary collaboration. However, both the metaverse itself and MER technology are still in the process of development. The ultimate form that they will take and how they will impact each of us remain to be seen. Nevertheless, we recommend actively embracing the emerging technologies of the metaverse era and optimizing mental health work by leveraging their strengths and mitigating their weaknesses in practical applications.