Leader: Antonio Lanatà (UNIFI); Other collaborator(s):
In this task innovative wearable textile sensors for physiological monitoring along with intelligent platform integration will be implemented employing spatially distributed systems model to ensure security and reliability of the information. This task includes innovative signal processing and artificial intelligence tools for psychophysiological assessment and physical body activity analysis. Developed methods will enable the investigation of user engagement, cognitive loading, emotional response and mood dynamics to prevent physical and mental dangerous conditions. Furthermore, the dynamics of emotional face expression and eye gaze pattern will be processed for quantitative index extraction emotional wellness.
During the specified period (November 2023 - March 2024), our activity focused on different experimental designs for the psychophysiological characterization of young and adult subjects by analysing multivariate physiological signals obtained from biosensors and imaging techniques. In particular, a public database of EEG and fMRI data relating to an adult population considered at risk of developing Alzheimer Disease (AD) was preliminarily explored, along with a genotypic and phenotypic investigation. In experimental designs proposed to a healthy population, electrophysiological characterization of the participants was carried out through analysis of Event-Related Potentials (ERP) on words and pseudowords aloud reading task, a clinical protocol widely adopted by speech therapists and laryngologists to diagnose speech and language disorders. The ERP analysis was also proposed in an experimental design to understand subjects' implicit attitudes through the Implicit Association Test (IAT). In this experimental paradigm, eye gaze, blinking activity and variation of the pupillary diameter were investigated. Bibliographical research focused on rehabilitation and treatment aspects of elderly population with neurodegenerative disorders has been carried out. Particular attention has been paid to the study of the effect of non-invasive brain stimulation (tES and rTMS) in experimental contexts related to multi-sensory stimulation integrated into contexts of extended reality (XR) or serious games (SG). Finally, new acquisitions utilizing wearable sensors were performed on an experimental design to investigate possible different emotional and cognitive responses on real or AIgenerated human faces. Additional questionnaires to establish the participants' memory abilities and to understand the adopted strategy during the proposed task have been implemented.
Please see the next reporting period.
This report details activities conducted from July to September 2024, focusing on clustering analysis of vocal properties of 287 patients diagnosed with benign lesions of the vocal folds (BLVF) and unilateral vocal fold paralysis (UVFP), which are among the highest causes of dysphonia. The research aimed to identify if these patients constituted separate vocal subtypes of dysphonia and to understand whether misclustered data could depend on a specific diagnosis and age. This was done primarily to identify straightforward differences in vocal fold motor dynamics that generally are assessed with visual inspection through high- resolution endoscopy, which is not always available. Also, a particular emphasis was placed on misclustered observations to understand the role of confounding factors (i.e., the BLVF subtypes and age) in clustering analysis. Data collection regarded 287 patients recruited at the Ospedale Maggiore Policlinico di Milano. The diagnosis was made by perceptual voice evaluation (GRB scale) and video- laryngostroboscopic assessment. 136 patients presented UVFP and 151 BLVF. For the acoustic analysis, patients were asked to utter the sustained vowel /a/ for at least 3s. Acoustic parameters were extracted, accounting for age, gender and type of vocal emission. Specifically, fundamental frequency F0, local jitter, adaptive normalised noise energy, the first three formants, the duration of voiced and unvoiced parts, and the median, standard deviation, minimum and maximum values were used for further analysis. The k-means algorithm was applied to find clusters on the event space by iteratively assigning each point to the nearest candidate cluster centre and updating the cluster’s centres until convergence. The silhouette score offered a means for assessing the clustering quality by comparing inter-cluster distances with intra-cluster ones. Unaware and aware analyses were performed for clustering, whilst a misclustered data analysis was additionally performed to provide interpretable results for clinicians. Key findings for the female dataset in the unaware condition report that the best results were obtained with the principal component analysis (PCA). Also, a higher silhouette score for the UVFP class could be helpful in clinical practice for better recognising the most severe of the considered pathologies, possibly reducing its misdiagnosis. Results from the male dataset remain similar between the unaware and aware conditions. Moreover, in both genders, misclustered observations seem to be independent of a specific pathology (and its subtypes). Finally, the PCA weight analysis highlighted that phonation parameters were the most contributive. In conclusion, the results showed valuable preliminary data on voice disorders with clustering techniques on voice features, offering a new perspective for clinical practice and professionals.
During October-December 2024, we undertook several significant initiatives to advance research in physiological signal acquisition and analysis in clinical and experimental contexts. Below is a summary of the key activities conducted during this period:
These studies are relevant to ageing and healthy ageing, exploring the physiological underpinnings of emotional and social interactions. Emotional regulation and social interactions are critical for maintaining mental health and cognitive function in older adults, underscoring the broader implications of this research. Future efforts will further refine these methodologies and expand collaborations to amplify the impact of our findings.
From January to March 2025, Task 5.5 focused on advancing the understanding of psychophysiological and neurocognitive dynamics in social, affective, and clinical contexts through a series of experimental studies; the results have been consolidated in four peer-reviewed conference papers.
A major line of investigation explored the role of electrodermal activity (EDA) in characterising emotional responses within dyads exposed to affective stimuli. In a controlled setting involving 22 participants (11 dyads), variations in EDA patterns were found to reflect both intra- and inter-subject dynamics as arousal levels increased, thereby supporting the use of EDA as a reliable marker in hyperscanning protocols to assess emotional synchrony and co-regulation in social contexts.
In parallel, a second study introduced a novel approach to intersubject neurophysiological monitoring by integrating shared biofeedback mechanisms in dyadic affective tasks. EEG and ECG signals were jointly acquired using a custom-developed hyperscanning system in 24 dyads, enabling real-time cross-subject monitoring. Notably, heart rate variability (HRV) metrics, especially cross-entropy indices, were found to vary across baseline and stimulation phases, suggesting that intersubject biofeedback may influence autonomic synchrony and warrants further exploration as a modulator of interpersonal dynamics during joint cognitive-affective processing.
The task also contributed to the clinical voice assessment field by implementing a multimodal evaluation framework that combined acoustic analysis with sentiment scoring of transcribed free speech. In a cohort of 26 dysphonic patients, this approach effectively captured psychophysiological changes before and after surgical intervention. Findings indicated that male patients experienced measurable improvements in vocal stability. In contrast, female patients exhibited a strong correlation between pitch and positive sentiment, suggesting an intertwined evolution of vocal and emotional recovery. This methodology provides an efficient, automatic, and gender-independent tool to assess therapeutic outcomes beyond traditional self-report measures.
Finally, in serious games (SG), a study evaluated the effects of auditory cues on memory and visual attention using a gamified version of the Corsi Block-Tapping Test. Although no direct enhancement in memory performance was observed with auditory cueing, uniform auditory stimuli significantly altered eye-tracking metrics, indicating a shift in attentional processing. Specifically, fixation and saccade parameters were modulated in opposing directions depending on auditory condition, pointing to perceptual rather than memory-related effects of auditory cues.
These studies reflect Task 5.5’s continued commitment to advancing multimodal and multisystemic approaches for investigating human behavior in emotionally, cognitively, and clinically relevant contexts.
During the reporting period, research activities primarily concentrated on the development and analysis of experimental paradigms aimed at investigating cognitive and emotional states through multimodal physiological recordings. The semester was marked by the continuation and refinement of hyperscanning-based experimental frameworks, alongside the integration of serious games into protocols for simultaneous acquisition and analysis of behavioral and physiological data. Two principal studies were conducted:
1. Electrodermal Activity in Dyads During A ective Visual Stimulation
This study explored the modulation of electrodermal activity (EDA) within dyads exposed to emotionally salient visual stimuli. Simultaneous EDA recordings were collected from pairs of participants using a custom-built, low-cost Arduino-based acquisition system. A total of 22 healthy participants (11 dyads) were presented with a ective images sourced from the International A ective Picture System (IAPS), organized into four blocks with progressively increasing arousal levels. The primary objectives were (i) to assess individual autonomic responses across varying arousal levels and (ii) to investigate the emergence of emotional synchrony within dyads. Both intra- and inter-subject analyses revealed distinctive trends in EDA responses, with significant inter-individual variability across experimental conditions. Notably, several dyads exhibited temporally aligned fluctuations in EDA, potentially indicative of emotional co-regulation or synchrony in response to shared a ective stimuli. These findings contribute to the expanding field of hyperscanning and support the use of EDA as a marker of emotional coordination in social interactions. This study, titled “Quantitative Electrodermal Activity Analysis of Simultaneous Recordings from Dyads during A ective Visual Task: Proof of Concept and Preliminary Results”, has been accepted for presentation at the Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC 2025), to be held in Copenhagen in July 2025.
2. Multimodal Investigation of Auditory Cues and Eye Movements within a Serious Game Framework
In parallel, a novel experimental protocol was designed, implemented, and tested, combining cognitive gaming with multimodal physiological monitoring. The task—developed as a digital serious game—integrated elements of the Simon Game and the Trail Making Test and was administered under two experimental conditions: with and without uniform auditory cues (UAC). Electroencephalography (EEG) and eye-tracking data were recorded throughout gameplay to examine the influence of uditory stimulation on visual exploration strategies. Twelve participants completed the task under both conditions. Although no significant differences in memory performance (indexed by game score) were observed, eye-tracking analysis revealed distinct visual behavior patterns. In the absence of auditory cues, higher performance correlated positively with fixation count and negatively with both saccade count and fixation velocity, suggesting a more focused visual exploration strategy. Conversely, the UAC condition was associated with increased saccade activity, possibly reflecting interference from concurrent auditory processing. These results indicate that even non-semantic auditory input can significantly modulate visual search behavior and should be carefully considered in the design of multimodal cognitive assessment tools. Ongoing EEG analyses aim to identify event-related potentials (ERPs) and power spectral patterns associated with visual and auditory stimuli. These neurophysiological markers are expected to yield further insights into the central nervous system dynamics underpinning cognitive performance during gameplay and to elucidate potential correlations between neural signatures and individual task performance. Additionally, this reporting period included visits to care facilities and centers supporting individuals with cognitive impairments, including mild cognitive impairment (MCI) and dementia, in collaboration with local healthcare associations. These interactions provided valuable exposure to real-world clinical environments and end-user needs. Concurrently, progress was made on integrating virtual reality (VR) technologies into experimental protocols to enhance ecological validity in the assessment of emotional responses. A dedicated protocol was developed to examine physiological activity during immersive VR scenarios related to climate change, specifically simulating a wildfire event. The experimental setup combined wet-electrode EEG, electrocardiography (ECG), and EDA sensors, enabling synchronized acquisition of central and autonomic nervous system signals. This setup supports the investigation of complex emotional responses to immersive, emotionally salient VR content.
Scientific publications
EMBC 2025 (accepted contributions) – Copenhagen, Denmark
GEM 2025 (submitted contribution) - Kaohsiung, Taiwan
Mustafa Can Gursesli, Pietro Tarchi, Andrea Guazzini, Mirko Duradoni, Arda Tunc Yaras, Emir Akinci, Ulas Yurdakul, Federico Calà, Alessandro Tonacci, Daniele Vilone, Antonio Lanatà