Leader: Emanuele Menegatti (UNIPD); Other collaborator(s): Edoardo Trombin, Stefano Tortora, Alessio Palatella (UNIPD), UNICAL
This task will see the development of a software architecture to drive an assistive robot, controlled by neuromuscular signals in closed loop. New techniques (e.g., Deep Learning) to combine information from EEG, EMG and IMUs will be explored to enhance the robustness and reliability of the predictor of human intentions and motion in daily and ecological operations. In particular, The coupling between the user and the device, the concept of the human-in-the-loop, and the user’s experience while operating the assistive robots will be investigated to foster the adoption and learning of the use of these new technologies by older adults.
We devise the application of the above techniques also to exoskeletons and a smart suit prototyped by an SME in a cascade funding call. This smart suit will be made of technical cloth with embedded sensors and eventually embedded actuators. The smart suit will be tested in home environments, outdoors and in the workplace.
Brief description of the activities and of the intermediate results:
During the reported period, we prepared a systematic review on solutions to plan the motion of lower limb exoskeletons based on the obstacles and the structure of the environment. The paper has been proposed to IEEE Transactions on Robotics and we are waiting for an answer from the Editor in Chief. Additionally, we implemented an innovative solution for Adaptive Gait Planning in exoskeletons to plan their motion and steps in everyday environments. Our solution can detect and overcome obstacles on the floor, estimating their geometry with a 3D camera. We implemented obstacle detection, low-level control of the exoskeleton, and an odometry framework to estimate the exoskeleton's actual motion. Our motion generation is based on Kernelized Movement Primitives (KMPs), which are implemented in Python on a Dell XPS 17 laptop.
Initially, we focused on planning single steps of the exoskeleton, but we have now begun implementing multi-step behavior. The solution was initially implemented on the ALICE exoskeleton and is currently under testing on the UANGO exoskeleton, which we obtained as a loan from the company U&O Srl.To enable people to control assistive devices with BMIs (Brain-Machine Interface), we have been working on a technique to train them. Our approach involves collaborative training, where an expert user guides a novice user in learning BMI control. In this period, we successfully implemented the coupling between the two users and a robotic device, and tested the system with three pairs of users. Each pair underwent five training sessions. We collected the world's first dataset of this kind. During the training, the users wear a 32channel EEG cap and are asked to control the position of a robotic arm on a 2D plane. After each session, they complete two questionnaires: the NASA TLX and a custom questionnaire to investigate how users are feeling and how confident they are during the training.
We have investigated the use of decoders that exploit Riemannian geometry and compared their performance against decoders that use Euclidean geometry. The initial results show promising raw accuracy, and we are currently working on transforming the output of the classifier from discrete classes to continuous probabilities.
Main policy, industrial and scientific implications:
The control loop for environment-aware adaptive gait for intelligent exoskeletons can be applied to several types of exoskeleton and ported to different commercial models. The technique to train people to BMI using a collaborative approach can be a new way to speed up the uptake of assistive devices controlled by BCI.
Please see the next reporting period.
During this reporting period, a general framework for the Environment-Adaptive Gait Planning of exoskeletons has changed allowing easier testing of different gait planning techniques. It has been ported to ROS2. The problem of multi-step planning has been investigated, to evaluate the optimal positions of consecutive steps in the presence of obstacles for the exoskeleton. Different 3D mapping techniques developed by the mobile robotics community have been tested on point clouds gathered by the exoskeleton’s onboard camera. The kinematic model of the full exoskeleton has been inserted in the Reinforcement Learning (RL) algorithm which will generate the foot trajectory according to the 3D map of the local environment. This RL algorithm can produce a trajectory of good quality after just 13 minutes of training.
The first BCI algorithms have been tested to trigger the movement of the exoskeleton. The experiment was a success and shows the ability to assess the user intention by classifying the user’s EEG signals in two classes: “rest” vr. “movement”.
To enhance the robustness and reliability of the predictors of human intentions and motion in daily and ecological operations, we are studying the user’s learning process and the associated neural correlates. The data are represented as SPD matrices within the Riemannian manifold for monitoring and assessing the user’s learning in the large BCI datasets available in the IAS-Lab, which are spanning three years. The correlations between the data have been investigated exploiting different distances: on the original manifold, in the tangent Euclidean space, and in a newly derived space following dimensionality reduction to 3D and 2D. However, all these distances have highlighted some limitations. Thus, a new mathematical method has been conceived to represent data that cannot be adequately handled by the current state-of-the-art data reduction techniques. This new method has been tested to represent data of various datasets that involve different motor imagery tasks and a general approach for data visualization has been developed. Riemannian features are extracted from the data to evaluate the user’s learning.
In the first month, the Environment-Adaptive Gait Planning with KMPs (Kernelized Movement Primitives) was coded and tested with real data, yielding promising results. Additionally, the team attended Cybathlon 2024 in Zurich, securing second place in the BCI competition. The following month saw the successful testing of the KMP solution for obstacle avoidance on the ALICE LLE located at IAS-Lab. A paper on environment-adaptive Gait Planning through Reinforcement Learning in Lower-Limb Exoskeletons was prepared and submitted to an international conference. Furthermore, simulated tests began on a new Environment-Adaptive Gait Planning solution using Deep Reinforcement Learning. This new approach employs Soft Actor-Critic (SAC) to reason in the continuous space. Initial results are promising, although more time will be required to obtain a stable solution.
In the quarter's final month, further testing was conducted on the KMP solution, gathering standardised data in various stepping conditions. The results were largely successful, with only one collision occurring among 20 different steps. Future work will focus on reducing the significant computation time, which remains the main weakness of the method. After reaching a plateau during the validation of the new Reinforcement Learning solution, a change in approach was decided. Deep Reinforcement Learning will be used to upgrade the existing CFFTG, enabling generating an arbitrary number of waypoints through DRL.
Regarding the understanding of the key factors influencing user learning during BCI training, nn in-depth exploration of various parameters and data configurations was conducted in this research period. An extensive set of new data was acquired within the Riemann space, aiming to improve the clarity of user learning over time. Leveraging this data, a detailed exploration of rigorous methods was conducted to assess whether classifier recalibration was necessary at various stages. Multiple statistical criteria were evaluated to determine how adjustments to the training protocol might enhance the accuracy and reliability of performance tracking. This approach identified key indicators in the Riemann space for recalibration or protocol changes, creating a more adaptive training framework that could serve as a model for future studies on user learning in similar contexts.
Several datasets spanning multiple months for a single subject were discovered and utilized to evaluate methods for measuring user learning in motor imagery-based Brain-Computer Interfaces. After preprocessing the datasets to ensure consistency and compatibility with the existing code, interesting results began to emerge. Based on these findings, an investigation was initiated to determine whether adjusting parameters and the region of interest of the electrodes could influence the outcomes. Through systematic data analysis, possible patterns and correlations were identified, pointing to the most critical features.
Scientific publications
Dissemination Events