ISI4NAVE is an Associate Team between INRIA project-team Rainbow (Rennes), Aspire CREATe (UCL – London), Rehabilitation Center of Pôle Saint Hélier (Rennes) and LGCGM lab, funded from 2016 to 2021.
ISI4NAVE context and work program
Context
For a better quality of life
Using a wheelchair allows people with disability to compensate a loss of mobility. However only 5 to 15% of the 70 million people worldwide who require a wheelchair have access to this type of technical aid [O+15a]. One of the main reasons comes from the fact that some people are not allowed to drive a power wheelchair. When they are actually allowed to drive, they sometimes cannot manoeuvre it safely enough by themselves and thus need to be accompanied and supervised. In particular, visual, visuo-spatial and/or cognitive impairments can alter the ability of an individual to independently operate a wheelchair safely [CVM16].
In this context, mobility issues can be addressed from an individual-perspective by deploying adapted assistive technologies improving mobility in all types of environment. Mobility aids can be provided in the form of smart devices taking into account environmental constraints to assist the user mobility in daily life. Smart power wheelchairs have been then widely studied. In the previous ISI4NAVE Associate Team, we proposed several shared control solutions and proved their ability to insure the safety of the user [1,7-8]
In [LH18], it has been highlighted that to enhance the driving performances and the related user Quality of Experience, smart wheelchairs should be equipped with multi-modal interfaces which combines computer vision, touch, voice, and brain control.
These interfaces can be envisaged in the form of control tools or individually adapted feedback. Feedback can be necessary when users suffer from visual and/or cognitive impairments and are not able to clearly observe their unsafe trajectory. It can be seen as a communication channel between the user and the wheelchair controller. Such an active feedback can lead to minimal interferences from the automatic trajectory correction system as the user can gain awareness of the assistance provided by the system [Nel2007].
The other approach is to use these specific devices to efficiently share the control of the wheelchair with the robotic assistance. Typically, interactions can be obtained by means of innovative interfaces such as Brain-Computer Interfaces (BCI) and wearable haptics.
Designing physical Human-Robot Interaction devices and coupling such solutions with our driving assistance is then of major importance in order to notify the user of danger and guide him/her over to a safer zone. In our preliminary works, we demonstrated the relevance of such a framework [3]. The scope of this new ISI4NAVE Associate Team is then to provide advanced and innovative solutions for controlling wheelchair as well as providing appropriate and relevant feedback to users.
Objectives
Research directions
This project focuses on two main complementary objectives:
- to compensate both sensorimotor disabilities and cognitive impairments by designing innovative and adapted interfaces,
- to enhance the driving experience and to bring a new tool for rehabilitation purposes by defining efficient physical Human-Robot Interaction.
To assess the proposed solutions, we envisage performing clinical tests and case studies throughout the project in order to ensure compliance with user needs and acceptability.
The realization of this project requires five main tasks:
- definition and design of interfaces adapted to neurological diseases (mechatronics and robotics issues), e.g haptic interfaces, tactile interfaces, BCI, wearable haptics…
- definition of control strategies while using interfaces defined in task 1;
- definition of feedback devices;
- definition of a methodology related to clinical trials for evaluating both the system and the medical condition of the wheelchair user;
- experiments within the PAMELA Lab in UCL and clinical trials within the Pôle Saint Hélier.
Challenges
In order to ensure a widespread use of robotic systems, innovative interfaces, enabling relevant feedback (medically validated), constitute a major challenge. Trajectory corrections, obtained thanks to an assistance module, will have to be perceived by the user by means of sensitive (visual, tactile…) feedback that will have to be easily adapted to the pathology. Conversely, user interaction with the robotic system can be interpreted to control the wheelchair. Designing such systems require a multidisciplinary study, including medical data collection and analysis. To envisage a future transfer of the proposed technologies, design of pragmatic and cost efficient solutions remain a challenge that will be addressed in this ISI4NAVE Associate Team.
Methodology
Naturally, the proposed assistance solutions have to be compliant with both user needs and medical advice. The Pôle Saint Hélier is specialized in the rehabilitation of disabled people suffering from neurological diseases and has designed rigorous methodologies related to clinical trials. From this, the idea is to determine relevant measures that can characterize medical conditions of the patient who navigates with the proposed robotized wheelchair and adapted interfaces. These objective measurements should help the medical staff to evaluate the evolution of a given pathology as well as the relevance of the proposed feedback solutions.
Work program
2016 – First year
All the research topics in the project are collectively studied by the partners, who will continue to advance beyond the state of the art. The objective of the project is to help the research teams associating their complementary approaches. During the first year, we will start working on tasks 1, 2, 3, 4 and 5 that are described in previous section.
Considering task 1, related to the definition of a multimodal sensor platform, we will focus our work on the study on innovative sensors for secure navigation on sidewalks. The definition of a set of (relatively) low-cost sensors able to detect changes in level or curbs remains a challenging task. Then, the proposed platform will be integrated onto the robotized wheelchair and a navigation framework on sidewalks will be designed.
In task 2, the navigation process will rely on a multimodal sensor based servoing. By coupling dynamic obstacles detection and tracking, with a more local obstacle avoidance framework, we intend to obtain a robust and reactive navigation assistance framework.
For task 3, during this first year, we will determine with the help of the medical staff the desired functionalities of an adapted interface and biofeedback. This will require the definition of a dedicated methodology of tests. This work will be realized in conjunction with task 4.
The results obtained in task 1 (navigation on sidewalks) will be reused and tested within the PAMELA lab platform (UCL) before realizing clinical trials with inpatients of the Pôle Saint Hélier.
2017 – Second year
Considering task 1, related to the definition of a multimodal sensor platform, we will continue the study of the foreseen solution for secure navigation on sidewalks. We plan to verify the ability of ultrasonic sensors platform to detect changes in level or curbs and to be coupled with a dedicated shared control law.
In task 2, we intend to augment the ability of the proposed shared control solutions by developing dynamic obstacles detection and tracking. As the solutions developed during the first year are complementary, we intend to blend them in order to propose a unified, robust and reactive navigation assistance framework.
For task 3, during this second year, thanks to the medical recommendations and the preliminary tests collected during first year, we will design an adapted haptic joystick as well as visual feedback. From this, we expect to enhance the user experience and to augment the cognitive abilities of users. To measure this, we have to design a methodology devoted to the evaluation of the impact of navigation assistance on the cognitive point of view (task 4) before realizing intensive clinical trials (task 5).
The results obtained in tasks 1, 2 and 3 will be reused and tested within the PAMELA lab platform (UCL) before realizing clinical trials with inpatients of the Pôle Saint Hélier. In addition, in order to improve the navigation on crowded sidewalks, tests will be conducted in order to determine how wheelchair user and a crowd can interact: from these tests, an interaction model will be determined and will be reused to enhance the navigation strategies.
2018 – Third year
Considering tasks 1 and 2, related to the safe navigation on sidewalks, we will continue our study of our proposed control law. We will first conduct trials with the medical staff of Pôle Saint Hélier in order to assess the quality of experience related to this driving assistance (tasks 4 and 5). We also intend to design a dedicated shared control law that able to safely cross curb ramps. To this aim, we first have to determine adequate features corresponding to the geometry of the ramp in order to define a sensor-based servoing process. ToF IR sensors and US sensors will be used. Experiments will be conducted both in PAMELA (UCL) and in the Pôle Saint Hélier with the collaboration of medical staff (tasks 4 and 5). In addition, from the analysis of the social navigation data recorded during tests performed both in Pamela and Ker Lann platform this year, we will design an additional navigation task for outdoor human-aware navigation.
One of UCL’s partner charities, MERU, is currently designing a new version of the Bugzi wheelchair for children, with the view of enabling it to work in outdoor environments. This will require significant improvements to its sensing and safety capabilities. Therefore, the ISI4NAVE team will test the suitability of our curb detection system on the Bugzi wheelchair platform. Finally, regarding tasks 3 and 5, we will continue the development and testing of shared control for alternative wheelchair interfaces, i.e. for people with medical conditions that prevent them from using a conventional joystick [3, 8]. This will include discrete inputs such as buttons and head arrays, sip-and-puff switches and Brain-Computer Interfaces, with tests taking place both at UCL and Rennes.
2019 – Fourth year
All the research topics in the project are collectively studied by the 4 partners, who will continue to advance beyond the state of the art. During the first year, we will start working on scientific and technical tasks 1, 2 and 3 which are described in previous section.
Considering task 1, first developments will be oriented toward haptic devices. Rainbow has designed a preliminary version of a haptic bracelet. Tests within PAMELA facility in UCL has been conducted in August 2018 in order to first evaluate the ability of such a device to provide relevant feedback. In addition, we target some evolutions of the device in order to enhance its acceptance. This preliminary work confirms the necessity to develop adapted interfaces for advanced interactions. Inria will then work on designing an advanced version of the haptic bracelet, and LGCGM will propose a innovative design of a force feedback joystick that is compliant with partners’ wheelchairs and that matches user needs.
For task 2 and 3, Inria and UCL will couple their shared control framework with the interface in order to either control the wheelchair or to notify users of environment configurations.
Task 4 will be carried out by both Living Lab ISAR and UCL. In particular, ethical applications will be finalized and submitted to respectively the “Comité de Protection des Personnes” and the UCL Research Ethics Committee. Evaluation methodology, including definition of inclusion criteria, and usage scenarios will be defined. First trials (task 5) will be set up during the second semester. The idea is to characterize the Human-Robot Interactions both from a physical and a cognitive point of view
2020 – Fifth year
Considering tasks 1 and 3, Rainbow and Pôle Saint Hélier teams will continue to co-create innovative haptic modalities and will focus their work on the rehabilitation process of hemi-neglect patients. Considering tasks 4 and 5, new ethical applications have been submitted in October 2019 by Pôle Saint Hélier for trials that could be organized by the beginning of 2020, to evaluate the interest of the obstacle avoidance solution for people experiencing driving difficulties. Additional ethical applications will be submitted to evaluate the clinical benefit of both the haptic bracelets and the haptic joystick for power wheelchair regular users. Complementary user tests in PAMELA Lab facility will be conducted in 2020.
Rainbow will co-organise (if accepted) a workshop at IEEE ICRA 2020 on “Assistive robotic technologies for autonomous humans”, where human-robot physical interactions and haptic interfaces will be at the heart of the debates. Rainbow and Pôle Saint Hélier teams will welcome François Routhier (CIRRIS, Laval University, Québec) during one month. François Routhier is a specialist of robotic assistive devices and of tools and methods for clinical evaluations. He will be fully involved in the envisaged clinical trials: his expertise is complementary to the ISI4NAVE team expertises and his help will be fully appreciated.
