EyeSkin-NF : Eye-tracking and skin conductance measures for neurofeedback analysis and validation.

As an Exploratory Action, this project aims at exploring original ideas to address difficult challenges.

Contributors

Principal investigator: Claire Cury

Collaborators: Hachim Bani, Elise Bannier, Antoine Coutrot, Pierre Maurel, Agustina Fragueiro, René-Paul Debroize

Publications

• Pilot study: Eye-tracking and skin conductance to monitor task engagement during bimodal neurofeedback. Agustina Fragueiro; René-Paul Debroize; Antoine Coutrot; Elise Bannier; Claire Cury
  SBI 2023 – IEEE International Symposium on Biomedical Imaging, Apr 2023, Carthagène des Indes, Colombia.

Abstract

Neurofeedback techniques (NF) or restorative brain-computer interfaces (BCI) consist in providing a subject with real-time feedback about its own brain activity, in order to learn self-regulate specific brain regions during NF training. Brain activity can be measured by various techniques such as EEG and/or fMRI. However, analysis of NF sessions is limited due to the difficulty in identifying the origin of failed training.
To enhance and monitor participants’ motivation in real-time during EEG-fMRI recording, bio-signal can be measured via eye-tracking (ET) or skin conductance (SC) devices.
For a precise evaluation of the motivation mental states of interest such as focus, arousal, mind wandering or mental load can be analysed.
The main objective of this project is to investigate measures from eye-tracking and skin conductance signals to evaluate in real-time the subject’s motivation during NF training.

Context

The project lies at the interface of behavioural neuroscience, signal processing and neurofeedback. Neurofeedback approaches (NF) [8], also known as a restorative brain-computer interface (restorative BCI), consist in providing real-time feedback to a patient about his or her own brain activity in order to self-regulate brain areas or networks, targeted by the neural rehabilitation or by a given task. The estimation of neurofeedback scores is done through online brain functional feature extraction relying for the majority on electroencephalography (EEG) or functional magnetic resonance imaging (fMRI) and some very recent ones employing both for bi-modal EEG-fMRI NF sessions (i.e. NF scores are estimated synchronously by features from both modalities), providing a more specific estimation of the underlying neural activity. NF is a very promising brain rehabilitation technique for psychiatric disorders, stroke and other neurological pathologies [8], yet with moderate results.

Motivation

One central question in NF training is to identify the origin of a failure [10]; it can be due to (a) the signal recording, (b) a too-difficult task, (c) the participants’ inability to learn via NF, or (d) a lack of attention from the participant during the task (he/she should be preserved from boredom to not disengage from the task). Also, motivation should be enhanced, and participants’ prevailing attention should be monitored [6].

Although exceptional progress has been obtained during the past decades to explore the human brain, researches based on different neuro-imaging modalities are crucial to shed light on healthy and disordered human brains. Simultaneous EEG-fMRI recording has received recognition as a promising multi-modal measurement of brain activity. However, the investigation of correlations between EEG and fMRI signals revealed highly variable links with the task, the location in the brain, and used frequency bands in EEG signals.

My original hypothesis has a wider scope than neurofeedback, as it also concerns neuro-imaging in general. I assume that evaluating motivation during EEG-fMRI recording has the potential (1) to bring synergy between both signals (when motivation is feedback to the participant), (2) to help the validation of developed methods by selecting the most relevant part of both signals, and (3) to be added as co-factors for subsequent analysis of EEG and/or fMRI signals.

In behavioural neuroscience, eye-tracking (ET) and skin conductance (SC) devices can be used to measure different aspects of a patient’s mental states related to focus, arousal, mind wandering, mental load or anxiety. All key indicators for a precise assessment of patient’s motivation.

Electro-dermal activity (EDA) is measured via SC system, which detects changes in the conductivity of the skin owing to perspiration corresponding to arousal, the so-called lies detector. EDA is a physiological signal associated with the autonomic nervous system, that comes from an electrical phenomenon caused by the glandular release of sweat during mental exertions. As EDA can reflect changes in a cognitive or emotional state of an individual, it is widely used in psychology and psychiatry [2]. The most frequently used measure of EDA is skin conductance (SC). The SC signal can be modelled as a summation of two components, tonic and phasic. The tonic component includes slow drifts of the baseline skin conductance level and spontaneous fluctuations in SC [2]. The phasic component is the skin conductance response (SCR), reflecting the direct response to the stimulus, arising within 5 seconds after the stimulus onset.

ET is a technology that measures eye movements at a high spatio-temporal resolution. Eye movements have been shown to reflect various dimensions of the participants’ mental state [5, 7]. Different metrics have been used in the literature, such as pupil size, blink frequency, saccades amplitude or fixation duration. It is difficult to distinguish between mental load and arousal, as the augmentation of the mental load can lead to fatigue and loss of concentration, but can be also due to a strong arousal if the participant involves in the task. Extensive research has implicated pupil diameter as a key measure of attention, arousal and task engagement [3]. The amplitude and speed of eyes saccades decrease with arousal and when tiredness rises [1]. Mind wandering has been associated to lasting fixation (another eye-tracking measure), more frequent blinks, wider saccades and larger pupil diameter [9].

Objectives

The main objective here is to accurately estimate, in real-time, participants’ motivation during a bi-modal EEG-fMRI recording (during a task or a NF session). I will also determine the potentials and limitations of the never-used ET and SC devices present on the Neurinfo platform. The short-term objective is to determine if SC and ET can accurately measure motivation in real-time. The challenges of this Action Exploratoire are : (i) Synchronising ET and SC clocks to EEG and fMRI data streams, to obtain the most accurate real-time evaluation and to avoid cumulative delays. (ii) Identifying measures, robust across subjects, for motivation evaluation and changes detection. (iii) Modelling the impact of motivation measures on EEG and fMRI signals and on EEG-fMRI signal coherence. No such original study has been carried out so far.

The long-term objectives of this project are two-fold. Those preliminary challenges, will allow addressing new questions in NF, such as: for an optimised NF training, how modulating NF targets according to motivation measures? Do these motivation measures improve the quality of NF training, to offer efficient brain rehabilitation to patients? They will also allow addressing new questions in signal processing, and neuro-imaging data fusion in general. As bringing synergy between EEG and fMRI signals might help extract information from one modality to the other and better understand their relationship; does motivation modulation during an EEG-fMRI functional imagery task can induce synergy between EEG and fMRI signals [4]? Can we use motivation measures as co-variables to study multi-modal data? As validation on real data is always an issue (due to the absence of ground truth and high level of noise in functional data), can this improve the validation quality of developed methods?

Exploratory aspect

The innovative aspect of this project comes from the use of eye-tracking and skin conductance devices, new to the Empenn team (dedicated to neuro-imaging methods), to the NF community, but also to the neuroimaging data fusion domain whose aim is to understand the relationship between different brain measures. Psychology and behaviour are also research domains unknown to the Empenn team, yet crucial to measure motivation. This project is then entirely new and exploratory to the Empenn team, but also for the NF community, and might have a strong impact in the neuroimaging data fusion domain.

In this project Action Exploratoire, I will use those new devices, process and analyse new signals with specific and different properties, measure mental states of interest, and therefore rely on knowledge from psychology and behaviour, to estimate in real-time the motivation level of a participant during an EEG-fMRI task or NF recording.

If monitoring focus and attention turns out to actually improve NF training quality and to bring better synergy between EEG and fMRI, measuring focus levels could be democratised in future NF studies in the Empenn team, in the young and fast-growing NF community, and in data fusion.

References