As an Exploratory Action, this project aims at exploring original ideas to address difficult challenges.
Principal investigator : Claire Cury
Collaborators : Antoine Coutrot, Elise Bannier, Pierre Maurel
Neurofeedback techniques (NF) or restorative brain computer interfaces (BCI) consist in providing a subject with real-time feedback about its own brain activity, in order to learn self-regulate specific brain regions during NF training. Brain activity can be measured by various techniques such as EEG and/or fMRI. However, analysis of NF sessions is limited due to the difficulty at identifying the origin of failed training.
To enhance and monitor participant’s motivation in real-time during EEG-fMRI recording, bio-signal can be measured via eye-tracking (ET) or skin conductance (SC) devices.
For a precise evaluation of the motivation mental states of interest such as focus, arousal, mind wandering or mental load can be analysed.
The main objective of this project is to investigate measures from eye-tracking and skin conductance signals to evaluate in real-time subject’s motivation during NF training.
The project lies at the interface of behavioural neuroscience, si- gnal processing and neurofeedback. Neurofeedback approaches (NF) , also known as restorative brain-computer interface (restorative BCI), consist in pro- viding real-time feedback to a patient about his or her own brain activity in order to self-regulate brain areas or networks, targeted by the neural rehabilitation or by a given task. The estimation of neurofeedback scores is done through online brain functional feature extraction relying for the majority on electroencephalo- graphy (EEG) or functional magnetic resonance imaging (fMRI) and some very recent ones employing both for bi-modal EEG-fMRI NF sessions (i.e. NF scores are estimated synchronously by features from both modalities), providing a more specific estimation of the underlying neural activity. NF is a very promising brain rehabilitation technique for psychiatric disorders, stroke and other neurological pathologies , yet with moderate results.
One central question in NF training, is to identify the origin of a failure  ; it can be due to (a) the signal recording, (b) a too difficult task, (c) the patients inability to learn via NF, or (d) a lack of attention from the participant during the task (he/she should be preserved from boredom to not disengage from the task). Also, motivation should be enhanced, and participant’s prevailing attention should be monitored .
Although exceptional progress has been obtained during the past decades to explore the human brain, researches based on different neuro-imaging modalities are crucial to shed light on healthy and disordered human brains. Simultaneous EEG-fMRI recording has received recognition as a promising multi-modal mea- surement of brain activity. However, the investigation of correlations between EEG and fMRI signals revealed highly variable links with the task, the location in the brain, and used frequency bands in EEG signals.
My original hypothesis has a wider scope than neurofeedback, as it also concerns neuro-imaging in general. I assume that evaluating motivation during EEG-fMRI recording has the potential (1) to bring synergy between both signal (when motivation is feedback to the participant), (2) to help the validation of developed methods by selecting the most relevant part of both signals, and (3) to be added as co-factors for subsequent analysis of EEG and/or fMRI signals.
In behavioural neuroscience, eye-tracking (ET) and skin conductance (SC) devices can be used to measure different aspects of patient’s mental states rela- ted to focus, arousal, mind wandering, mental load or anxiety. All key indicators for a precise assessment of patient’s motivation.
Electro-dermal activity (EDA) is measured via SC system, that detects changes in the conductivity of the skin owing to perspiration corresponding to arousal, the so-called lies detector. EDA is a physiological signal associated with the autonomic nervous system, that comes from an electrical phenomenon caused by the glandular release of sweat during mental exertions. As EDA can reflect changes in cognitive or emotional state of an individual, it is widely used in psychology and psychiatry . The most frequently used measure of EDA is the skin conductance (SC). The SC signal can be modelled as a summation of two components, tonic and phasic. The tonic component includes slow drifts of the baseline skin conductance level and spontaneous fluctuations in SC . The phasic component is the skin conductance response (SCR), reflecting the direct response to the stimulus, arising within 5 seconds after the stimulus onset.
ET is a technology that measures eye movements at a high spatio-temporal resolution. Eye movements have been shown to reflect various dimensions of the participants mental state [5, 7]. Different metric have been used in the literature, such as pupil size, blink frequency, saccades amplitude or fixation duration. It is difficult to distinguish between mental load and arousal, as the augmentation of the mental load can lead to fatigue and loss of concentration, but can be also due to a strong arousal if the participant involves in the task. Extensive research has implicated pupil diameter as a key measure of attention, arousal and task engagement . The amplitude and speed of eyes saccades decrease with arousal and when tiredness rises . Mind wandering has been associated to lasting fixation (another eye-tracking measure), more frequent blinks, wider saccades and larger pupil diameter .
The main objective here is to accurately estimate, in real-time, participant’s motivation during a bi-modal EEG-fMRI recording (during a task or a NF session). I will also determine the potentials and limitations of the never-used ET and SC devices present at the Neurinfo platform. The short-term objective is to determine if SC and ET can accurately measure motivation in real time. The challenges of this Action Exploratoire are : (i) Synchronising ET and SC clocks to EEG and fMRI data streams, to obtain the most accurate real-time evaluation and to avoid cumulative delays. (ii) Identifying measures, robust across subjects, for motivation evaluation and changes detection. (iii) Modelling the impact of motivation measures on EEG and fMRI signals, and on EEG-fMRI signal coherence. No such original study has been carried out so far.
The long-term objectives of this project are two-fold. Those preliminary challenges, will allow to address new questions in NF, such as : for an optimised NF training, how modulating NF targets according to motivation measures ? Do these motivation measures improve the quality of NF trainings, to offer efficient brain rehabilitation to patients ? They will also allow to address new questions in signal processing, and neuro-imaging data fusion in general. As bringing synergy between EEG and fMRI signals might help extracting information from one modality to the other and better understand their relationship ; does motivation modulation during an EEG-fMRI functional imagery task can induce synergy between EEG and fMRI signals ? Can we use motivation measures as co- variables to study multi-modal data? As validation on real data is always an issue (due to the absence of ground truth and high level of noise in functional data), can this improve validation quality of developed methods ?
The innovative aspect of this project comes from the use of eye-tracking and skin conductance devices, new to the Empenn team (dedicated to neuro- imaging methods), to the NF community, but also to the neuroimaging data fusion domain whose aim is to understand the relation between different brain measures. Psychology and behaviour are also research domains unknown to the Empenn team, yet crucial to measure motivation. This project is then entirely new and exploratory to the Empenn team, but also for the NF community, and might have strong impact in the neuroimaging data fusion domain.
In this project Action Exploratoire, I will use those new devices, process and analyse new signals with specific and different properties, measure mental states of interest, and therefore rely on knowledge from psychology and behaviour, to estimate in real time the motivation level of a participant during an EEG-fMRI task or NF recording.
If monitoring focus and attention turns out to actually improve NF training quality and to bring better synergy between EEG and fMRI, measuring focus level could be democratised in future NF studies in the Empenn team, in the young and fast growing NF community, and in data fusion.
-  I. P. Bodala, J. Li, N. V. Thakor, and H. Al-Nashash. EEG and Eye Tracking De- monstrate Vigilance Enhancement with Challenge Integration. Frontiers in Human Neuroscience, 10, 2016. Publisher : Frontiers.
-  W. Boucsein. Electrodermal Activity. Springer Science & Business Media, Feb. 2012. Google-Books-ID : 6N6rnOEZEEoC.
-  D. Clewett, C. Gasser, and L. Davachi. Pupil-linked arousal signals track the temporal organization of events in memory. Nature Communications, 11(1) :4007, Aug. 2020. Number : 1 Publisher : Nature Publishing Group.
-  C. Cury, G. Lioi, L. Perronnet, A. Lécuyer, P. Maurel, and C. Barillot. Impact of 1D and 2D Visualisation on EEG-fMRI Neurofeedback Training During a Motor Imagery Task. In 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), pages 1018–1021, Apr. 2020. ISSN : 1945-8452.
-  T. Kootstra, J. Teuwen, J. Goudsmit, T. Nijboer, M. Dodd, and S. V. d. Stigchel. Machine learning-based classification of viewing behavior using a wide range of sta- tistical oculomotor features. Journal of Vision, 20(9) :1–1, Sept. 2020. Publisher : The Association for Research in Vision and Ophthalmology.
-  F. Nijboer, N. Birbaumer, and A. Kübler. The influence of psychological state and motivation on brain-computer interface performance in patients with amyotrophic lateral sclerosis – a longitudinal study. Frontiers in Neuroscience, 2010.
-  I. M. Pavisic, N. C. Firth, S. Parsons, D. M. Rego, T. J. Shakespeare, K. X. X. Yong, C. F. Slattery, R. W. Paterson, A. J. M. Foulkes, K. Macpherson, A. M. Car- ton, D. C. Alexander, J. Shawe-Taylor, N. C. Fox, J. M. Schott, S. J. Crutch, and S. Primativo. Eyetracking Metrics in Young Onset Alzheimers Disease : A Win- dow into Cognitive Visual Functions. Frontiers in Neurology, 8, 2017. Publisher : Frontiers.
-  R. Sitaram, T. Ros, L. Stoeckel, S. Haller, F. Scharnowski, J. Lewis-Peacock, N. Weiskopf, M. L. Blefari, M. Rana, E. Oblak, N. Birbaumer, and J. Sulzer. Closed-loop brain training : the science of neurofeedback. Nature Reviews Neu- roscience, 18(2) :86–100, Feb. 2017.
-  J. Smallwood and J. W. Schooler. The Science of Mind Wandering : Empiri- cally Navigating the Stream of Consciousness. Annual Review of Psychology, 66(1) :487–518, 2015.
-  B. Sorger, F. Scharnowski, D. E. Linden, M. Hampson, and K. D. Young. Control freaks : Towards optimal selection of control conditions for fMRI neurofeedback studies. NeuroImage, 186 :256–265, Feb. 2019.