Ahmed Abdelhamid

Advisors

Wolfgang Mehringer (M. Sc.), Madeleine Flaucher (M. Sc.), Prof. Dr. Björn Eskofier, Prof. Dr. rer. nat. Andreas Hess

Duration

01 / 2023 – 07 / 2023

Abstract

The purpose of scientific visualization is to present data in a visual form that can be easily comprehended to make interpreting the data simpler [1]. As scientific technologies continue to advance, it is vital to have advanced visualization software that can handle large and complex datasets. This software should be able to handle the data in various forms, such as 2D, 3D, and even 4D, to provide a comprehensive understanding of the data. Additionally, it should be interactive and customizable to the user’s specific needs. This is particularly true in neuroscience, where there are numerous different brain imaging modalities and, as a result, a wide variety of potential analytical approaches and data representations [1]. The availability of models for data representation is limited by the different imaging modalities used to acquire the data. Each imaging modality, such as fMRI, MRI, or CT, has unique characteristics that affect the resolution, contrast, and overall quality of the resulting images. As a result, it is impossible to develop a single model that can effectively represent all data acquired from different imaging modalities.

In this thesis, the specific case of fMRI is researched. fMRI is a widely used technique for visualizing brain activity [2]. It measures changes in blood flow and oxygen levels in the brain, which are thought to be related to neural activity. There are several different methods for visualizing fMRI data, including activation maps, which show areas of the brain that are more active during a specific task or condition [3] connectivity maps, which show the functional connections between different regions of the brain [4], time-series analysis, which involves analyzing the changes in brain activity over time [5], and decoding, which involves analyzing the fMRI data to determine the mental state or cognitive process that a subject is experiencing [6].

Viewing 3D content on a 2D screen has substantial intrinsic limitations [7]. However, with the recent commercialization and popularization of virtual reality (VR), many neuroscientific researchers are hopeful that it may provide the next advancement in data-driven explorations and visualizations. Thus, it allows the creation of new hypotheses [7]. 3D brain visualization using VR offers several advantages over traditional 2D visualization on screens. One major advantage is the ability to provide a more immersive and interactive experience for the user. With VR, users can explore the brain from different perspectives, allowing them to gain a deeper understanding of the brain’s structure and function. Additionally, VR can provide a more realistic representation of the brain, allowing for the display of complex, multi-layered structures in their true 3D form. This can be especially useful for studying the brain in disease states or surgical planning.

Brain network connectivity maps or connectomes can be assessed using multimodal noninvasive neuroimaging techniques, and MRI is assumed to be the Gold-standard [8] for whole brain assessments. The structural connectivity is captured by anatomical data from the subcellular electron microscopic level up to diffusion MRI-derived white matter tracts. The functional connectivity is assessed over time as correlated activity across different single neurons or brain structures. A functional connectome can be displayed as a graph representing the functional similarity of the different brain structures as adjacency matrices. The latter can be visualized as node-link diagrams, in which nodes are positioned regarding the 3D anatomical positions of the brain structures and links, here edges, encode the connectivity between each pair of nodes [8]. Such node-link diagrams offer a clear picture of the entire 3D brain network, making it simple to see connections between functionally connected nodes.

The main goal of this thesis is to provide solutions for highly interactive 3D visualization of such node-link brain connectome data using a VR headset. We intend to develop a software tool to create node-link diagrams of the brain’s connectome. This software tool will include features available in most VR applications, such as grabbing, rotating, and scaling. Also, it will include application-specific features. First, filtering nodes and edges by anatomical/functional labeling, by their activation thresholds interactively in 3D and value-based. Second, saving the full model and features after manipulation. Third, taking snapshots and exporting images. Fourth, load two datasets on two different canvases. Furthermore, as input data, the functional connectome data used in our VR software tool is extracted from several lab software packages in .json format. The node spatial encoding is provided by a text-based atlas file containing 3D positions, coloring schemes, and functional assessments for all relevant brain structures. Besides these node positions, Amira [9] is a software application for processing, analyzing, and visualizing 3D and 4D data, which will provide 3d surface models of the brain structures. The VR implementation will be done in the Unity framework.

Since the primary objective of this thesis is to offer a 3D visualization/interaction solution for the brain’s connectome as an alternative to the traditional 2D visualization techniques on computers, we will conduct a usability case study with 10 participants to be able to evaluate the
value of using our virtual reality headsets in brain visualization. As a use case for our study, we will use resting brain state data of Crohn’s disease patients being treated with the anti-TNF drug Remicade and compare it to age and sex-matched controls.

References
[1] Combrisson, Etienne ; Vallat, Raphael ; O’Reilly, Christian ; Jas, Mainak ; Pascarella, Annalisa ; Saive, Anne L. ; Thiery, Thomas ; Meunier, David ; Altukhov, Dmitrii; Lajnef, Tarek ; Ruby, Perrine ; Guillot, Aymeric ; Jerbi, Karim: Visbrain: A multipurpose GPU-accelerated open-source suite for multimodal brain data visualization. In: Frontiers in Neuroinformatics 13 (2019), 3. http://dx.doi.org/10.3389/fninf.2019.00014. – DOI 10.3389/fninf.2019.00014. – ISSN 16625196
[2] Logothetis, Nikos K. ; Pauls, Jon ; Augath, Mark ; Trinath, Torsten ; Oeltermann, Axel: Neurophysiological investigation of the basis of the fMRI signal. In: Nature 412 (2001), 7, S. 150–157. http://dx.doi.org/10.1038/35084005. – DOI 10.1038/35084005. – ISSN 0028–0836
[3] Eickhoff, Simon B. ; Stephan, Klaas E. ; Mohlberg, Hartmut ; Grefkes, Christian; Fink, Gereon R. ; Amunts, Katrin ; Zilles, Karl: A new SPM toolbox for combining probabilistic cytoarchitectonic maps and functional imaging data. In: NeuroImage 25 (2005), 5, S. 1325–1335. http://dx.doi.org/10.1016/j.neuroimage.2004.12.034. – DOI 10.1016/j.neuroimage.2004.12.034. – ISSN 10538119
[4] Smith, Stephen M. ; Fox, Peter T. ; Miller, Karla L. ; Glahn, David C. ; Fox, P. M.; Mackay, Clare E. ; Filippini, Nicola ; Watkins, Kate E. ; Toro, Roberto ; Laird, Angela R. ; Beckmann, Christian F.: Correspondence of the brain’s functional architecture during activation and rest. In: Proceedings of the National Academy of Sciences 106 (2009), 8, S. 13040–13045. http://dx.doi.org/10.1073/pnas.0905267106. – DOI 10.1073/pnas.0905267106. – ISSN 0027–8424
[5] Calhoun, Vince D. ; Kiehl, Kent A.; Pearlson, Godfrey D.: Modulation of temporally coherent brain networks estimated using ICA at rest and during cognitive tasks. In: Human Brain Mapping 29 (2008), 7, S. 828–838. http://dx.doi.org/10.1002/hbm.20581. – DOI 10.1002/hbm.20581. – ISSN 10659471
[6] Naselaris, Thomas ; Prenger, Ryan J. ; Kay, Kendrick N. ; Oliver, Michael ; Gallant, Jack L.: Bayesian Reconstruction of Natural Images from Human Brain Activity. In: Neuron 63 (2009), 9, S. 902–915. http://dx.doi.org/10.1016/j.neuron.2009.09.006. – DOI 10.1016/j.neuron.2009.09.006. – ISSN 08966273
[7] Ard, Tyler ; Krum, David M. ; Phan, Thai ; Duncan, Dominique ; Essex, Ryan ; Bolas, Mark ; Toga, Arthur: NIVR: Neuro imaging in virtual reality, IEEE Computer Society, 4 2017. – ISBN 9781509066476, S. 465–466
[8] Keiriz, Johnson J. ; Zhan, Liang ; Ajilore, Olusola ; Leow, Alex D. ; Forbes, Angus G.: NeuroCave: A web-based immersive visualization platform for exploring connectome datasets. In: Network Neuroscience 2 (2018), 1, S. 344–361. http://dx.doi.org/10.1162/netn-a-00044. – DOI 10.1162/netn–a–00044. – ISSN 24721751
[9] Stalling, Detlev ; Westerhoff, Malte ; Hege, Hans C.: amira: A Highly Interactive System for Visual Data Analysis. In: Visualization Handbook (2005), 1, S. 749–767. http://dx.doi.org/10.1016/B978-012387582-2/50040-X. – DOI 10.1016/B978–012387582–2/50040–X. ISBN 9780123875822