Victoria Müller

Advisors

Robert Richer (M. Sc.), Luca Abel (M.Sc.), Prof. Dr. Katharina Schultebraucks (NYU Grossman School of Medicine), Prof. Dr. Nicolas Rohleder (Chair of Health Psychology), Prof. Dr. Björn Eskofier

Duration

12/2023 – 05/2024

Abstract

Stress, a ubiquitous presence in our daily lives, plays a dual role. While it is a natural bodily reaction to cope with challenges, excessive stress can heavily impact both physical health and mental well-being, leading to long-term sickness globally [1], [2], [3]. The body’s response to stress involves various neuroendocrine reactions, primarily through the sympathetic nervous system (SNS) and the hypothalamic-pituitary-adrenocortical (HPA) axis. These pathways are crucial for secreting key stress markers like alpha-amylase and cortisol [4]. While the effects of stress and its consequences are extensively studied, traditional methods for measuring neuroendocrine and electrophysiological markers are often labor-intensive for researchers. These methods are mostly invasive, particularly those assessing inflammatory responses which rely on blood samples [5]. Even noninvasive techniques tend to disrupt natural human behavior to some extent [6].

However, there’s a shift towards noninvasive, contactless stress measurement using video recordings to analyze facial expressions, voice, speech content, and head movements – known as digital biomarkers [7]. These markers are key indicators of human psychology and psychopathology [8]. Previous research has successfully linked distinct variations in facial expressions, speech, voice, and head movement features to mental and physical illnesses [9]. For example, digital biomarkers extracted from video-recorded interviews accurately predicted mental well-being in trauma survivors [10]. Studies have individually explored changes in digital biomarkers such as speech behavior [11], facial expressions [12], and movement [13] in response to acute stress. However, there is still a lack of comprehensive research that combines all these markers to achieve a more complete understanding of acute stress responses.

To gain a more comprehensive picture of stress responses, it is crucial to also consider internal physiological changes. The novel method of remote photoplethysmography (rPPG) can provide a promising solution by enabling the estimation of heart rate remotely from facial videos, eliminating the need for physical contact [14]. While current deep learning-based rPPG models have shown superior performance compared to conventional methods in controlled contexts, uncertainties persist regarding their effectiveness in scenarios encompassing diverse skin colors, varying heart rate levels, illumination disparities, and sudden head movements [15]. Moreover, there is a lack of prior testing of rPPG algorithms in an interview setting, where heart rate estimation occurs while participants are speaking.

The goal of this master’s thesis is, therefore, to explore the measurement of acute stress in a non-invasive and contactless manner using video analysis. It aims to develop a holistic approach to stress measurement by evaluating digital biomarkers derived from speech, facial expressions, and upper body movements. Leveraging data from the EmpkinS collaborative research center’s ongoing study [16], which involves participants undergoing the Trier Social Stress Test (TSST) [17] and its friendly control condition (f-TSST) [18], this project will enhance an existing digital biomarker pipeline. It will be investigated how facial expressions, speech, and movement changes due to stress and how these changes correspond to traditional biological and psychological stress markers. Further, this thesis aims to validate state-of-the-art deep learning rPPG methods in a more realistic setting addressing factors such as varying heart rate levels, speech sequences, and head movements.

Full Thesis

 

References

[1] D. B. O’Connor, J. F. Thayer, and K. Vedhara, “Stress and Health: A Review of Psychobiological Processes,” Annu. Rev. Psychol., vol. 72, no. 1, pp. 663–688, 2021, doi: 10.1146/annurev-psych-062520-122331.
[2] APA, “Stress in America 2019,” Am. Psychol., 2019.
[3] I. Polanowski, “Work-related stress, anxiety or depression statistics in Great Britain, 2021,” UK Health Saf., 2021.
[4] Y. M. Ulrich-Lai and J. P. Herman, “Neural regulation of endocrine and autonomic stress responses,” Nat. Rev. Neurosci., vol. 10, no. 6, pp. 397–409, Jun. 2009, doi: 10.1038/nrn2647.
[5] G. C. Slavish, J. E. Graham-Engeland, J. M. Smyth, and C. G. Engeland, “Salivary markers of inflammation in response to acute stress,” Brain. Behav. Immun., vol. 44, pp. 253–269, Feb. 2015, doi: 10.1016/j.bbi.2014.08.008.
[6] A. E. Kazdin, “Unobtrusive measures in behavioral assessment,” J. Appl. Behav. Anal., vol. 12, no. 4, pp. 713–724, 1979, doi: 10.1901/jaba.1979.12-713.
[7] T. R. Insel, “Digital phenotyping: technology for a new science of behavior,” JAMA, 2017 Oct; 318(13):1215–6.
[8] C. Darwin, (1872/1965). “The expression of the emotions in man and animals.” Chicago: University of Chicago Press.
[9] K. Schultebraucks, B. Chang, “Digital Biomarkers for predicting PTSD, Depression and Burnout in Emergency Department Clinicians.” Psychosomatic Medicine 2022; 84(5):A84.
[10] K. Schultebraucks, V. Yadav, A. Y. Shale, G. A. Bonanno, I. R. Galatzer-Levy. „Deep learning-based classification of posttraumatic stress disorder and depression following trauma utilizing visual and auditory markers of arousal and mood.” Psychol Med. 2022 Apr;52(5):957-967. doi: 10.1017/S0033291720002718. Epub 2020 Aug 3. PMID: 32744201.
[11] M. Oesten, R. Richer, L. Abel, N. Rohleder and B. M. Eskofier, “VoStress – Voice-based Detection of Acute Psychosocial Stress,” 2023 IEEE EMBS International Conference on Biomedical and Health Informatics (BHI), Pittsburgh, PA, USA, 2023, pp. 1-4, doi: 10.1109/BHI58575.2023.10313458.T
[12] G. Giannakakis, et al. “Stress and anxiety detection using facial cues from videos.” Signal Process. Control.31 (2017): 89-101.
[13] Richer, R., Koch, V., Abel, L., Hauck, F., Kurz, M., Ringgold, V., Müller, V., Küderle, A., Schindler-Gmelch, L., Eskofier, B. M., & Rohleder, N. (2024). Machine learning-based detection of acute psychosocial stress from body posture and movements. Scientific Reports, 14(1), 8251. https://doi.org/10.1038/s41598-024-59043-1.
[14] J. Allen (2007). Photoplethysmography and its application in clinical physiological measurement. Physiological Measurement, 28(3), R1. https://doi.org/10.1088/0967-3334/28/3/R01.
[15] “EmpkinS – Website für den SFB-Antrag Empathokinästhetische Sensorik.” https://empkins.de/ (accessed Jan. 04, 2023)
[16] C. Kirschbaum, K.-M. Pirke, and D. H. Hellhammer, “The ‘Trier Social Stress Test’ – A Tool for Investigating Psychobiological Stress Responses in a Laboratory Setting,” in Neuropsychobiology, 1993, vol. 28, no. 1–2, pp. 76–81. doi: 10.1159/000119004.
[17] S. Wiemers, D. Schoofs, and O. T. Wolf, “A friendly version of the Trier Social Stress Test does not activate the HPA axis in healthy men and women,” Stress, vol. 16, no. 2, pp. 254–260, Mar. 2013, doi: 10.3109/10253890.2012.714427.