Eduardo Javier Feria Rendón

Advisors

Daniel Krauß (M. Sc.), Dr. Eva Dorschky, Lukas Engel (M. Sc.), Prof. Dr. Björn Eskofier

Duration

09 / 2023 – 02 / 2024

Abstract

Smart home applications are increasingly finding their way into the everyday lives of many people. Audio and camera systems have already established voice and gesture control in some households. However, the audiovisual sensor technology is a relatively strong intrusion into people’s privacy. In this respect, mmWave radar sensors offer a decisive advantage since the data does not have any immediately recognizable personal reference. Unlike cameras, radars sensors are robust against changes in light conditions. Therefore, for both private and public buildings, radar sensors offer many attractive possibilities for performing tasks, which involve human behavior, even with high data protection requirements. They can be used, for example, to detect people in the building, their general movements, pose estimation [1], [2] and even to measure vital parameters [3].
Human pose estimation using radar data is a challenging task, since the spacial resolution of radar sensors is often limited. Each radar frame contains information about the location of objects in space and their corresponding radial velocities. The raw radar data is typically converted to point clouds, which are sparse and contain a variable number of points. The point clouds from radar data can be used in a number of tasks, from people detection, task identification, pose estimation, among others. Different methods have been proposed for extracting feature information from point clouds. For example, the points can be projected into orthogonal planes, resulting in matrix structures that can be processed with Convolutional Neural Networks [1]. Another approach is to create graphs by calculating adjacency matrices for each point and feed them to an ensemble of graph neural networks [4]. There are also works that take the point clouds as permutation invariant bags of points [5]. They use shared weights to extract features from each point independently, apply affine transformations to all points and extract global features by using pooling operations. However, current literature does not provide a comparison of different methods using point clouds or raw radar data for human pose estimation [2].
In this work, a 60 GHz FMCW MIMO radar of the mmWave family produced by Texas Instruments will be used in conjunction with a Microsoft Azure Kinect device to obtain ground truth data for the training and evaluation of our models. Due to it’s comparatively low number of antennas, the acquired radar data has a lower resolution and fewer points than usual will be available in the extracted point clouds. The recorded data will then be used in the adaption of at least two approaches such as PointNet [5] or PointNet++ [6] to the radar domain and evaluating them on the task of human pose estimation [7]. They will be compared to state-of-the-art algorithms on own and publicly available data using metrics such as the mean per joint error.

References

[1] A. Sengupta, F. Jin, R. Zhang, and S. Cao, “Mm-pose: Real-time human skeletal posture estimation using mmWave radars and CNNs,” IEEE Sensors Journal, vol. 20, no. 17, pp. 10 032–10 044, Sep. 2020. doi: 10.1109/jsen.2020.2991741. [Online]. Available: https://doi.org/10.1109%2Fjsen.2020.2991741.
[2] S.-P. Lee, N. P. Kini, W.-H. Peng, C.-W. Ma, and J.-N. Hwang, Hupr: A benchmark for human pose estimation using millimeter wave radar, 2022. arXiv: 2210.12564 [cs.CV].
[3] M. Mercuri, I. R. Lorato, Y.-H. Liu, F. Wieringa, C. V. Hoof, and T. Torfs, “Vital-sign monitoring and spatial tracking of multiple people using a contactless radar-based sensor,” Nature Electronics, vol. 2, no. 6, pp. 252–262, 2019.
[4] P. Gong, C. Wang, and L. Zhang, “Mmpoint-gnn: Graph neural network with dynamic edges for human activity recognition through a millimeter-wave radar,” in 2021 International Joint Conference on Neural Networks (IJCNN), 2021, pp. 1–7. doi: 10.1109/IJCNN52387.2021.9533989.
[5] C. R. Qi, H. Su, K. Mo, and L. J. Guibas, Pointnet: Deep learning on point sets for 3d classification and segmentation, 2017. arXiv: 1612.00593 [cs.CV].
[6] C. R. Qi, L. Yi, H. Su, and L. J. Guibas, “Pointnet++: Deep hierarchical feature learning on point sets in a metric space,” in Advances in Neural Information Processing Systems, I. Guyon, U. V. Luxburg, S. Bengio, et al., Eds., vol. 30, Curran Associates, Inc., 2017. [Online]. Available: https : / / proceedings . neurips . cc / paper _ files / paper / 2017 / file / d8bf84be3800d12f74d8b05e9b89836f-Paper.pdf.
[7] A. Singh, S. Sandha, L. Garcia, and M. Srivastava, “Radhar: Human activity recognition from point clouds generated through a millimeter-wave radar,” Oct. 2019, pp. 51–56, isbn: 978-1-4503-6932-9. doi: 10.1145/3349624.3356768.