Skeletal-based action recognition for medical condition detection using PoseConv3D

Early detection of medical conditions is important for elderly people, as early diagnosis can save them from getting late help and swift intervention. In this work, we are interested in developing PoseConv3D, a novel design to address such challenges raised in traditional GCNs with apply skeletal based action recognition. GCNs are good at analyzing human joint movements, however, they fail at processing noisy pose estimations, being unscalable in recognizing a group of activities, and integrating other data types. To overcome this limitation, PoseConv3D represents 2D skeletal data with time stacked heat maps, and analyzes them with a 3D convolutional neural network. A model targeting 12 medical conditions, including falls and posture related issues, is trained using the NTU RGB+D 120 dataset. The model was trained using Google Colab’s GPU as training and outperformed the GCN based approaches. We developed a user friendly system involving a React Native frontend that would allow users upload videos or URLs to be detected on the real time condition. Docker containerization allows for efficient deployment of the backend using Fast API and local systems, while providing a backend that also processes. The results show PoseConv3D is robust, scalable and accurate, and thus can be used as a real world medical condition detection and an elderly care tool.