Deep Learning-based Subtask Segmentation method for Timed Up-and-Go Test with RGB-D camera

Abstract

Recently, our society has been suffering from a population aging problem due to an increase in life expectancy and a decrease in the fertility rate. According to statistics from the American Association of Medical Colleges (AAMC), it warned that the continuous and rapid aging rate will lead to a shortage of physical therapists. To overcome these problems and ease the burden on physical therapists, research is being conducted on simplification of complex rehabilitation systems and automation of gait assessment tools. Physical therapists use gait assessment tools to determine the functional level of the subject's gait ability before and after rehabilitation. Currently, clinical trials evaluate subjects' balance ability, functional mobility, and level of Activities of Daily Life (ADL) performance using various tools such as Berg balance scale, 10M walking test, and 6 Minute walking test. In this study, a Timed Up-and-Go test (TUG test) was selected as an assessment tool for automation and segmentation among various gait assessment tools. This is because the TUG test is relatively simple from the perspective of physical therapists and patients, so the burden is low, and it contains various activities within the test, so various clinical information can be extracted. The TUG test originated from the Get Up-and-Go test (GUG test). GUG test is a clinical method in which a physical therapist observes the movement of a patient and evaluates it on a 5-point scale. The subject waits while sitting on a chair, and the test begins according to the physical therapist's instructions. The subject gets up from his seat and walks 3M, turns around and comes back to sit on a chair. The version developed to increase objectivity and reliability in the GUG test is a TUG test. The TUG test is used to determine the risk of falls by measuring the total time to complete the test, and the functional evaluation is performed through the movements seen during the test. e.g., short stride, slow walking speed, slow rotation speed, etc. However, the traditional TUG test used in clinical practice evaluates the subject with the total time to complete the test according to the instructions of the therapist, so there is a problem such as subjective and rater biases. In addition, there is a limitation in that useful clinical information such as walking speed cannot be known because only the total time is measured as a result. To overcome these limitations, research on automation and segmentation of the TUG test is being conducted. Conventional automation studies can be classified into rule-based method and Artificial Neural Network-based (ANN-based) method. Rule-based automation is an approach to segmenting subtask using rules such as maximum, minimum, threshold, or peak detection based on acquired data. This method has the advantage of being simple to implement, but has the disadvantage of having a large deviation in classification results and requiring optimization depending on the subject. ANN-based method is a approach to segment subtasks by learning features of the data, so the classification deviation is smaller than rule-based methods, but they have the disadvantage of requiring a large number of data for training. In this paper, we propose a novel deep learning-based subtask segmentation of TUG test using a single Azure Kinect in order to overcome problems in the rule-based method with RGB-D cameras. To the author’s best knowledge, the RGB-D-based TUG subtask segmentation studies have not been conducted by DL approach but only in a rule-based manner. RGB-D camera (Azure Kinect) was used instead of a stopwatch to measure the entire process of the test. The proposed method obtained skeleton data of the subject, segmented the subtask of TUG test, and extracted clinical information on the time and speed of each motion as well as the total time based on the classification results. In this paper, proposed method was evaluated for the newly collected TUG data for healthy young and elderly people as well as stoke patients. To evaluate proposed method, an input comparison study was conducted to find the most suitable input for the proposed method, and as a result, a pelvis single input was selected. Similar to IMU-based research, it is assumed that it can expresses the movement of the subject because it is closest to the center of gravity. After the input was selected, DL model optimization study was conducted to optimize proposed method. Finally, performance was evaluated on three groups based on the optimized proposed method, and as a result, the performance of healthy young = 95.458%, healthy adult = 94.525%, stroke = 93.578% were shown. In addition, in this paper, a comparison with conventional vision-based TUG test automation studies was conducted to verify the performance of the proposed method. To the author’s best knowledge, the RGB-D-based TUG subtask segmentation studies have not been conducted by DL approach but only in a rule-based manner. Therefore, performance comparisons with conventional studies have been proven through two experiments. Furthermore, comparison with Bi-LSTM was also conducted to verify the performance of this study as much as possible. This Bi-LSTM is a famous deep learning network that is widely used in other time series and TUG automation studies. The test results show that the proposed method has better accuracy and robustness than those of the rule-based subtask segmentation method and ANN-based subtask segmentation. Furthermore, result of comparison with Bi-LSTM shows that the proposed method does not differ significantly by approximately 0.3% higher in terms of accuracy but requires approximately 13 times fewer parameters for training. This can be interpreted that the proposed method is more suitable for the development of 'self-diagnosis systems in home' through embeddings. On the other hand, all conventional research on automation of TUG tests conducted for specific target groups. e.g., automation for Parkinson's disease, automation for older adults, and automation for stroke. However, TUG test can be performed on a variety of patients, including Alzheimer's, osteoarthritis, stroke, Parkinson's disease, Mild Cognitive Impairment(MCI), Frailty, and so on. For these diverse patient populations, acquiring data every time and training deep models occurs a tremendous human resource and material waste. In other words, it is necessary to generalize so that the segmentation results can be used for various target groups. In this paper, the possibility of TUG test segmentation studies whether it can be applied to various patient groups without separate data collection was reviewed. Since the TUG test consists of simple daily life movements, it is assumed that it can be expressed only by moving the center of gravity. Accordingly, the generalization performance of the three groups collected by using the pelvis joint closest to the center of gravity as an input was evaluated. As a result, all groups showed relatively high generalization performance for pelvis input. As I expected, among them, the best generalization performance was shown when testing healthy young data after training for older adult data. Also, worst generalization performance was shown when testing health young data after training for stroke patient data. These results demonstrate the possibility that, regardless of the patient's type, TUG test can be automated and segmented with only pelvic input.