Assistive Control of a Hip Exoskeleton Robot, using a DQN-Adjusted Delayed Output Feedback Method

Document Type : Research Article

Authors

1 Department of Mechanical Engineering, Sadjad University, Mashhad, Iran

2 Center of Advanced Rehabilitation and Robotic Research (FUM-CARE), Mechanical Engineering Department, Ferdowsi University of Mashhad, Mashhad, Iran

Abstract

A major challenge in the development of an assistive exoskeleton robot is to design appropriate control algorithms. These algorithms should be trajectory-independent and require a minimum number of sensors to work in any intended motion and to be easily implementable. As a simple assistive strategy with all promising features, Delayed Output Feedback Control (DOFC) is shown to be effective in assisting the wearers in different types of motion. In this method, the assistive torques are defined in proportion to delayed feedback from the angle difference between the two legs. The authors have recently suggested an intelligent version of DOFC, in which a Deep Q-Network (DQN) was used to adjust the feedback delay according to the speed of the motion. Simulation studies were used to investigate the idea. By conducting some real-world experiments, the present paper extends the results to practical conditions. The provided results clearly verify that if the time delay is adjusted according to the walking speed, the DOFC method can effectively help the users in their motions of any speed. The results also indicated that a fixed or an inappropriate value of the delay may result in resistance against the user motion.

Keywords

Main Subjects

References

[1]  M. Grimmer, R. Riener, C.J. Walsh, A. Seyfarth, Mobility related physical and functional losses due to aging and disease-a motivation for lower limb exoskeletons, Journal of neuroengineering and rehabilitation, 16(1) (2019) 1-21.
[2] A. Rodríguez-Fernández, J. Lobo-Prat, J.M. Font[1]Llagunes, Systematic review on wearable lower[1]limb exoskeletons for gait training in neuromuscular impairments, Journal of neuroengineering and rehabilitation, 18(1) (2021) 1-21.
[3] F. Hussain, R. Goecke, M. Mohammadian, Exoskeleton robots for lower limb assistance: A review of materials, actuation, and manufacturing methods, Proceedings of the Institution of Mechanical Engineers, Part H: Journal of Engineering in Medicine, 235(12) (2021) 1375-1385.
[4] T. Wang, B. Zhang, C. Liu, T. Liu, Y. Han, S. Wang, J.P. Ferreira, W. Dong, X. Zhang, A Review on the Rehabilitation Exoskeletons for the Lower Limbs of the Elderly and the Disabled, Electronics, 11(3) (2022) 388.
[5] J. Zhou, S. Yang, Q. Xue, Lower limb rehabilitation exoskeleton robot: A review, Advances in Mechanical Engineering, 13(4) (2021) 16878140211011862.
[6] D. Shi, W. Zhang, W. Zhang, X. Ding, A review on lower limb rehabilitation exoskeleton robots, Chinese Journal of Mechanical Engineering, 32(1) (2019) 1-11.
[7] R. Baud, A.R. Manzoori, A. Ijspeert, M. Bouri, Review of control strategies for lower-limb exoskeletons to assist gait, Journal of NeuroEngineering and Rehabilitation, 18(1) (2021) 1-34.
[8] S. Luo, G. Androwis, S. Adamovich, H. Su, E. Nunez, X. Zhou, Reinforcement Learning and Control of a Lower Extremity Exoskeleton for Squat Assistance, Frontiers in Robotics and AI, 8 (2021).
[9] K. Kiguchi, K. Tamura, Y. Hayashi, Estimation of joint force/torque based on EMG signals, in: 2013 IEEE Workshop on Robotic Intelligence in Informationally Structured Space (RiiSS), IEEE, 2013, pp. 20-24.
[10] U. Nagarajan, G. Aguirre-Ollinger, A. Goswami, Integral admittance shaping: A unified framework for active exoskeleton control, Robotics and Autonomous Systems, 75 (2016) 310-324.
[11] G. Aguirre-Ollinger, J.E. Colgate, M.A. Peshkin, A. Goswami, Inertia compensation control of a one-degree[1]of-freedom exoskeleton for lower-limb assistance: Initial experiments, IEEE transactions on neural systems and rehabilitation engineering, 20(1) (2012) 68-77.
[12] G. Lv, J. Lin, R.D. Gregg, Trajectory-free control of lower-limb exoskeletons through underactuated total energy shaping, IEEE Access, 9 (2021) 95427-95443.
[13] T. Zhang, M. Tran, H. Huang, Admittance shaping-based assistive control of SEA-driven robotic hip exoskeleton, IEEE/ASME Transactions on Mechatronics, 24(4) (2019) 1508-1519.
[14] L.D. da Silva, T.F. Pereira, V.R. Leithardt, L.O. Seman, C.A. Zeferino, Hybrid Impedance-Admittance Control for Upper Limb Exoskeleton Using Electromyography, Applied Sciences, 10(20) (2020) 7146.
[15] K. Seo, K. Kim, Y.J. Park, J.-K. Cho, J. Lee, B. Choi, B. Lim, Y. Lee, Y. Shim, Adaptive oscillator-based control for active lower-limb exoskeleton and its metabolic impact, in: 2018 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2018, pp. 6752-6758.
[16] T. Xue, Z. Wang, T. Zhang, M. Zhang, Adaptive oscillator-based robust control for flexible hip assistive exoskeleton, IEEE Robotics and Automation Letters, 4(4) (2019) 3318-3323.
[17] H. Kalani, S.M. Tahamipour-Z, I. Kardan, A. Akbarzadeh, Application of DQN Learning for Delayed Output Feedback Control of a Gait-Assist Hip Exoskeleton, in: 2021 9th RSI International Conference on Robotics and Mechatronics (ICRoM), IEEE, 2021, pp. 341-345.
[18] B. Lim, J. Lee, J. Jang, K. Kim, Y.J. Park, K. Seo, Y. Shim, Delayed Output Feedback Control for gait assistance with a robotic hip exoskeleton, IEEE Transactions on Robotics, 35(4) (2019) 1055-1062.
[19] H. Kalani, A. Akbarzadeh, Application of Reinforcement Learning for Navigation of a Planar Snake Robot in Serpentine Locomotion, Journal Of Applied and Computational Sciences in Mechanics, 26(1) (2015) 97- 118.
[20] E.S. Low, P. Ong, K.C. Cheah, Solving the optimal path planning of a mobile robot using improved Q-learning, Robotics and Autonomous Systems, 115 (2019) 143-161.
[21] X. Wenxia, B. Yu, L. Cheng, Y. Li, X. Cao, Multi[1]fuzzy Sarsa learning-based sit-to-stand motion control for walking-support assistive robot, International Journal of Advanced Robotic Systems, 18(5) (2021) 17298814211050190.
[22] M. Gaeta, V. Loia, S. Miranda, S. Tomasiello, Fitted Q-iteration by Functional Networks for control problems, Applied mathematical modelling, 40(21-22) (2016) 9183-9196.
[23] Y. Ouyang, L. Dong, Y. Wei, C. Sun, Neural network based tracking control for an elastic joint robot with input constraint via actor-critic design, Neurocomputing, 409 (2020) 286-295.
[24] L.C. Melo, M.R.O.A. Máximo, Learning humanoid robot running skills through proximal policy optimization, in: 2019 Latin american robotics symposium (LARS), 2019 Brazilian symposium on robotics (SBR) and 2019 workshop on robotics in education (WRE), IEEE, 2019, pp. 37-42.
[25] A. Konar, I.G. Chakraborty, S.J. Singh, L.C. Jain, A.K. Nagar, A deterministic improved Q-learning for path planning of a mobile robot, IEEE Transactions on Systems, Man, and Cybernetics: Systems, 43(5) (2013) 1141-1153.
[26] M. Hamaya, T. Matsubara, T. Noda, T. Teramae, J. Morimoto, Learning task-parametrized assistive strategies for exoskeleton robots by multi-task Reinforcement Learning, in: 2017 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2017, pp. 5907-5912.
[27] X. Tu, M. Li, M. Liu, J. Si, H.H. Huang, A data-driven Reinforcement Learning solution framework for optimal and adaptive personalization of a hip exoskeleton, in: 2021 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2021, pp. 10610-10616.
[28] Y. Yuan, Z. Li, T. Zhao, D. Gan, DMP-based motion generation for a walking exoskeleton robot using Reinforcement Learning, IEEE Transactions on Industrial Electronics, 67(5) (2019) 3830-3839.
[29] D. Rastogi, Deep Reinforcement Learning for Bipedal Robots, (2017).
[30] I. Osband, C. Blundell, A. Pritzel, B. Van Roy, Deep exploration via bootstrapped DQN, Advances in neural information processing systems, 29 (2016).
 [31] X. Chen, C. Fu, J. Huang, A Deep Q-Network for robotic odor/gas source localization: Modeling, measurement and comparative study, Measurement, 183 (2021) 109725.
[32] X. Xue, Z. Li, D. Zhang, Y. Yan, A deep Reinforcement Learning method for mobile robot collision avoidance based on double dqn, in: 2019 IEEE 28th International Symposium on Industrial Electronics (ISIE), IEEE, 2019, pp. 2131-2136.
[33] Y. Liu, Y. Xu, Free Gait Planning of Hexapod Robot Based on Improved DQN Algorithm, in: 2020 IEEE 2nd International Conference on Civil Aviation Safety and Information Technology (ICCASIT, IEEE, 2020, pp. 488-491.
[34] L. Rose, M.C. Bazzocchi, G. Nejat, End-to-end deep Reinforcement Learning for exoskeleton control, in: 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC), IEEE, 2020, pp. 4294-4301.
[35] L. Rose, M.C. Bazzocchi, G. Nejat, A model-free deep Reinforcement Learning approach for control of exoskeleton gait patterns, Robotica, 40(7) (2022) 2189- 2214.
AUT Journal of Electrical Engineering
Volume 55, Issue 1
March 2023
Pages 99-106

Files

Share

How to cite

Statistics