基于组合RNN网络的EMG信号手势识别

肌肉计算机接口(MCI)系统是虚拟现实、人机交互研究的热点之一, 其核心问题是EMG肌电信号分类, 因而MCI系统可以与深度学习方法有效结合。表面EMG信号分为高密度瞬时信号与稀疏多通道信号, 前者类似于图像, 可以采用CNN网络处理; 本文应用RNN网络对后者进行研究, 并利用MYO臂环实现了相应MCI系统。稀疏多通道EMG信号是不定长时间序列信号, 前后时间相关性高, 采用RNN网络进行分类。通过对原始信号进行时域、时频域、频域特征拓展, 获得原始信号的多流特征序列, 并提出两类组合RNN网络架构处理相应多流信号。用户依赖时算法准确率达90.78%, 非用户依赖的人群测试中手势识别准确率达78.01%, 实时动作识别准确率达82.09%, 算法能在61.7毫秒内识别手势动作。本文所提出的组合RNN网络方法可以有效区分基于EMG信号的不同动作, 且所设计的MCI系统用户泛化性与工作实时性表现好。

Abstract

The Muscle Computer Interface (MCI) system is one of the areas of active interest in virtual reality and human-computer interaction research. The main problem associated with the MCI was the EMG signal classification, to facilitate the effective combination of an MCI system with deep learning methods. Surface EMG signals include high-density transient signals and sparse multi-channel signals. The former was analogous to an image that can be recognized by a CNN network. The latter was studied in this investigationin which an MCI system with an MYO armband was realized. Sparse multi-channel EMG signals were long-term sequence signals with a high correlation between time and time that can be recognized by an RNN network. We proposd a combined RNN network architecture to recognize gestures with multi-stream feature sequence signals that were obtained by extending the original signals in the time-domain and time-frequency domain. The accuracy of the net is 90.78%. We perform cross-validation without a self-training set using 35 individuals, and the accuracy of the classification is 78.01%. The accuracy of real-time gesture recognition in the MCI system is 82.09%, and the action can be recognized within 61.7 milliseconds. We establish that the combined RNN nets can classify different gestures using EMG signals, and the MCI system performs well in generalization and real-time recognition.

参考文献

[1] WAHID M F, TAFRESHI R, AL-SOWAIDI M, et al.. Subject-independent hand gesture recognition using normalization and machine learning algorithms［J］. Journal of Computational Science, 2018, 27: 69-76.

[2] REAZ M B I, HUSSAIN M S, MOHD-YASIN F. Techniques of EMG signal analysis: detection, processing, classification and applications (Correction)［J］. Biological Procedures Online, 2006, 8(1): 163-163.

[3] DU Y, JIN W, WEI W, et al.. Surface EMG-based inter-session gesture recognition enhanced by deep domain adaptation［J］. Sensors, 2017, 17(3): 458.

[4] GENG W, DU Y, JIN W, et al.. Gesture recognition by instantaneous surface EMG images［J］. Scientific Reports, 2016, 6: 36571.

[5] ROJAS-MARTNEZ M, MAANAS M A, ALONSO J F. High-density surface EMG maps from upper-arm and forearm muscles［J］. Journal of Neuroengineering and Rehabilitation, 2012, 9(1): 85.

[6] ROJAS-MARTNEZ M, MAANAS M A, ALONSO J F, et al.. Identification of isometric contractions based on High Density EMG maps［J］. Journal of Electromyography and Kinesiology, 2013, 23(1): 33-42.

[7] ZHANG X, ZHOU P. High-density myoelectric pattern recognition toward improved stroke rehabilitation［J］. IEEE Transactions on Biomedical Engineering, 2012, 59(6): 1649-1657.

[8] ARIEF Z, SULISTIJONO I A, ARDIANSYAH R A. Comparison of five time series EMG features extractions using myo armband［C］. 2015 International Electronics Symposium (IES). IEEE, 2015: 11-14.

[9] LUH G C, MA Y H, YEN C J, et al.. Muscle-gesture robot Hand control based on sEMG signals with wavelet transform features and neural network classifier［C］. 2016 International Conference on Machine Learning and Cybernetics (ICMLC). IEEE, 2016, 2: 627-632.

[10] PATRICIA N, TOMMASIT T, CAPUTO B. Multi-source adaptive learning for fast control of prosthetics hand［C］. 2014 22nd International Conference on Pattern Recognition. IEEE, 2014: 2769-2774.

[11] SU Y, FISHER M H, WOLCZOWSKI A, et al.. Towards an EMG-controlled prosthetic hand using a 3-D electromagnetic positioning system［J］. IEEE Transactions on Instrumentation and Measurement, 2007, 56(1): 178-186.

[12] TAO W, LAI Z H, LEU M C, et al.. Worker activity recognition in smart manufacturing using IMU and sEMG signals with convolutional neural networks［J］. Procedia Manufacturing, 2018, 26: 1159-1166.

[13] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. Imagenet classification with deep convolutional neural networks［C］. Advances in Neural Information Processing Systems, 2012: 1097-1105.

[14] ABDEL-HAMID O, MOHAMED A, JIANG H, et al.. Convolutional neural networks for speech recognition［J］. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2014, 22(10): 1533-1545.

[15] YANG J, NGUYEN M N, SAN P P, et al.. Deep convolutional neural networks on multichannel time series for human activity recognition［C］. Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015.

[16] ATZORI M, COGNOLATO M, MLLER H. Deep learning with convolutional neural networks applied to electromyography data: A resource for the classification of movements for prosthetic hands［J］. Frontiers in Neurorobotics, 2016, 10: 9.

[17] WEI W, WONG Y, DU Y, et al.. A multi-stream convolutional neural network for sEMG-based gesture recognition in muscle-computer interface［J］. Pattern Recognition Letters, 2017.

[18] CHEOK M J, OMAR Z, JAWARD M H. A review of hand gesture and sign language recognition techniques［J］. International Journal of Machine Learning and Cybernetics, 2019, 10(1): 131-153.

[19] DU W, LI H. Vision based gesture recognition system with single camera［C］. WCC 2000-ICSP 2000. 2000 5th International Conference on Signal Processing Proceedings. 16th World Computer Congress 2000. IEEE, 2000, 2: 1351-1357.

[20] NEFIAN A V, GRZESCZUK R, ERUHIMOV V. Dynamic gesture recognition from stereo sequences: U.S. Patent 7, 274, 800［P］. 2007-9-25.

[21] PATSADU O, NUKOOLKIT C, WATANAPA B. Human gesture recognition using kinect camera［C］. 2012 Ninth International Conference on Computer Science and Software Engineering (JCSSE). IEEE, 2012: 28-32.

[22] LU W, TONG Z, CHU J. Dynamic hand gesture recognition with leap motion controller［J］. IEEE Signal Processing Letters, 2016, 23(9): 1188-1192.

[23] WANG P, SONG Q, HAN H, et al.. Sequentially supervised long short-term memory for gesture recognition［J］. Cognitive Computation, 2016, 8(5): 982-991.

[24] REN Z, MENG J, YUAN J. Depth camera based hand gesture recognition and its applications in human-computer-interaction［C］. 2011 8th International Conference on Information, Communications & Signal Processing. IEEE, 2011: 1-5.

[25] PU Q, GUPTA S, GOLLAKOTA S, et al.. Whole-home gesture recognition using wireless signals［C］. Proceedings of the 19th Annual International Conference on Mobile Computing & Networking. ACM, 2013: 27-38.

[26] GEORGI M, AMMA C, SCHULTZ T. Recognizing hand and finger gestures with IMU based motion and EMG based muscle activity sensing［C］. Biosignals, 2015: 99-108.

[27] SIMO M, NETO P, GIBARU O. Natural control of an industrial robot using hand gesture recognition with neural networks［C］. IECON 2016-42nd Annual Conference of the IEEE Industrial Electronics Society. IEEE, 2016: 5322-5327.

[28] TAN S, YANG J. WiFinger: leveraging commodity WiFi for fine-grained finger gesture recognition［C］. Proceedings of the 17th ACM International Symposium on Mobile Ad Hoc Networking and Computing. ACM, 2016: 201-210.

[29] WAN Q, LI Y, LI C, et al.. Gesture recognition for smart home applications using portable radar sensors［C］. 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE, 2014: 6414-6417.

[30] AMMA C, KRINGS T, BER J, et al.. Advancing muscle-computer interfaces with high-density electromyography［C］. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 2015: 929-938.

[31] MITRA S, ACHARYA T. Gesture recognition: a survey［J］. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 2007, 37(3): 311-324.

[32] MNTYJRVI J, KELA J, KORPIP P, et al.. Enabling fast and effortless customisation in accelerometer based gesture interaction［C］. Proceedings of the 3rd International Conference on Mobile and Ubiquitous Multimedia. ACM, 2004: 25-31.

[33] HAKONEN M, PIITULAINEN H, VISALA A. Current state of digital signal processing in myoelectric interfaces and related applications［J］. Biomedical Signal Processing and Control, 2015, 18: 334-359.

[34] PHINYOMARK A, PHUKPATTARANONT P, LIMSAKUL C. Feature reduction and selection for EMG signal classification［J］. Expert Systems with Applications, 2012, 39(8): 7420-7431.

[35] ZHANG X, CHEN X, LI Y, et al.. A framework for hand gesture recognition based on accelerometer and EMG sensors［J］. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 2011, 41(6): 1064-1076.

[36] JU Z, LIU H. A generalised framework for analysing human hand motions based on multisensor information［C］. 2012 IEEE International Conference on Fuzzy Systems. IEEE, 2012: 1-6.

[37] KAUFMANN P, ENGLEHART K, PLATZNER M. Fluctuating EMG signals: investigating long-term effects of pattern matching algorithms［C］. 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology. IEEE, 2010: 6357-6360.

[38] AL-TIMEMY A H, BUGMANN G, ESCUDERO J, et al.. Classification of finger movements for the dexterous hand prosthesis control with surface electromyography［J］. IEEE Journal of Biomedical and Health Informatics, 2013, 17(3): 608-618.

[39] LI Z, WANG B, YANG C, et al.. Boosting-based EMG patterns classification scheme for robustness enhancement［J］. IEEE Journal of Biomedical and Health Informatics, 2013, 17(3): 545-552.

[40] CZUSZYN＇SKI K, RUMIN＇SKI J, KWAS＇NIEWSKA A. Gesture recognition with the linear optical sensor and recurrent neural networks［J］. IEEE Sensors Journal, 2018, 18(13): 5429-5438.

[41] ELMAN J L. Finding structure in time［J］. Cognitive Science, 1990, 14(2): 179-211.

[42] MIAO Y, METZE F, RAWAT S. Deep maxout networks for low-resource speech recognition［C］. 2013 IEEE Workshop on Automatic Speech Recognition and Understanding. IEEE, 2013: 398-403.

[43] CHEN X, GUO H, WANG G, et al.. Motion feature augmented recurrent neural network for Skeleton-Based Dynamic Hand Gesture Recognition［C］. 2017 IEEE International Conference on Image Processing (ICIP). IEEE, 2017: 2881-2885.

[44] MURAKAMI K, TAGUCHI H. Gesture recognition using recurrent neural networks［C］. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1991: 237-242.

[45] VAMPLEW P, ADAMS A. Recognition and anticipation of hand motions using a recurrent neural network［C］. Proceedings of ICNN'95 - International Conference on Neural Networks, 1995.

[46] MARAQA M, ABU-ZAITER R. Recognition of arabic sign language (ArSL) using recurrent neural networks［C］. 2008 First International Conference on the Applications of Digital Information and Web Technologies (ICADIWT). IEEE, 2008: 478-481.

[47] XU S, XUE Y. A long term memory recognition framework on multi-complexity motion gestures［C］. 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR). IEEE, 2017, 1: 201-205.

[48] ENGLEHART K, HUDGINS B. A robust, real-time control scheme for multifunction myoelectric control［J］. Ieee Transactions on Biomedical Engineering, 2003, 50(7): 848-854.

[49] 梁浩, 刘克俭, 刘康, 等. 引入再检测机制的孪生神经网络目标跟踪［J］. 光学精密工程, 2019, 27(7): 1621-1631.

LIANG H, LIU K J, LIU K, et al.. Siamese network tracking with redetection mechanism［J］. Opt. Precision Eng., 2019, 27(7): 1621-1631. (in Chinese)

[50] 赵传, 张保明, 余东行, 等. 利用迁移学习的机载激光雷达点云分类［J］. 光学精密工程, 2019, 27(7): 1601-1612.

ZHAO CH, ZHANG B M, YU D X, et al.. Airborne LiDAR point cloud classification using transfer learning［J］. Opt. Precision Eng., 2019, 27(7): 1601-1612. (in Chinese)

[51] 方明, 孙腾腾, 邵桢. 基于改进YOLOv2的快速安全帽佩戴情况检测［J］. 光学精密工程, 2019, 27(5): 1196-1205.

FANG M, SUN T T, SHAO ZH. Fast helmet-wearing-condition detection based on improved YOLOv2［J］. Opt. Precision Eng., 2019, 27(5): 1196-1205. (in Chinese)

[52] CHEN I C, HILL J K, OHLEMLLER R, et al.. Rapid range shifts of species associated with high levels of climate warming［J］. Science, 2011, 333(6045): 1024-1026.

[53] DIETTERICH T G. Ensemble methods in machine learning［C］. International workshop on multiple classifier systems. Springer, Berlin, Heidelberg, 2000: 1-15.

[54] 邱青菊. 表面肌电信号的特征提取与模式分类研究［D］. 上海: 上海交通大学, 2009.

QIU Q J. Feature Extraction and Pattern Classification of Surface EMG Signals ［D］. Shanghai: Shanghai Jiaotong University, 2009 (in Chinese).

[55] SHI H, XU M, LI R. Deep learning for household load forecasting-A novel pooling deep RNN［J］. IEEE Transactions on Smart Grid, 2018, 9(5): 5271-5280.

[56] SAK H, SENIOR A, BEAUFAYS F. Long short-term memory recurrent neural network architectures for large scale acoustic modeling［C］. Fifteenth Annual Conference of the International Speech Communication Association, 2014.

[57] 王鑫, 吴际, 刘超, 等. 基于 LSTM 循环神经网络的故障时间序列预测［J］. 北京航空航天大学学报, 2018, 44(4): 772-784.

WANG X, WU J, LIU CH, et al.. Fault time series prediction based on LSTM cyclic neural network［J］. Journal of Beijing University of Aeronautics and Astronsutics, 2018, 44(4): 772-784. (in Chinese)

[58] RANJAN R, CASTILLO C D, CHELLAPPA R. L2-constrained softmax loss for discriminative face verification［J］. arXiv preprint arXiv: 1703.09507, 2017.

[59] KINGMA D P, BA J. Adam: A method for stochastic optimization［J］. arXiv preprint arXiv: 1412.6980, 2014.

[60] 谭宇彤, 周旭峰, 孔令芝, 等. 面向肌电信号的虚拟现实提线木偶动画研究［J］. 软件学报, 2019, 30(10): 2964-2985.

TAN Y T, ZHOU X F, KONG L ZH, et al.. Research on puppet animation controlled by electromyography (emg)in virtual reality environment［J］. Journal of Software, 2019, 30(10): 2964-2985. (in Chinese)

周旭峰, 王醒策, 武仲科, Vladimir Korkhov, Luciano Paschoal Gaspary. 基于组合RNN网络的EMG信号手势识别[J]. 光学精密工程, 2020, 28(2): 424. ZHOU Xu-feng, WANG Xu-feng, WU Zhong-ke, Vladimir Korkhov, Luciano Paschoal Gaspary. Gesture recognition with EMG signals based on ensemble RNN[J]. Optics and Precision Engineering, 2020, 28(2): 424.

基于组合RNN网络的EMG信号手势识别

关于本站 Cookie 的使用提示

全站搜索

基于组合RNN网络的EMG信号手势识别

相关论文

相关资讯

关于本站 Cookie 的使用提示

全站搜索