This book provides a series of gesture and behavior recognition methods based on multimodal datarepresentation. The data modalities include image data and skeleton data, and the modeling methods includetraditional codebook, topological graph, and LSTM architectures. The tasks include single gesture recognitionclassification, single action recognition classification, continuous gesture classification, complex behaviorclassification of human interaction and other tasks of different complexity. This book focuses on the dataprocessing methods of each modality, and the modeling methods for different tasks. We hope the reader canlearn basic gesture and action recognition methods from this book, and develop a model system that suits theirneeds on this basis.
This book can be used as a textbook for graduate, postgraduate and PhD students majoring in computerscience, automation, etc. It can also be used as a reference for the reader who is interested in gesturerecognition, human action interaction, sequence data processing, and deep neural network design, and whohopes to contribute to the fields.

张亮，男，汉族，1981年5月生，西安电子科技大学教授，博士生导师，本硕博毕业于浙江大学，现任西安电子科技大学计算机科学与技术学院“嵌入式技术与视觉处理中心”主任，全国计算机学会嵌入式专委会委员，IEEE会员，ACM会员。主要研究方向为深度学习、手势手语识别、场景语义理解、嵌入式多核系统等，作为负责人先后承担国家重点研发计划、国家自然科学基金及企业合作项目多项。

Chapter 1 Human Action Recognition Using MultMayer Codebooks of Key Poses and Atomic Motions
1.1 Introduction
1.2 Related Work
1.2.1 Feature Representation
1.2.2 Classification Model
1.3 Construction of Multi-layer Codebook
1.3.1 Feature Representation
1.3.2 Feature Sequence Segmentation
1,3.3 Pose-layer Codebook
1.3.4 Motion-layer Codebook
1.3.5 Multi-layer Codebook Construction
1.4 Classification Methods
1.4.1 Naive Bayes Nearest Neighbor
1.4.2 Support Vector Machine
1.4.3 Random Forest
1.5 Experimental Results
1.5.1 Experiments on the CAD-60 dataset
1.5.2 Experiments on the MSRC-12 dataset
1.5.3 Discussion
1.6 Conclusion and Future Work
Acknowledgements
References
Chapter 2 Topology-learnable Graph Convolution for Skeleton-based Action Recognition
2.1 Introduction
2.2 Related Work
2.2.1 Graph Convolutional Network for Action Recognition
2.2.2 Adaptive Graph Convolution
2.3 Topology-learnable Graph Convolution
2.3.1 Graph Convolution
2.3.2 Graph Topology Analysis
2.3.3 Topology-learnable Graph Convolution
2.3.4 Topology-learnable GCNs
2.4 Experiments
2.4.1 Datasets
2.4.2 Ablation Study
2.4.3 Comparison with the State-of-the-art Methods
2.4.4 Discussion
2.5 Conclusion
Acknowledgements
References
Chapter 3 Recurrent Graph Convolutional Networks for Skeleton-based Action Recognition
3.1 Introduction
3.2 Related Work
3.2.1 Graph Convolution for Action Recognition
3.2.2 LSTM on Graphs
3.3 Recurrent Graph Convolutional Network
3.3.1 Graph Convolution
3.3.2 Adaptive Graph Convolution
3.3.3 Recurrent Graph Convolution
3.3.4 Recurrent Graph Convolutional Network
3.4 Experiments
3.4.1 Datasets
3.4.2 Training Details
3.4.3 Ablation Study
3.4.4 Comparison with the State-of-the-art Methods
3.4.5 Visualization of the Evolved Graph Topologies
3.5 Conclusion
Acknowledgements
References
Chapter 4 Graph-temporal LSTM Networks for Skeleton-based Action Recognition
4.1 Introduction
4.2 Related Work
4.3 GT-LSTM Networks
4.3.1 Pipeline Overview
4.3.2 Topology-learnable ST-GCN
4.3.3 GT-LSTM
4.3.4 GT-LSTM Networks
4.4 Experiments
4.4.1 Datasets
4.4.2 Training Details
4.4.3 Ablation Study
4.4.4 Comparison with the State-of-the-art Methods
4.5 Conclusion
References
Chapter 5 Spatio-temporal Interaction Graph Parsing Networks for Human-object Interaction Recognition
5.1 Introduction
5.2 Related Work
5.3 Overview
5.4 Proposed Approach
5.4.1 Video Feature Extraction
5.4.2 Spatio-temporal Interaction Graph Parsing
5.4.3 Inference
5.4.4 Implementation Details
5.5 Experiments
5.5.1 Dataset
5.5.2 Ablation Study
5.5.3 Comparison with the State-of-the-arts Methods
5.5.4 Visualization of Parsed Graphs
5.6 Conclusion
Acknowledgements
References
Chapter 6 Learning Spatio-temporal Features Using 3DCNN and Convolutional LSTM For Gesture Recognition
6.1 Introduction
6.2 Related Work
6.3 Method
6.3.1 2D Spatio-temporal Feature Map Learning
6.3.2 Classification Based on the 2D Feature Maps
6.3.3 Network Training
6.4 Experiments
6.4.1 Datasets
6.4.2 Implementation
6.4.3 Architecture Analysis
6.4.4 Comparison with the State-of-the-art Methods
6.5 Conclusion
Acknowledgements
References
Chapter 7 Multimodal Gesture Recognition Using 3D Convolution and Convolutional LSTM
7.1 Introduction
7.2 Related Work 1 l
7.2.1 Handcrafted Feature Based Methods
7.2.2 Neural Network Based Methods
7.3 Proposed Method
7.3.1 Input Preprocessing
7.3.2 3DCNN
7.3.3 Convolutional LSTM
7.3.4 Spatial Pyramid Pooling
7.3.5 Multimodal Fus