In order to solve the defects such as inaccurate target location, target drift and recognition error induced by influence of illumination change, target rotation, occlusion in complex environment, a motion recognition algorithm based on double feature fusion and adaptive boosting was proposed. Firstly, in order to reduce the influence of illumination variation and occlusion on behavior, spatiotemporal context was used to extract the image sequences feature based on spatiotemporal context and the visual system characteristics. At the same time, the convolution neural network was introduced to operate the image sequence’s features for obtaining the STC and CNN features. Secondly, the principal component analysis operator was introduced to effectively combine the STC features and features to form a more accurate and complete feature representation. Then, by the new features, the adaptive boosting algorithm was used for classification training, the decision making of action was completed. The tests on the current popular data set show that, compared with the current commonly used behavior recognition methods, the proposed algorithm can recognize and understand all kinds of action, recognition rate is greatly improved, able to adapt for complex background and behavioral changes. This algorithm has higher accuracy and practical value in video surveillance and humancomputer interaction.