To help nursing staff in senile apartment find the elderly fall and other actions timely, an action recognition method based on video surveillance is proposed. Firstly, the foreground images are extracted by the GMM background modeling method in HS color space. Feature extraction is performed by combining the motion features and morphological features. And action recognition can be achieved by HMM with Gaussian output. The method proposed in this paper can adapt to the changes of illumination. The method also has good robustness to the change of motion direction and motion range, and the recognition accuracy rate reaches 90%. The result shows that the method can meet the basic requirements of action recognition and the method has certain practical value.