A joint infrared video pedestrian method is proposed based on Gauss background modeling and convolution neural network, to address the problems of inability to extract foreground targets and low detection rate in traditional method. Firstly, for continuous sequence infrared images, the foreground targets are extracted by the mixture Gaussian model. Then, for the foreground targets which are occluded by pedestrians, the valley bottom of the luminance curve is used as the segmentation point. While, the single pedestrian target area is separated by directional projection. Finally, the determined region of interest is input into the trained LeNet-7 system. Experiments on three different test sets demonstrate that the proposed method has good detection effect. Compared with the traditional method, the detection rate of the proposed method is over 99%, and the false alarm rate is as low as 0%.