With the development of deep learning in recent years, remote sensing image fusion methods based on convolutional neural network were proposed and presented with good performance. Because there is no high-resolution multispectral image as a reference, the convolutional neural network is trained in the degraded images. The trained network is used to predict high resolution multispectral images. However, the fusion process of degraded images cannot reflect the fusion process of original images. In order to improve fusion performance, a semi-supervised fusion method based on convolutional neural network is proposed. The same fusion network is trained in the degraded image and the original image simultaneously. Because degraded image fusion has the corresponding reference image, the supervised learning method is used to train the fusion network. Moreover, the spectral loss is added to preserve the spectral information. However, there is no high-resolution multispectral reference image in the original image fusion. Spectral degradation network and spatial degradation network are designed to train the fusion network. The experimental results show that the proposed method is better than the compared method in preserving the spectral and details.