When recognizing images with complex scenes and obscure features such as weather images, there are often problems such as low recognition rate and feature redundancy. Based on this, an image classification algorithm based on deep transfer learning is proposed in this paper. The algorithm uses the model parameters of ImageNet dataset to construct three network models, ResNeXt, Xception and SENet, to extract image features, and uses a domain-adaptive discriminative joint distribution adaptive algorithm to resemble the feature vectors to complete a high-quality feature representation, and uses the result as a criterion to fuse the model features, and trains the fused features through a multilayer perceptron to achieve image classification with high accuracy recognition. The experimental results show that the algorithm outperforms the traditional single network model and further improves the upper limit of image classification accuracy.