In order to solve the LED recognition problem that the number formed by the factors such as illumination, background, and image distortion in natural scene, a recognition algorithm of LED-LeNet convolutional network is proposed. Firstly, the self collected LED light font data set was classified according to the number. Image data preprocessing includes image ROI operation, resolution adjustment to 32 × 32 and data enhancement. The network was reconstructed by convolution kernel, swish activation function and dropout regularization which referred to LeNet-5 network. The algorithm was verified by TST digital image database of traffic signal countdown collected in natural scene. The recognition accuracy of the algorithm can reach 99. 52%, and the recognition speed was 1 ms. The experimental results show that the algorithm has obvious advantages in recognizing LED light fonts after adjusting the network structure and convolution kernel parameters and changing the training strategy.