Aiming at the problem that vibration signals collected in gear fault diagnosis are often accompanied by noise interference and fault features are difficult to extract, this paper is based on Fourier-Bessel series expansion (FBSE). A noise reduction method of gear vibration signal (FBSE-ESEWT), which combines FBSE and Energy Scale Space Empirical Wavelet Transform (ESEWT), is proposed. Firstly, the frequency spectrum of the acquired gear vibration signal is obtained by using FBSE technology to replace the traditional Fourier spectrum. Then, the obtained FBSE frequency spectrum is adaptive segmtioned and screened by using the energy scale space partitioning method to accurately locate the boundary points of the effective frequency band. Then the signal components are obtained by constructing wavelet filter banks and reconstructed to reduce noise and redundant information interference. Then, in order to capture more comprehensive feature information, the processed signal is transformed by generalized S-transform to obtain time-frequency graph, and 2D convolutional neural network is input for fault diagnosis to verify the feasibility of the algorithm. Through experiments on Simulink simulation signals and actual acquisition signals, the results show that compared with the original EWT, EMD and other methods, FBSE-ESEWT has better noise reduction effect, the signal-to-noise ratio is increased by 13.96dB, and the diagnosis accuracy is up to 98.03%.