Aiming at the problem that vibration signal of inter-turn short circuit fault in permanent magnet synchronous motor (PMSM) is easily affected by noise and it is difficult to accurately extract the fault feature of it, an improved whale optimization algorithm (IWOA) optimized variational mode decomposition (VMD) denoising method is proposed and applied to vibration signal of inter-turn short circuit fault in PMSM. Firstly, the nonlinear convergence factor, adaptive weight and the Cauchy operator are introduced into the traditional whale optimization algorithm, and the IWOA algorithm is used to optimize the VMD parameters to achieve adaptive signal decomposition. Secondly, according to the principle of selecting the optimal intrinsic mode function based on multi-scale permutation entropy and variance contribution rate, the signal components are divided into the noise-dominated components and effective signal components. The noise-dominated components are denoised by the non-local mean filtering (NLM). Finally, the denoised and effective signal components are reconstructed as denoised signal. A motor short circuit fault model is established using ANSYS finite element software, and a short circuit fault experimental platform is built. Using this method to denoise the simulated and measured signals, it is further compared with many denoising methods such as wavelet threshold denoising method. The signal to noise ratio of the simulated signal is improved from 8 dB to 20.273 8 dB, and the signal to noise ratio of the measured signal is improved by 77.01% compared with wavelet threshold denoising method, which proved the effectiveness and practicality of the proposed method.