In order to precision measure interpolation error of photoelectric encoder at work and improve the subdivision precision of encoder, the interpolation error measuring analysis method of grating signal in photoelectric encoder based on improved genetic algorithm was proposed. The basic principles of genetic algorithms and implementations were explained. The algorithm was improved and optimized. Using the collected discrete data of signal, the improved genetic algorithm extracted parameters from optical signal waveform. The influence of the signal quality in the encoder interpolation error was analyzed. According to parameters of the extracted signal (DC component, amplitude, frequency and phase) , the interpolation error of photoelectric encoder was measured. Experimental results show that parameters of the extracted signal through improved genetic algorithm was Shigh precision and quick speed in operating. The maximum and the minimum interpolation error were +2.51 and -4.52". This method can effectively measure the photoelectric encoder interpolation error. It may play an important role in the further research of encoder signals about compensation and correction.