In order to enhance the ability to resist plain attack and better the management in current optical image encryption technology. An asymmetric color image encryption algorithm based on chaotic Gyrator transform and matrix decomposition was designed in this paper. Firstly, the R, G and B components were obtained from the color image. A twodimensional coupled chaotic system was designed to generate three random phase masks for modulating R, G and B components respectively. And the modulated R, G and B components were fused into a grayscale image. Then the rotation angle of the Gyrator mechanism was calculated by the random sequence, and the Gyrator transform was used to process the grayscale image for obtaining the Gyrator spectrum. The Gyrator spectrum was processed based on the phaseamplitude truncation mechanism to output two coded images. Finally, the phase truncated image was decomposed into a unitary matrix and a triangular matrix based on matrix decomposition mechanism, and image encryption was completed by using the Gyrator transform of different rotation angles to process the unitary matrix and triangulation matrix. In the process of image encryption, the decryption key of R, G and B components were generated by using its amplitudetruncate image information which make its encryption and decryption key completely different, and the image can be decrypted by only using the private key. The experimental results show that the proposed algorithm has higher security level compared with the current optical image encryption technology.