In this paper, with the combination of convolutional autoencoders (CAE), an algorithm named stacked denoising autoencoders based on Fisher criterion (FSDAE) is proposed to solve the problem of the difficulty of manual features extraction and the limitation of defect samples on traditional fabric defect detection. Firstly, the sparse autoencoder (SAE) is used to obtain the sparse characteristics of the small patches cut out from the original images. Secondly, the CAE network parameters are initialized by using the sparse characteristics and the lowdimensional features of the original image are extracted. Finally, the features data are sent to the FSDAE network for defect detection and classification. The experimental results show that the algorithm can effectively extract the classification characteristics of the fabric image, and the detection rate of the fabric defect is improved by adding the Fisher criterion.