Aiming at the problem of irregular and random distribution of defects on the surface of texture images, such as scratches and cracks, which leads to low accuracy of defect detection, a self-supervised defect detection method based on the bi-radial fusion of positive and negative sample difference features is proposed. Firstly, Otsu threshold segmentation is used to extract image foreground information, and Perlin noise is superimposed on the data-enhanced positive samples or the texture images, from the DTD dataset, to simulate defects on the positive sample images and synthesize the negative samples. Then, the mean-square error is calculated for feature matching using the intermediate features output from the encoder, while the Coordinate Attention (CA) and Path Aggregation Network (PANet) are combined to enhance the information fusion of the matched features. Finally, the fused features are input into the decoder together with the low-level and high-level features output from the encoder, and the weights of Focal, L1, and Dice loss functions are optimized and adjusted to realize the prediction of the defective masks more accurately. Experiments show that the average image-level and pixel-level AUROC of the proposed model on the texture category of the MVTec AD dataset reaches 0.995 and 0.968, respectively, which improves the classification and segmentation accuracies compared with the other defect detection models, demonstrating the effectiveness of the proposed method in texture defect detection.