Optical element defects will directly affect the performance of the entire optical system. In the detection of optical element defects, scratch defects are undoubtedly the difficulty of detection. The scratch defects have the problems of small size, large aspect ratio, and easy to be affected by impurities. In this paper, depth learning algorithm is applied to optical element defect detection, and according to the characteristics of scratch defects, the Mask R-CNN network model is improved. The algorithm also has a better detection effect on scratch defects. First, the original ResNet is replaced by CSPRepResNet proposed in this paper, and ESE attention mechanism is added to improve the ability of feature extraction and reduce the amount of computation. Secondly, K-means algorithm is used to recluster the length width ratio of anchor boxes. Thirdly, the loss function of target detection is changed from Cross Entropy to gradient balanced Focal Loss, which solves the problem of imbalance between positive and negative samples, is more conducive to the detection of difficult samples, and can also eliminate the influence of outliers. In general, the tested mAP@ . 5 The original 52. 1% is increased to 57. 3%, an increase of 5. 2%, and the reasoning speed is almost unchanged. It can be seen that the improved Mask R-CNN has a better detection effect on optical element scratch defects.