Aiming at the problems of small targets, scattered distribution, and susceptibility to background and noise interference in the process of transmission line insulator defect detection, an improved YOLOv8based defect detection method is proposed. Firstly, LSKNet is introduced to replace the original path aggregation network, enabling the model to adaptively select and adjust convolution kernel sizes based on the characteristics of different targets. This allows for more precise matching of target features and background information at various scales, significantly enhancing the robustness of defect recognition in complex scenarios. Furthermore, the SPPF-LSKA module is integrated into the network. By fusing global context information, this module greatly improves the aggregation efficiency and discriminative capability of multi-scale features, providing more refined feature representation for defect detection. Additionally, the proposed method incorporates a spatial attention mechanism into the neck network of YOLOv8, enhancing the model’s global feature comprehension and strengthening its focus on key information, particularly for small targets. To address the practical requirements of model efficiency and deployment, part of the conventional convolution layers in the neck network are replaced with GhostConv, effectively reducing the model’s parameter count and computational overhead. This achieves a balance between detection performance and resource efficiency. Experimental results demonstrate that the proposed method achieves a mAP of 93.1%, representing a 4.4% improvement compared to the original model, effectively enabling accurate detection of small targets.