Coal and gangue have the characteristics of dense targets and small feature differences, and the recognition methods based on image processing generally have the problems of slow detection speed and low accuracy. To further improve the speed and accuracy of coal gangue detection, a GE-YOLOv5s coal gangue detection model is proposed. Firstly, Ghost Conv is introduced based on YOLOv5s instead of convolution operation, and a new module GhostCSP is designed to improve the detection speed of the model while realizing the lightweight of the network; secondly, the GC self-attention mechanism is added in the prediction layer, which integrates the lightweight of SENet and the advantage of global capture of long-distance information of NLNet, to enable the network to memorize and magnify the Then in the Neck part, a bidirectional feature pyramid network (BiFPN) structure is adopted, and BiFPN is used to fuse the features of three different dimensions to improve the computational efficiency of the model through the weighted feature fusion mechanism to further enhance the speed of coal gangue detection; finally, a new type of activation function is designed to replace the activation function of SiLU, which can improve the utilization rate and accelerate the convergence of the model. Finally, a new activation function Eswish is designed to replace the SiLU activation function, which improves the parameter utilization rate, accelerates the convergence speed of the model and improves the robustness. The experimental data show that compared with the YOLOv5s model, the number of parameters is reduced by 34.1% the amount of floating-point operations is reduced by 38.6%, and the mAP 0.5:0.95 index is improved by 1.9%. Comparison experiments show that the mAP 0.5:0.95 metric is improved by 16.6%, 4.8%, 13.6% and 3.8% compared to YOLOv3, SSD, FasterR-CNN, and YOLOv5-scSE, respectively. Applying the GE-YOLOv5s model to the gangue target detection process has better recognition performance, robustness, and network generalization ability, and can effectively avoid the phenomena of leakage, misdetection and overlapping.