In semiconductor manufacturing, wafer map defect detection is crucial for the rapid localization and identification of defects, which is significant for enhancing wafer product quality and production efficiency. However, existing methods have limitations, such as overly complex models and excessively deep network structures that struggle to leverage multi-level features for accurate classification. To address these issues, this paper combines a Stem-Dense feature extraction module with a multi-scale attention feature fusion module to propose a novel network architecture—multi-scale defect detection network with enhanced feature extraction (MSD-DFE). MSD-DFE effectively captures rich shallow feature information through the dense connection structure of Stem-Dense and multi-scale attention-based feature fusion technology, while significantly reducing the number of parameters and computational complexity of the model. The multi-scale feature extraction module integrates wafer map information from various scales, enhancing the model’s ability to extract defect features. Additionally, the introduced attention mechanism allows the model to focus more on defect areas, thereby improving classification accuracy. Experimental results show that MSD-DFE achieves an average accuracy of 97.4% on the WM-811K dataset, outperforming current mainstream methods, indicating its high potential for practical application in industrial settings.