Along with the emergence of convolutional neural networks (CNN), pedestrian detection has been largely improved. Although CNN models can learn different variations of objects, pedestrian detection in autonomous driving still faces various challenges, which mainly include large scale variation, illumination variation and occlusion of different levels. In this paper, based on the previous CNN models, a robust pedestrian detection method is proposed. The main idea lies in combining the semantic information into the original detection framework for further supervision. It firstly extracts feature maps of different scales in CNN, based on the paved anchor boxes with various scales, and an additional convolutional layer is appended to be responsible for classification and regression. Meantime, semantic segmentation maps are generated from these feature maps. Finally, two streams are utilized to supervise detection and segmentation. Experiments on the recent CityPersons pedestrian detection dataset show that the semantic segmentation can significantly improve the detection accuracy without taking extra time, and the processing time is only 03 second per image in 1 280×384 pixels images in the dataset.