In autonomous driving technology, multi-task environment perception algorithm is one of the key technologies to ensure the safe operation of driverless vehicles in complex traffic environments. In view of the poor robustness of the existing environment perception algorithms in dealing with complex driving scenarios caused by weather, illumination, occlusion and other factors, and the problems of missed detection, and blurred segmentation boundary, an improved high-performance hybrid network IPHNet based on HybridNets network is proposed to more accurately complete real-time perception tasks. This network uses a decoder-encoder structure and adopts an improved EfficientNetV2-S as the backbone network to enhance feature extraction capability and processing speed. By reconstructing BiFPN to increase the feature fusion of intermediate information of different levels, the lightweight up-sampling module DySample is introduced to reduce the complexity of the model and retain more information. The innovative design of the segmentation module (SPN) greatly ensures the integrity and accuracy of the underlying information extraction. Experiments on BDD100K dataset show that compared with the baseline network HybridNets, IPHNet achieves 81.4% mAP on vehicle detection tasks, which is improved by 4.1%. The accuracy of the lane line segmentation task reaches 86.84%, which is increased by 1.44%, and the IoU reaches 33.32%, which is increases by 1.72%. The mIoU of the drivable area division task is increased by 1.8%. The FPS reaches 34, which verified that IPHNet has a certain real-time processing capability.