Addressing the challenges of low measurement accuracy and restricted applicability range in existing passive sound source localization algorithms, this paper proposes a passive sound source localization estimation method based on an optimal quadruple-array. This method constructs an optimal quadruple-array structure to enable multi-element point sharing, aiming to achieve fusion localization estimation of the target sound source with a reduced total number of elements, thereby enhancing localization accuracy. Spatial target localization equations are derived from the array model, transforming the problem of solving position coordinates into that of determining time delay differences between array elements. Subsequently, a second-order fractional low-order covariance algorithm is employed to resolve the corresponding time delay differences between array elements in an impulse noise environment. After obtaining the self-fractional low-order covariance of array signals and the mutual-fractional low-order covariance between two array elements, the mutual-fractional low-order covariance of both is recalculated to further mitigate the impact of impulse noise and improve time delay estimation accuracy. Finally, the obtained time delay estimation information is incorporated back into the localization equation set to achieve localization estimation of spatial sound sources. The feasibility of the proposed method and the superiority of the array structure are validated through numerical simulations and field experiments. In the field experiments, the estimation error of sound source localization is only 0.085 1 meters, demonstrating the method’s capability to achieve high accuracy in sound source localization under impulse noise environments. This work extends the application scenarios of passive sound source localization algorithms and holds practical application value.