The triaxial array induction logging tool is an important tool for measuring complex formation information, but the key of design and application is to eliminate borehole effects of coplanar coil systems. The response characteristics of coplanar coil systems and coaxial coil systems of the triaxial array induction are analyzed numerically in the center hole by the FEM, and field experiment results are compared with modeling simulation on the coplanar coil systems. The results show that the borehole effects of coplanar coil systems are significantly different from that of coaxial coil systems besides the shortest subarray. The borehole effect causes the negative logging response. For the longest subarray, although the borehole effect is week, its skin effect is serious and the measurement range of formation conductivity is smaller than that of the coaxial coil systems. It reveals the reason that the greater difference is occurred between the coplanar coil systems and the coaxial coil systems through the eddy current distribution of coplanar coil systems around the borehole at different mud and formation conductivity. A new coil system which has two transmitters and one receiver is raised and its advantages are analyzed on the decreasing borehole effect and reducing negative response from the geometrical factor. The research lays a theoretical basis for the development and application of the triaxial induction logging tool.