Aiming at the problem that the traditional Camshift algorithm is susceptible to the sudden acceleration and deceleration of the target and the background or the target interference of the similar colors when the intelligent vehicle is visually tracking the pilot vehicle in front, a pilot vehicle tracking algorithm that combines the improved Camshift and Kalman filter was proposed. The algorithm tracked the back projection of the three-dimensional histogram established by the target template hue, saturation, and edge gradient amplitude feature components. The Bhattachayya coefficient was used as the basis for determining the accuracy of target tracking. If the coefficient was greater than the set threshold, the target tracking would be judged to be inaccurate. At this time, the LBP cascade classifier was used to detect and recognize the pilot vehicle, and finally the Kalman filter was introduced to predict the position of the pilot vehicle in the next frame. The experimental results demonstrate that the proposed algorithm can accurately track the pilot vehicle in real time in a complex background.