Pedestrian detection is the principal technique for various applications, such as surveillance, tracking system and autonomous driving. Although the topic has been intensively investigated and significant improvement has been achieved in recent years, pedestrian detection is still a challenging task, limited by occluded appearances, cluttered backgrounds, and low image resolution. Besides, since most of recent researches focus on the detection of pedestri-ans in visible spectrum images, they are very likely to be stuck with images captured at night or bad lighting. However, ambient lighting has less effect on thermal imaging. Thermal images usually present clear silhouettes of human, but lose fine visual details of pedestrian, which can be captured by visual cameras. To overcome the drawbacks of visi-ble images, it’s helpful to fuse the information of visible images and long wave length infrared images. Aggregate channel feature is an easy but useful way to detect pedestrian. However, it only uses the information of visible spec-trum images. For the above reasons, an improved pedestrian detection algorithm based on multispectral aggregate channel features is proposed. First, the aggregate channel features of the visible image and the infrared image are extracted, respectively. Specifically, the channel features extracted from the visible images include three LUV color channel features, one normalized gradient magnitude channel feature, and six histogram of oriented gradients channel (HOG) features. The channel features extracted from infrared images include one brightness channel feature and nine HOG features. All the channel features make up the aggregate multispectral channel features. Then, to use the symmetry information of pedestrian in infrared images, the improved central symmetric local binary pattern is proposed. The improved pattern feature is achieved by changing the pixel contrast rule and comparing the contrast result with the adaptive threshold. The improved central symmetric local binary pattern feature is added to feature channels to get the aggregate multispectral channel features. Next, to learn more local features and observe the ef-fect of filters, different filter banks are designed to filter the aggregate multispectral channel features. Finally, the real adaptive boosting learning method is used to train the classifier to realize the multispectral pedestrian detection. Ex-periments show that the improved local binary pattern feature can better describe the symmetry of pedestrians of infrared images and the intermediate filter layer enriches the candidate feature pool. The algorithm makes use of the complementary information provided by color and thermal images, which can effectively detect pedestrians in var-ious scenes and improve pedestrian detection accuracy. Compared with the previous multispectral aggregate channel detection work, the algorithm reduces the log-average miss rate.