A new physical explanation of the fringes in the high-order harmonic spectra generated by ellipticity-modulated infrared pulses is proposed, which is different from those in previous works. In this work we find that the fringes are due to the spectral interference effect induced by the two outgoing pulses with a delay, which are generated by the wave plates for ellipticity control. The visibility of interference fringes observed experimentally increases with the increase of the asymmetry of the two pulses, with the decrease of the delay and with the increase of peak intensity of the driving laser pulse. The experimental results are in good agreement with the calculated ones.