A background removal method based on two-dimensional notch filtering in the frequency domain for polarization interference imaging spectrometers (PIISs) is implemented. According to the relationship between the spatial domain and the frequency domain, the notch filter is designed with several parameters of PIISs, and the interferogram without a background is obtained. Both the simulated and the experimental results demonstrate that the background removal method is feasible and robust with a high processing speed. In addition, this method can reduce the noise level of the reconstructed spectrum, and it is insusceptible to a complicated background, compared with the polynomial fitting and empirical mode decomposition (EMD) methods.