The nonreciprocal circular dichroism and Faraday rotation effect for terahertz (THz) waves in longitudinally magnetized InSb were investigated by theoretical and experimental studies in the THz regime, which indicated its ability for a THz circularly polarized isolator, THz circular polarizer, tunable polarization converter, and polarization modulator by manipulation of different magnetic fields. Furthermore, we demonstrated the InSb plasmonics based on its magneto-optical effects combined with artificial microstructure. We found the magneto-optical enhancement mechanisms in this magneto-plasmonic structure, achieving broadband near-perfect orthogonal linear polarization conversion modulated by the weak magnetic field in an experiment with an extinction ratio of 33 dB. Moreover, the magneto-optical modulation with an amplitude modulation depth of 95.8% can be achieved by this device under a weak magnetic field of 150 mT. InSb and its magneto-plasmonic device have broad potential for a THz isolator, magneto-optical modulator, and polarization convertor in THz application systems.