An adaptive digital backward propagation (ADBP) algorithm is proposed and experimentally demonstrated based on the variance of the intensity noise. The proposed algorithm can self-determine the unknown nonlinear coefficient γ and the nonlinear compensation parameter ξ. Compared to the scheme based on the variance of phase noise, the proposed algorithm can avoid the repeated frequency offset compensation and carrier phase recovery. The simulation results show that the system’s performance compensated by the proposed method is comparable to conventional ADBP schemes. The performance of the proposed algorithm is simulated in 40/112 Gb/s polarization-division multiplexing (PDM)-quadrature phase-shift keying (QPSK) and 224 Gb/s PDM-16-quadrature amplitude modulation (QAM) systems and further experimentally verified in a 40 Gb/s PDM-QPSK coherent optical communication system over a 720 km single-mode fiber.