ing at the influence of the complicated deep-space communication environment on the anti-noise performance of X-ray communication system, a bit error rate optimization model is proposed based on quadrature amplitude modulation (QAM). The mathematical expression of the primary noise source is derived through analyzing the noise source in actual communication scenario combined with an existing X-ray communication link model. On the basis of Poison distribution model, the calculation model of the communication error rate in intensity modulation direct detection (IM/DD) system is established under the QAM mode, and its effectiveness is verified by simulation. The simulation result shows that the bit error rate of QAM system can be reduced to 10 ^-6 orders of magnitude.When the same amount of photons are received, the proposed method launched less photons compared with the existing binary on-off keying (OOK) and pulse position modulation (PPM) methods. Furthermore, this proposed method is suitable for the modulation of the space X-ray communication system when it is deployed in the environment with an unstable noise intensity.