A method for compensating the measuring error of the grating displacement measurement system with absolute zero mark is presented. It divides the full scale range into piece-wise subsections and compares the maximum variation of the measuring errors of two adjacent subsections with the threshold. Whether the specified subsection is divided into smaller subsections is determined by the comparison result. After different compensation parameters and weighted average values of the random errors are obtained, the error compensation algorithm is implemented in the left and right subsections, and the whole measuring error of the grating displacement measurement system is reduced by about 73%. Experimental results show that the method may not only effectively compensate the spike error but also greatly improve the precision of the measuring system.