Relaxation time spectra (RTS) derived from time domain induced polarization data (TDIP) are helpful to assess oil reservoir pore structures. However, due to the sensitivity to the signal-to-noise ratio (SNR), the inversion accuracy of the traditional singular value decomposition (SVD) inversion method reduces with a decrease of SNR. In order to enhance the inversion accuracy and improve robustness of the inversion method to the SNR, an improved inversion method, based on damping factor and spectrum component residual correction, is proposed in this study. The numerical inversion results show that the oscillation of the RTS derived from the SVD method increased with a decrease of SNR, which makes it impossible to get accurate inversion components. However, the SNR has little influence on inversion components of the improved method, and the RTS has high inversion accuracy and robustness. Moreover, RTS derived from core sample data is basically in accord with the pore-size distribution curve, and the RTS derived from the actual induced polarization logging data is smooth and continuous, which indicates that the improved method is practicable.