Abstract: High-precision and real-time diagnosis of sucker rod pumping system (SRPS) is important for quickly mastering oil well operations. Deep learning-based method for classifying the dynamometer card (DC) of oil wells is an efficient diagnosis method. However, the input of the DC as a two-dimensional image into the deep learning framework suffers from low feature utilization and high computational effort. Additionally, different SRPSs in an oil field have various system parameters, and the same SRPS generates different DCs at different moments. Thus, there is heterogeneity in field data, which can dramatically impair the diagnostic accuracy. To solve the above problems, a working condition recognition method based on 4-segment time-frequency signature matrix (4S-TFSM) and deep learning is presented in this paper. First, the 4-segment time-frequency signature (4S-TFS) method that can reduce the computing power requirements is proposed for feature extraction of DC data. Subsequently, the 4S-TFSM is constructed by relative normalization and matrix calculation to synthesize the features of multiple data and solve the problem of data heterogeneity. Finally, a convolutional neural network (CNN), one of the deep learning frameworks, is used to determine the functioning conditions based on the 4S-TFSM. Experiments on field data verify that the proposed diagnostic method based on 4S-TFSM and CNN (4S-TFSM-CNN) can significantly improve the accuracy of working condition recognition with lower computational cost. To the best of our knowledge, this is the first work to discuss the effect of data heterogeneity on the working condition recognition performance of SRPS.