Abstract: This study aims to clarify the factors influencing oil recovery of surfactant–polymer (SP) flooding and to establish a quantitative calculation model of oil recovery during different displacement stages from water flooding to SP flooding. The conglomerate reservoir of the Badaowan Formation in the seventh block of the Karamay Oilfield is selected as the research object to reveal the start-up mechanism of residual oil and determine the controlling factors of oil recovery through SP flooding experiments of natural cores and microetching models. The experimental results are used to identify four types of residual oil after water flooding in this conglomerate reservoir with a complex pore structure: oil droplets retained in pore throats by capillary forces, oil cluster trapped at the junction of pores and throats, oil film on the rock surface, isolated oil in dead-ends of flow channel. For the four types of residual oil identified, the SP solution can enhance oil recovery by enlarging the sweep volume and improving the oil displacement efficiency. First, the viscosity-increasing effect of the polymer can effectively reduce the permeability of the displacement liquid phase, change the oil–water mobility ratio, and increase the water absorption. Furthermore, the stronger the shear drag force of the SP solution, the more the crude oil in a porous medium is displaced. Second, the surfactant can change the rock wettability and reduce the absorption capacity of residual oil by lowering interfacial tension. At the same time, the emulsification further increases the viscosity of the SP solution, and the residual oil is recovered effectively under the combined effect of the above two factors. For the four start-up mechanisms of residual oil identified after water flooding, enlarging the sweep volume and improving the oil displacement efficiency are interdependent, but their contribution to enhanced oil recovery are different. The SP flooding system primarily enlarges the sweep volume by increasing viscosity of solution to start two kinds of residual oil such as oil droplet retained in pore throats and isolated oil in dead-ends of flow channel, and primarily improves the oil displacement efficiency by lowing interfacial tension of oil phase to start two kinds of residual oil such as oil cluster trapped at the junction of pores and oil film on the rock surface. On this basis, the experimental results of the oil displacement from seven natural cores show that the pore structure of the reservoir is the main factor influencing water flooding recovery, while the physical properties and original oil saturation have relatively little influence. The main factor influencing SP flooding recovery is the physical and chemical properties of the solution itself, which primarily control the interfacial tension and solution viscosity in the reservoir. The residual oil saturation after water flooding is the material basis of SP flooding, and it is the second-most dominant factor controlling oil recovery. Combined with the analysis results of the influencing factors and reservoir parameters, the water flooding recovery index and SP flooding recovery index are defined to further establish quantitative calculation models of oil recovery under different displacement modes. The average relative errors of the two models are 4.4% and 2.5%, respectively; thus, they can accurately predict the oil recovery of different displacement stages and the ultimate reservoir oil recovery.