Recently, nanoparticles have proven to enhance oil recovery on the core-flood scale in challenging high-pressure high-temperature reservoirs. Nanomaterials generally appear to improve oil production through wettability alteration and reduction in interfacial tension between oil and water phases. Besides, they are environmentally friendly and cost-effective enhanced oil recovery techniques. Studying the rheological properties of nanoparticles is critical for field applications. The instability of nanoparticle dispersion due to aggregation is considered as an unfavorable phenomenon in nanofluid flooding while conducting an EOR process. In this study, wettability behavior and rheological properties of surface-treated silica nanoparticles using internal olefins sulfonates (IOS20–24 and IOS19–23), anionic surfactants were investigated. Surface modification effect on the stability of the colloidal solution in porous media and oil recovery was inspected. The rheology of pure and surface-treated silica nanoparticles was investigated using a HPHT rheometer. Morphology and particle size distributions of pure and coated silica nanoparticles were studied using a field emission scanning electron microscope. A series of core-flood runs was conducted to evaluate the oil recovery factor. The coated silica nanoparticles were found to alter rheological properties and exhibited a shear-thinning behavior as the stability of the coated silica nanoparticles could be improved considerably. At low shear rates, the viscosity slightly increases, and the opposite happens at higher shear rates. Furthermore, the surface-modified silica nanoparticles were found to alter the wettability of the aqueous phase into strongly water-wet by changing the contact angle from 80° to 3° measured against glass slides representing sandstone rocks. Oil–water IFT results showed that the surface treatment by surfactant lowered the oil–water IFT by 30%. Also, the viscosity of brine increased from 0.001 to 0.008 Pa s by introducing SiO2 nanoparticles to the aqueous phase for better displacement efficiency during chemical-assisted EOR. The core-flood experiments revealed that the ultimate oil recovery is increased by approximately 13% with a surfactant-coated silica nanofluid flood after the conventional waterflooding that proves the potential of smart nanofluids for enhancing oil recovery. The experimental results imply that the use of surfactant-coated nanoparticles in tertiary oil recovery could facilitate the displacement efficiency, alter the wettability toward more water-wet and avoid viscous fingering for stable flood front and additional oil recovery.