In this paper we investigate the reinforcement mechanism of high viscosity rubber/SBS modified asphalt mortar mixed with fiber (mineral, lignin or carbon fiber) and deoiled asphalt (DOA). The softening point, penetration and viscosity tests were conducted to characterize the engineering properties of asphalt-fiber mortar. The microstructure of fiber was observed using scanning electron microscopy (SEM), and the results indicated that fiber can effectively improve the toughness of asphalt matrix through forming a spatial network structure, and then adhesion and stabilization of asphalt binder. The cone penetration test was designed to study the rheological property of fiber modified asphalt. The results indicated that the reinforcement effect increased with fibers and DOA fraction increasing to a certain threshold, and the optimal fiber content was dependent on the fiber type and its length. Fiber content and filler-asphalt ratio had significant effects on the softening point, penetration, viscosity and cone penetration of asphalt mortar.