For improving the hole-enlarging capability, roundness and rock-breaking efficiency of the nozzle in radial jet drilling, a new structure of self-rotating nozzle was put forward. The flow structure and rock-breaking features of the self-rotating nozzle were investigated with sliding mesh model and labortary tests and also compared with the straight and the swirling integrated nozzle and multi-orifice nozzle which have been applied in radial jet drilling. The results show that the self-rotating jet is energy concentrated, has longer effective distance, better hole-enlarging capability and roundness and impacts larger circular area at the bottom of the drilling hole, compared with the other two nozzles. Forward jet flow generated from the nozzle is peak shaped, and the jet velocity attenuates slowly at the outer edge. Due to periodic rotary percussion, the pressure fluctuates periodically on rock surface, improving shear and tensile failures on the rock matrix and thereby enhancing rock-breaking efficiency. The numerical simulation results of the flow structure of the nozzle are consistent with the experiments. This study provides an innovative approach for radial jet drilling technology in the petroleum industry.