An in-line, highly sensitive refractive index (RI) sensor based on a tapered multicore fiber (MCF) structure sandwiched between two single-mode fibers is proposed and demonstrated. The fiber tapering technique was employed to fabricate in-line interferometers based on the multicore fiber. The waist diameter is one of the dominant factors of the intercore coupling. The tapered MCF interferometer is highly sensitive to the surrounding refractive index with a maximum sensitivity of 9194.6 nm/RIU in the RI range of 1.4264 to 1.4278 when the waist diameter is 9 µm. The enhancement of the evanescent field by graphene coating is proved to be able to improve the RI sensitivity further. A graphene-coated MCF interferometer with waist diameter of 9 µm offers the maximum sensitivity of 12617.6 nm/RIU in the RI range of 1.4144 to 1.4159. The experimental data have good agreement with the simulated results.