Chip away at the caulk around the outside edges of the current recessed toilet paper holder with a putty knife.
Pull the caulk strips out with a needle-nose pliers.
Place duct tape around the toilet paper holder to prevent the tiles from chipping when you pull the holder out. The duct tape works better than other tapes because it doesn't leave residue.
Place a chisel behind the edge of the toilet paper holder and hit the end of the chisel with a mallet. Twist the chisel to start pulling the holder out. Move the chisel along all edges of the holder to pull it out.
Hold the edge of the recessed toilet paper holder with needle-nosed pliers and pull until the holder comes out.
Caulk the edge of the hole carefully by applying a thin bead of caulk that will not seep out the edge of the toilet paper holder when it is added.
Place the new recessed toilet paper holder into the hole. Make sure the holder is not upside down.
Tap the edges of the holder with the mallet until the holder is flat against the wall. Allow the caulk to set for about three hours.
Remove the tape from the tile by carefully pulling from one corner down diagonally.
Clean the tile wall with a soapy sponge until any adhesive on the tiles is removed.