We analyze the dynamics of the helical edge modes of a quantum spin Hall state in the presence of a spatially non-uniform Rashba spin-orbit (SO) interaction. A randomly fluctuating Rashba SO coupling is found to open a scattering channel which causes localization of the edge modes for a weakly screened electron-electron (e-e) interaction. A periodic modulation of the SO coupling, with a wave number commensurate with the Fermi momentum, makes the edge insulating already at intermediate strengths of the e-e interaction. We discuss implications for experiments on edge state transport in a HgTe quantum well.