The most primitive models can be conceptually understood as a particle responding to unbalanced local charge that arises near a crystal impurity; similar to an electron encountering an electric field.
[1] This effect is the mechanism by which doping decreases mobility.
In the current quantum mechanical picture of conductivity the ease with which electrons traverse a crystal lattice is dependent on the near perfectly regular spacing of ions in that lattice.
Only when a lattice contains perfectly regular spacing can the ion-lattice interaction (scattering) lead to almost transparent behavior of the lattice.
Impurity atoms in a crystal have an effect similar to thermal vibrations where conductivity has a direct relationship with temperature.