Quantifying NV-center Spectral Diffusion by Symmetry