The issue is that you shouldn't use a voice if the pitch shift is zero. I could remove the latency in that case but then you wouldn't realize that you have a voice set to zero shift.
Is the pitch-shifted tone different at zero lag vs some lag? If so, having both options could warrant including the capability if you're specifically looking for a pitch-shifted, zero lag voice in parallel with your dry sound.
Perhaps there's a way to get the best of both worlds.
Allow Zero Lag settings to be dialed in, but default each voice to include some lag.
This way you won't accidentally setup your tone with a redundant, zero-shifted voice.