It's likely just a result of the summing of the left and right channels making the output of the block higher when you center the pan controls. Someone will be along shortly to correct me, but I believe the difference between hard panned and centered here would be +6 dB when the controls are centered.
So, not "more effected", just louder at the block output. You could leave the pans hard left and right and just put the output at 6 dB and it should be equivalent.
Though, I must say, on my mono setup here, I heard no volume or tonal differences panning or centering these controls. Will check with a meter in a bit though.