Not sure this is the cause of your problem, but I like to do a quick mix at the end of a home record session night, complete with plug-ins on the master buss. And invariably, I forget they are there, the next time I add more tracks to the song. In my case, the culprit is IZOTOPE mastering plugins. And the latency remains even if I bypass them. That doesn't solve the issue. I have to remove the plugins from the channel, to fix the latency problem if I'm going to record additional guitar tracks.
Also, to bridge upon what the others said, I output my Axe into a small mixer, and the output from my iMac into the mixer, and then listen to the mixer's output with headphones. I do not use input monitoring.
And maybe I'm just confused, but I appears with the 8x8 capability in the Axe III, I can send the song tracks into some of the inputs, and monitor everything through the headphone jack of the Axe III --- once again, not using Logic's input monitoring.