Well, it depends what I'm doing, but I use my AF3 as a fully featured rig for my church gig- everything in the box. But at home, 90% of the time, I use it for effects only running into real amps. It's currently in the music room with Output 1 connected to a Swart SST-30, and Output 4 connected to a Swart Stereo Master 20. The beauty of the whole thing is that you can then set up whole rigs with just a preset- I have a preset that's effects only for the SST in mono, another preset that's for the Stereo Master 20 which is in stereo, and I have a third preset that uses both amps at the same time in a w/d/w rig, and it honestly couldn't be easier. I have a few decent pedals sitting around, but to be honest, the effects in the Axe sound just as good, and often better- much less noise, no weird impedance issues, etc.
I've also run it using the four cable method with my Carol Ann Tucana 3- works great. I just wish there was a way to control the amp's switching via midi (there was a box made for this, but Alan Philips shut down the business and hasn't made them any longer). No complaints on the sound quality though- just make sure you're using hum buster cables.
So yea, it works great as an effects unit only. I don't think there are really any downsides if you prefer using real amps.
Edit: I should say 90% of the time at home when I'm playing loud- plenty of times I end up practicing silently on the Axe only using headphones. For the record, the amps in the Axe 100% hold up to the real amps.