The answer to this is 100% yes. I wish it wasn’t. This was how I planned to work around this, and what I did at first. But once I hooked it up for a level matched, blind A/B test, I had to go the digital route. The difference is not subtle. Once you hear it, you can’t unhear it and it’s hard to listen to stuff I recorded that way before I switched.
The workaround that I’m using is a $1,500 format converter from RME ($800 on eBay if you’re patient). It was tricky to set up at first, but I haven’t thought about or touched it since. Other than the cost, I’m very happy with this solution. I can set my session to any sample rate, and no matter what the converter is doing to the sound, it still sounds a whole other league better than the analog path with the D/A/A/D conversion. If you don’t like this answer and don’t want to buy the external converter, just don’t listen to them side by side. The analog path sounds fine until you hear it next to the digital.
Most of the session files that I play on were not created in my studio, so I don’t get the luxury of deciding what sample rates they come in as. Most of them are 96k, but not all. To me, the question of what sample rate is best doesn’t play into it. It’s not a philosophical, technical or religious discussion. The only decision is, do I want to take the job or don’t I?