True, no doubt. That said, the guy playing the solo knows if it sounds wrong, or if he botches a note. And there may be a few others.
Let's stipulate that some of the nuances we obsess over as musicians are overlooked and unfelt by 90% of the audience. Let's further stipulate that of the remaining 10%, many of them can only feel when something isn't quite as good as they'd have hoped, even if they can't say why.
Still, we play to bring joy and excitement and pathos to...whom? Not 90% of the audience, but, if we can, to 100% of the audience, right? And that includes even the golden-eared musicians in the crowd. And when we make those few golden-eared musicians nod their heads and say, "yeah, that's nice," we've just done something of real value. "The praise of the praiseworthy is worthy praise, indeed."
So our efforts aren't wasted if the two or three musicians in the audience who can pick up these nuances do recognize them and are happier for it.
And there's no law against us having a better time by loving the "just-rightness" of our own sound, either.