So many great responses! I am so grateful for all of you guys for willing to help!
I think now it starts to clear up a little bit in terms of my understanding what happens. What has helped recently was to use LB-2 impedance curve - it clears up at lot.
I will try to respond to everyone, hope it doesn't look ugly to have a long reply.
If TAE makes the amp sound this much better (to me at least), and this were to be due to impedance curve, I'd like to have that in fm3 for sure.
Yes, it does sound that much better to my ear. It's called unrealistic in many of the replies, but if that's unrealistic, I will choose that. To get this exact sound in FM3, I did this EQ in the amp block (that's for Studio 20 model) - and it sounds almost indistinguishable from the real thing with a fairly bright IR (LT TV Mix 2 that Leon Todd posted on Axe-Change is an example, but there are many others built in like it):
Know that you told us that you're using a Tube Amp Expander, could you also tell us the Resonance Z and Presence Z settings that you're using on it? Do you use any other EQs, switches, on the TAE?
Yeah, sure. I use recommended settings for Brit Stack (4x12 Celestion Greenback). Everything else on the unit is deliberately turned off, IR processing, EQ, effects, switches, everything off.
On lower Presence-Z settings it sounds very dark as well, but then it's hard to hear the sound and it's not exactly usable IMO.
What do their Z controls do? Do they control analog circuit elements that can change the effective impedance curve or are those just "EQ" shortcuts?
Yes, they control analog citcuitry - you can hear relays switching every time you change settings on the front panel.
I recommend also posting the settings you use on the Boss TAE. The reactive load settings could cause things like overly hyped high end and/or bass cut which you find preferable. You could try playing with the different speaker impedance curve settings in the FM3 model and see if that gets you closer.
This is actually what got me much closer.
So summing up, seems like what I was experiencing was a combination of two things:
1) Default impedance curve in the model, whereas LB-2 sounds closer to my preference
2) Boss TAE is bright - which I like, but it's something that needs to be taken into account
All that in conjunction gives me an opportunity to get 1987X tone that is very close to
this video for example without any EQ, while I still need to EQ Studio 20 model somewhat, but not as dramatic as before.
Because it was two factors at once, I was getting such massive difference. Now need to spend more time with this setup to really understand it better.
BTW, Studio 20 model bass response is noticeably less than it is on the real thing, again - not critical, but the difference is there. I can hear it
in this video, and it's the same experience I have right now.