To get the IR a sinus sweep signal is sent to the Cab and as said, with neutral amp that doesn't color the signal.
And also as already said, every Cab (IR or Physiyal CAB) will sound different, even with same speaker.
If you search, you can find thousands of statements (also here in Froum) that aprox. 50% of total sound is from the cab.
And its true.
If you use the Fractal without CAB sim and your own physical cab, you are more restricted generarting sounds than using FRFR and IRs (cab sim on) or you have to dial in your amp-tone completely different, adapted to your cab to get the sound you want.
Or you can make different amps sound very similar if choosing a cab fitting for the sound you want to get.
There are many users of modelers that want to use their own physical cab because they like the sound of it. Ok thats a way and everybody have his/her own taste and if you want to have the amp in the room sound that you only can get with a appropriate amount of moving air. But you also can get this with cabs built with widerange or fullrange speakers that are near neutral in sound. Thats what I use.
But for me personally it makes no sense to use a normal guitar cab when using a modeler (lousy excepted with bad CAB sim) becasue of restrictions and the fact that not every amp sounds good with every cab.
I use factory IRs or actually IRs from YA Mesa cab but even the IRs of this single cab with several differnt micros on different positions sounds different.
I choose the IRs depending on amp and dialed in tone and also make own mixes of the bought IRs. For example one IR with SM57 mixed with a MD421 and so on.
Thats also the reason why I do not understand that there are still so much guitarists or studio (even stage) technicians that prefer micing a cab instead of using IRs.
Because using different microphones or placements of the mic will change the sound and so its more efford to get the sound you want and stay consistent. And the chance of having problems is bigger (catching unwanted sound with mic, humming, bad or loose stand, defective cables or jacks, and so on...). Escpecially on stage.
But this is my own, personal opinion. May I'm too progressive and modern for a grandpa