hey man, haven`t read the other replies....
However, let me tell you that I am 100% sure that your approach is wrong and that`s why you are frustrated. Been there myself, just trust me
When it comes to dialing in tones in the Axe Fx and you want to have similar sound like [insert whatever you want here] then please forget that NOW.
A tone in a mix is being carried so damn hard by the context/drums/bass and even if there is a solo`d tone somewhere you will NEVER achieve such a tone with your Axe Fx only (pretty sure about that).
There is so much going on (compression, limiting, multiband compression, EQ, harmonic excitement, etc.). So unless you have good knowledge about mixing you will struggle to get a REALLY good tone.
That being said: it is not your guitar. Or let`s say: the components are alright but it is damn hard to dial in guitar tones, especially high gain tones since it`s basically just broadband noise. Plus, having 8 string means having a freaking big frequency range especially in the lower register so you WILL end up with something like a compromise tone: either it `s going to sound muddy on the low stuff but full sounding on the higher notes (E to e) or tight on the low end but thin on higher notes.
About the strings: .102 is definitely fucking thick but you will FOR SURE hear if it`s too thick for your guitar and/or technique.
So you are battling physics and high standards created by pro mixing/recording engineers.
In the end, you can make any guitar on this planet sound more or less good, that`s for sure.
Now let me give you concrete help and if that doesn`t work out, you will have to dig a little bit deeper into mixing I guess.
Also, don`t take the advice to low pass your tone to 5khz too serious as I find this utter shit. It works with some tones but really, you just cut out all the life in the high end.
Now fire up a drive block, use the TubeScreamer MOD because this one`s tighter and go for an Amp that sounds tight. Some FAS modern II or III sounds pretty tight I think but also very dead and cold. Go with a peavey Amp or the 5153, they are fine. Next is the crucial part: go find a cab that isn`t super muddy or have spikes in the low end (sounding as if the chugs are behind a wall). Choose a tight that you like the voicing as well and then you have your basic tone for recording! It`s that simple. Don`t use microphones on the cabs unless you can`t find one, then you can try to experiment with mics in the cab block. But really, it`s nothing more than just EQ on your chosen cab so that`s definitely nothing you couldn`t do yourself with an EQ and a DAW.
After that, you analyze your mix/context and tweak bass, mids, treble and presence from there and the rest is mixing, really. Of course you can apply a high and low pass filter but only if your tone has massive sub frequencies or super fizzy high end but I think there must be some high end in your tone because they are important to get an in your face tone that sounds open, clear and tight, especially with 8 strings.
EDIT: listened to your soundclip, doesn`t sound bad. I could make this sound KILLER in a mix, trust me.
I personally would try to dial out the djent character (that 2khz kchhhhhh wah wah noise which is ugly but the djenttards like it, I know
). Take down a little bit 1.8khz with a wider Q, if you don`t like it, reduce 700-900 hz and when done carefully and nicely you should get a better and more mix friendly tone IMO.
EDIT 2: to me, it sounds as if there is some over saturation going on (audible mostly when you palm mute and then let it sustain). Sounds like either too much input gain, too much gain on the drive, wrong drive pedal (sounds like the octave dist haha
).
EDIT 3: should have read your last post but aww well, I am still waiting for someone to reply to my thread so it`s okay to waste time actually doing something and maybe this post will help some noobs out there.