I can only relay my own experience with the two. I've tried them through all kinds of setups at various volume levels:
- Helix preamps into Bogner and Victory tube amps' fx returns.
- Helix and FM3 amp models into a Fryette Power Station 100 and real guitar cabs.
- Helix and FM3 amp models into the BluGuitar Amp 1 fx return and real guitar cabs.
- Helix and FM3 amp models into computer and VST plugins (cab sims etc).
- Helix and FM3 into Genelec M040 studio monitors.
- Helix and FM3 into Beyerdynamic DT990 Pro 250 ohm and Sennheiser HD6XX 300 ohm headphones.
- Helix and FM3 into PA/FRFR speakers.
All of these setups will sound different and respond in a different way no matter which modeler is in use. At low volume I like the studio monitors the best and at higher volume the PS-100 and guitar cabs is probably the most satisfying. Headphones need both frequency correction and room reverb to sound natural and are probably the worst option but sometimes a necessary one.
Many pros who use Fractal use it for effects. Fractal has better modulation and reverb effects than the Helix. No contest there.
I don't agree about "fizzy and artificial". Or boomy for that matter. Many of these aspects will depend on the room and your output system. For example there's a big 130 Hz boost in the room I play in at home so I use global EQ to dial that out or else some notes will sound boomy. Likewise placement of speakers and if they emphasize low end will matter.
I don't do anything particularly special with my presets. I mostly use ML Sound Lab multi-mic IR mixes exported from the MIKKO cab sim plugin if I am using IRs but you might use something from say York Audio just as well. I add some low cut usually around 80-100 Hz, high cut to taste. The cab or IR block low cut is very helpful for avoiding any boominess. I don't like many of the default settings on Helix amp models so I dial my own. For high gain tones I often reduce sag a bit from the default setting to tighten up the feel. I might use a graphic or parametric EQ either in front or after the amp to tweak things a bit more.
My experience is that Fractal modeling is not in some sort of different echelon from the competition. There's plenty of videos you can find on YT where people have dialed various modelers to sound very very close regardless of what genre of music is being played. That aligns with my own experience. Using the same IRs will give you very similar results. If we were to compare with their respective cab sims then I would prefer Fractal because its vast IR library is just plain better than Helix's stock cab sims.
When comparing it's also important to pick a reference tone. It can be a real amp or whichever modeler's dialed in tone you prefer. Then you figure out how to make them sound and feel the same. Putting in the same values for knobs does not work. It's also useful to use a decibel meter to make sure what you are hearing is coming out at the same volume because usually we hear louder as better. That alone probably accounts for a lot of the "X sounds so much better than Y" hyperbole we read on the internet.
Note that I am not trying to put down Fractal here. It sounds stellar and has a huge feature set. But I feel that the Helix amp modeling is not the aspect that is holding your tones back.
