Long thread, but my two cents.
First off, I'm not dogmatic about any device, so that may not make me popular in some circles. I've bought and sold everything with my own money and really only speak from experience.
I like Helix quite a bit for a number of reasons and I've used it off and on for many years. It really needs aftermarket impulses to sound its best because IMO the stock cabs are terrible especially for higher gain/heavier music. I don't get up into the djent levels of modern gain but I use a heavy rock tone (think a Friedman BE) 90% of the time. If I have an issue with the Helix heavy tones, it's the lack of a musical low end thump/presence. Everything sounds like it's high passed a bit. Again, impulses help quite a bit here to scoop the mids and boost the lows a bit, but it's still a bit flat feeling.
That's where I like the Fractal better, although to be honest especially with Cygnus it's gone the other way. A lot of times there's so much low end I'm using low cuts in the cab, turning down the depth, using the low cut control in the amp, etc. But generally when you dial that down it's a little more satisfying to play. I will also say it may take me longer to dial in a tone on the Fractal but generally I will like that tone a little better than Helix. I also find that, in general, the Fractal effects (outside of drive pedals which I rarely use) sound better and richer to me than Helix.
Now, all that said, there's the very real possibility that your issue with the Helix sound isn't completely due to the Helix but due to the monitoring setup. Which means you could spend $1200 or whatever the cost is for a Fractal FM3 and still not be happy with the sound. I'm not discouraging you from getting the Fractal (because it's awesome), but I will say you may need to spend some time and money on getting the sounds out of the modeler. And I would do that before spending a lot of money on presets and impulses and what not.
EDIT: I see you are looking at the Yamaha HS5. I haven't personally owned them but I've tested those against others, and you'll definitely need some more low end with those monitors as they are skewed towards being brighter. Now I actually think that's a good thing, because most of the cheaper monitors I've heard were too hi-fi sounding and having purchased a few of them, they sounded like crap playing guitar through them. There's not enough midrange and detail. I really like the Yamaha HS7 which, if I had enough space to run full size monitors, I'd probably get those. They seemed to have adequate low end but they are really big for a regular desk. I've given up on studio monitors on my desk though and would likely either run a regular guitar cab or get some floor monitors (which I liked playing through a lot more in the past).
One other thing you'll want to know coming from the Helix, there's no way to run either dual amps or gapless amp switching within a preset. You can set up four different amp channels within a preset and switch between them, but there will be a very slight audio gap when switching. This is significantly better than it used to be on older devices and it's not that much longer than switching channels on some tube amps, but it's a thing you want to be aware of. That's one of the draws of the new FM9 I believe (or the Axe 3). Otherwise I find there's plenty of power for an amp, cab, reverb, delay, and a few other effects.
