@Mr Pond, after a quick read, I think you may be misunderstanding the concept of Unity Gain. It does NOT mean that the meters for out 2 should be reaching "0 dB" -- it means that the level going out should be the same as the level coming in.
Here's a test to demonstrate things are working.
1. Connect your guitar to your amp using a patch cable. Listen and test.
2. Now place the FM3 inline on that same run: guitar, cable, FM3, cable, amp.
3. Create a preset with IN 1 and OUT 2 blocks.
4. Set the FM3 for unity gain.
5. Listen and test.
The sound and level of steps 1 and 5 should be the same. If you want to meter anything, connect these two setups to an audio interface instead of an amp. It seems that you are focusing instead on the two meters of the "Preset Leveling" screen, which are completely irrelevant in this case when Out 2 is set to Unity gain.