Use the OUT3 block meter to gauge your level going into the D/A converter.
Per the manual, max output for Output 3 is +20dBu. I would guess that should be roughly at 0dBFS, the clipping point for the D/A converter. When the output meter for OUT3 is at the 0 dB line, it's at about -12 dBFS peak at the D/A converter. That means 0 dB on that meter should give you roughly 12 dB below the max output or roughly +8 dBu analog output level when the front panel Out 3 knob is all the way up. If you want -10 dBV (about -7.78 dBu) from the output, you'll want the OUT3 block meter to read about -15.78 dB. Since the D/A level is then fairly low, you can improve your SNR by turning up the Boost/Pad to compensate. Setting it to 18 dB will then give you -10 dBV at the output when the OUT3 meter is now reading around 2.2 dB.
Judging by the amount the Output meter changes, it appears that the 18 dB Boost/Pad setting only boosts the D/A level 14 dB instead of the full 18, so aim for around -1.78 dB on the OUT3 block meter and that should give you pretty close to -10 dBV line level at the output when the Out 3 knob is up full.
To verify, I fed a synth block outputting a 60 Hz sine wave into the OUT3 block, set Boost/Pad for Output 3 to 18 dB, cranked the front panel Out 3 knob and measured the output voltage with my cheapo multimeter. I adjusted the synth block output level until I got about 0.316 volts (-10 dBV) at the output. I then checked the OUT3 block meter and it read -0.8 dB. That's within 1 dB of my estimate above. Not bad for some quick and dirty figures. My cheap meter could easily be off that much. I'd need a proper oscilloscope to measure it right.