Setting XVTR power?

The max power setting in the XVTR dialog box is the output from the radio's transverter port not the transverter power output. The radio has no knowledge of what transverter you have connected.

You have to know the sensitivity of the transverter as well. The manufacturer will publish the sensitivity (or gain) of the transverter. Also the manufacturer will state the maximum input drive level and the corresponding output level.

For example will -10dBm from the transverter port drive the transverter to 10 watts?

So, you could, for example, set the XVTR port max power level to correspond to the maximum transverter input level and control the power level from the slider.

My transverter operating manual stated that the input power should normally be set to 1 mw/0 dBm (with a maximum of 14 mw) for 12 Watts out. I used a calibrated Wattmeter into a VHF dummy load to check the levels, and I found that setting the transverter Max Power on the Flex 6400M XVTR setting to -10 dBm drove the transverter output to 12 watts. I think you will have to experiment a little with the drive levels to get the output you want.