Thanks satcomer, I'll try that when I get home.
I phoned Apple last night... We have found:
1. No speakers plugged in: Mavericks allows internal speaker volume operation as normal. Sound preferences knows to output via internal speakers.
2. My Headphones plugged in (3.5mm jack): Mavericks allows volume operation as normal. Sound preferences knows to output via Audio Out Port.
3. My Amplifier plugged in (3.5mm jack): Mavericks does NOT allow volume operation, Speaker icon greyed out again. Sound preferences thinks I have an optical cable plugged in.
I can only deduce that the iMac measures the impedance of the connection. The headphones impedance is identified correctly as Headphones, but the Amp cable impedance is outside the "range" the iMac recognises, and is identified incorrectly as an optical connection.
I am going to measure the resistance of the 2x 3.5mm plugs, and try to modify the amp cable to match the headphone cable...