Are you sure that it's an audio output on the back of the TV and not an audio input? Which end is which? Red/white phonos are usually audio input for component video and 3.5mm is usually the audio input for VGA.
The audio setting in the PS3 is for the PS3... It won't affect what's coming out of the TV, it's for sending audio from the PS3's HDMI, AV, and optical at the same time - nothing to do with the TV, which is what you said you plugged the headphones into?
http://manuals.playstation.net/docum...udiomulti.html