I'd connect the left and right audio to the center set of a DPDT (double pole double throw switch) and use the other two sets for PC or TV. Keep all the ground returns connected to reduce interference. Put it into a little box and make the connectors as needed.
http://tinyurl.com/2mgy85
Very quick sketch attached.