For audio stuff you are best taking it away from you PC internals. Most audio cards and onboard sound systems are not for those actually wanting to use them for anything worthwhile.
My old ASUS M/B with onboard sound had a noise floor 40dB above the theoretical value. A cheap and cheerful USB one (circa £30) I borrowed from work was about 10dB above.
Moving cables around inside the PC will have a limited effect as much of the noise is induced in the ground/shielded side anyway. Just because two points are electrically ground does not mean they are at the frequencies flying around inside a PC. At 3.4GHz a tiny unshielded wire is a very effective aerial.
Yamaha do some very good pro equipment for PCs. A bit expensive though so it depends how serious you want to be.
As for it going away when the HD is being accessed...most switching power supplies require a minimum load for stability. Few peripherials in a PC use 5V apart from HDs and CDs/DVDs. If for effiency the design of the power supply is a "burst mode" design (I have no idea one way or t'other), they are notorious for radiating from DC to XRAYs.
As previously mentioned, a balanced input with twisted pair wires will gain you alot.