Bet you're uploading at full whack when you're done downloading, which is crippling your download speeds later on.
Check out some of the information on torrent use such as capping seeding speeds to avoid this.
I suspect it's your torrenting that's causing the problem, not his relatively low bandwidth XBL usage.
BitComet is poo by the way, use uTorrent -
http://www.utorrent.com/