The shaky latency is the 4 other users seeding P2P @ 15Mbps+ all day and my 40GB of downloads...
Although the core does show some latency increase at peak times, this is really just +/- 1ms, and apparently only at weekends.
I might have a go next weekend to see if it's upstream or downstream, if the former then it's quite possible that P2P is causing increased load as you say, if the latter then it would just be peak-time load.
(Also even at the increased levels my average latency still remains 50% lower than on VM during peak hours, and Youtube streaming at 1080p and 4K work just fine

)