[
This was your graph. The bit that matters is the blue - the average latency. By comparison with some/many of the horrors we see on the forums, this is a reasonable graph. Is it still like that?
Have you tried working in modem mode? If it's not external (congestion) then it could be the SH router tying itself in knots as has been reported (with more radical symptoms) in THenry's Crawley thread and in Ravenstar's thread in the VM forums.