I've seen this behaviour before, it happens on some Draytek routers as well due to the ICMP requests being put on a lower priority. That's why the connection is unaffected.
Here's an example of what I mean (from my work connection, which is not VM):
Looks bad, but the connection is actually fine.