350M Latency / Packet Loss to Cloudflare - Page 3

fizzyade · 31-12-2020, 02:17

Whatever it is, it's either co-incidence that it seems to stop at around 00:30 (people going to sleep) or something else.

spiderplant · 31-12-2020, 09:27

Quote:

Originally Posted by fizzyade

Whatever it is, it's either co-incidence that it seems to stop at around 00:30 (people going to sleep) or something else.

It may well be traffic-related. Do you know roughly what time it starts happening? It seems to be OK at the moment.

fizzyade · 31-12-2020, 10:09

Quote:

Originally Posted by spiderplant

It may well be traffic-related. Do you know roughly what time it starts happening? It seems to be OK at the moment.

I was running the test all night, the ICMP dropping seemed to start occurring at about 4 in the morning, and got drastically worse as time went on:

However, today (as you have said) it doesn't seem to be preventing packets getting to the final destination, the hop itself seems to be dropping ICMP packets with a TTL which would cause a time exceeded reply from that hop.

Synthetic · 31-12-2020, 10:22

Quote:

Originally Posted by spiderplant

I've reported this. Ticket ref F008701533

Thank you

fizzyade - I was thinking the same, it'd probably be less stressful to just deal with the packet loss than try and explain it to the VM call centre!

spiderplant · 31-12-2020, 15:01

This should be fixed now. VM have two links to Cloudflare, but one of them was down, so all traffic was being routed down the other, which became overloaded at peak times. The second link is now restored. Please can you check & confirm?

fizzyade · 31-12-2020, 16:23

Quote:

Originally Posted by spiderplant

This should be fixed now. VM have two links to Cloudflare, but one of them was down, so all traffic was being routed down the other, which became overloaded at peak times. The second link is now restored. Please can you check & confirm?

Still seeing high packet loss on that hop, but as per the post this morning the cloudflare hop is working fine, so I assume Hop 7 is just dropping ICMP packets.

spiderplant · 31-12-2020, 16:29

Quote:

Originally Posted by fizzyade

Still seeing high packet loss on that hop, but as per the post this morning the cloudflare hop is working fine, so I assume Hop 7 is just dropping ICMP packets.

Yes, I see that too, but can't really report it as a fault

fizzyade · 31-12-2020, 22:42

Quote:

Originally Posted by spiderplant

Yes, I see that too, but can't really report it as a fault

Out of interest, is there a reason that it does it on Hop 2? I assume it's some sort of rate limiting or DDOS mitigation kicking in.

If you ping the second hop say every 5 seconds, it will happily reply. But start pinging at 1s (or faster intervals) and after a few replies it then stops.

If you run 2 separate pings say at 2s intervals, you'll see that one of the pings losing replies, then all of a sudden it'll switch to the other one losing replies, one of them though will usually reply, you can see it flip flop.

spiderplant · 31-12-2020, 23:08

Sorry, I don't know why that happens

Synthetic · 01-01-2021, 10:36

All seemed good last night, thanks again spiderplant for raising, wondering if virgin were already aware, seems like the sort of thing they should be monitoring..

spiderplant · 01-01-2021, 10:57

Quote:

Originally Posted by Synthetic

All seemed good last night, thanks again spiderplant for raising, wondering if virgin were already aware, seems like the sort of thing they should be monitoring..

Excellent.

I think they were aware that one link was down but didn't realise it was customer-affecting.

Synthetic · 01-01-2021, 13:04

Quote:

Originally Posted by spiderplant

Excellent.

I think they were aware that one link was down but didn't realise it was customer-affecting.

Ah I see, so if this happens again we might need to (kindly) ask you to give them a prod again?

fizzyade · 02-01-2021, 01:32

Quote:

Originally Posted by spiderplant

I think they were aware that one link was down but didn't realise it was customer-affecting.

That's slightly concerning! Link down, "no bother lads, we've got another one".

The fact that it took us, end-users, using monitoring tools to identify and confirm that there was an issue is not great, as I said earlier if you weren't around then I dread to think how long this could have rolled on for.

This is especially true when the frontline support will invariably run some diagnostics, possibly put our routers back into routed mode before declaring that an engineer needs to be sent out.

As I've said multiple times, business users desperately need an online support forum where there are actually people who are technically switched on so that when issues occur we can get them fixed. I t just seems very strange that residential users have a support forum (with techs who actually know that many issues don't involve resetting the hub or sending out an engineer).

Anybody who's lost the will to live speaking to technical support will probably feel the same.

Synthetic · 02-01-2021, 09:36

Quote:

Originally Posted by fizzyade

That's slightly concerning! Link down, "no bother lads, we've got another one".

The fact that it took us, end-users, using monitoring tools to identify and confirm that there was an issue is not great, as I said earlier if you weren't around then I dread to think how long this could have rolled on for.

This is especially true when the frontline support will invariably run some diagnostics, possibly put our routers back into routed mode before declaring that an engineer needs to be sent out.

As I've said multiple times, business users desperately need an online support forum where there are actually people who are technically switched on so that when issues occur we can get them fixed. I t just seems very strange that residential users have a support forum (with techs who actually know that many issues don't involve resetting the hub or sending out an engineer).

Anybody who's lost the will to live speaking to technical support will probably feel the same.

My thoughts exactly, how long would this have gone on for if we didn't have spiderplant to help us!

spiderplant · 02-01-2021, 11:11

Quote:

Originally Posted by fizzyade

That's slightly concerning! Link down, "no bother lads, we've got another one".

It's because of the Christmas change freeze. Faults that aren't believed to be customer-affecting don't get fixed until after freeze ends. As soon as I pointed out this one actually was customer-affecting, it was fixed straight away.