350M : Latency / Packet Loss to Cloudflare

[Synthetic]

Originally Posted by spiderplant

It's because of the Christmas change freeze. Faults that aren't believed to be customer-affecting don't get fixed until after freeze ends. As soon as I pointed out this one actually was customer-affecting, it was fixed straight away.

Makes sense, and to be fair i was impressed how quick it was sorted after you raised it

[fizzyade]

I think "Sir Spiderplant" has a certain ring to it!

Seriously though, I'm very grateful that you hang out here and that you're so proactive in getting things done when us more technical bods think something isn't right.

The funny thing is, that when we spot a fault of this kind (that others probably wouldn't even think is a fault or notice) we are much less likely to even consider talking to customer support because we know beforehand how the conversation is going to go.

The flakyness of the connection is partly what inspired me to write pingnoo, so that I would have the ability to see what was going on with the connection over a period of time, the other part was that pingplotter is crippled and I didn't feel like buying it....so I did what any programmer does, rolls their own.

[Skie]

It's a common failure scenario. One link goes down, other links take the strain and everything looks okay and you can sort the dead link in due course. But because the links and kit on them were specced years ago and are now carrying 25% more traffic individually, they aren't going to handle 2x the present load.

The signs of an overloaded route are also incredibly hard to create alerting for, as different bits of kit will do different things when faced with too much to do. Way too easy to create false alarms and those are the bane of monitoring systems, so unless you know what the failure looks like exactly and it has been accounted for in your alerting, you usually wont know it's happening without someone experiencing it directly.

And don't get me started on asynchronous routing...

[fizzyade]

Was just working on some stuff and noticed that routing to Cloudflare has changed now, it seems to hop across to Europe for a laugh now.

[Synthetic]

Interesting, those twelve99 hops look new

1 10.53.34.245 (10.53.34.245) 11.661 ms 9.542 ms 10.772 ms
2 gate-core-2b-xe-705-0.network.virginmedia.net (62.252.43.45) 11.037 ms 11.897 ms 13.197 ms
3 * * *
4 86.85-254-62.static.virginmediabusiness.co.uk (62.254.85.86) 27.816 ms 28.488 ms 28.142 ms
5 ldn-b1-link.telia.net (213.248.84.25) 26.882 ms 25.126 ms 33.002 ms
6 ldn-bb1-link.ip.twelve99.net (62.115.120.74) 36.019 ms 39.188 ms
ldn-bb4-link.ip.twelve99.net (62.115.122.180) 29.973 ms
7 dln-b2-link.ip.twelve99.net (62.115.120.105) 37.692 ms
dln-b2-link.ip.twelve99.net (62.115.120.101) 38.170 ms 36.564 ms
8 cloudflare-ic-310815-dln-b2.c.telia.net (62.115.55.14) 40.052 ms 42.718 ms 38.989 ms
9 one.one.one.one (1.1.1.1) 35.234 ms 39.239 ms 37.009 ms