Anyone else getting really borked speeds this afternoon?

qasdfdsaq · 02-01-2012, 15:01

Certain destinations seem to have a lag spike of ~100ms and pretty bad speeds as a result:

BBC - Fine:

Code:

C:\Users\Administrator>tracert www.bbc.co.uk

Tracing route to www.bbc.net.uk [212.58.246.93]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  n1 [10.0.0.254]
  2    17 ms    12 ms    17 ms  cpc5-sgyl34-2-0-gw.sgyl.cable.virginmedia.com [92.235.188.1]
  3    10 ms    13 ms    32 ms  sgyl-core-1b-ge243.network.virginmedia.net [81.97.48.45]
  4    18 ms     7 ms     9 ms  sgyl-core-2b-ae2-0.network.virginmedia.net [195.182.178.94]
  5    13 ms    15 ms    12 ms  leed-bb-1b-ae5-0.network.virginmedia.net [81.97.48.81]
  6    19 ms    17 ms    21 ms  leed-bb-1a-ae0-0.network.virginmedia.net [62.253.187.185]
  7    29 ms    20 ms    20 ms  nrth-bb-1b-as2-0.network.virginmedia.net [62.253.185.101]
  8    19 ms    31 ms    16 ms  nrth-tmr-2-ae6-0.network.virginmedia.net [213.105.159.34]
  9    24 ms    38 ms    21 ms  tele-ic-1-as0-0.network.virginmedia.net [62.253.184.2]
 10    18 ms    24 ms    18 ms  pos6-1.rt0.thdo.bbc.co.uk [212.58.239.237]
 11    25 ms    19 ms    21 ms  212.58.238.153
 12    67 ms    30 ms    19 ms  te12-1.hsw1.cwwtf.bbc.co.uk [212.58.239.234]
 13    29 ms    19 ms    23 ms  212.58.255.12
 14    30 ms    29 ms    24 ms  bbc-vip014.cwwtf.bbc.co.uk [212.58.246.93]

OVH - Not fine:

Code:

C:\Users\Administrator>tracert s2.qasdfdsaq.com

Tracing route to s2.qasdfdsaq.com [178.32.52.252]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  n1 [10.0.0.254]
  2     5 ms     7 ms    12 ms  cpc5-sgyl34-2-0-gw.sgyl.cable.virginmedia.com [92.235.188.1]
  3    16 ms     7 ms     7 ms  sgyl-core-1a-ge133.network.virginmedia.net [81.97.48.37]
  4    11 ms     7 ms    13 ms  sgyl-core-2a-ae2-0.network.virginmedia.net [195.182.178.90]
  5    15 ms    15 ms    30 ms  manc-bb-1c-ae6-0.network.virginmedia.net [81.97.48.77]
  6    16 ms    15 ms    21 ms  manc-bb-1a-ae11-0.network.virginmedia.net [213.105.159.85]
  7    24 ms    23 ms    24 ms  glfd-bb-1b-ae1-0.network.virginmedia.net [213.105.175.146]
  8    63 ms    20 ms    18 ms  glfd-tmr-1-ae5-0.network.virginmedia.net [213.105.159.46]
  9    24 ms    28 ms    23 ms  tcl5-ic-1-as0-0.network.virginmedia.net [62.253.185.78]
 10    26 ms     *       20 ms  sw1.tc.lon.ovh.net [195.66.224.220]
 11   117 ms   121 ms   115 ms  rbx-g2-a9.fr.eu [91.121.131.181]
 12     *      117 ms   121 ms  vss-5a-6k.fr.eu [178.33.100.69]
 13   118 ms   114 ms   134 ms  s2.qasdfdsaq.com [178.32.52.252]

JANET - Also not fine, so not OVH's fault:

Code:

C:\Users\Administrator>tracert www.ja.net

Tracing route to www.ja.net [212.219.98.101]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  n1 [10.0.0.254]
  2    27 ms    12 ms    12 ms  cpc5-sgyl34-2-0-gw.sgyl.cable.virginmedia.com [92.235.188.1]
  3    10 ms    23 ms     8 ms  sgyl-core-1b-ge243.network.virginmedia.net [81.97.48.45]
  4     9 ms     7 ms     7 ms  sgyl-core-2b-ae2-0.network.virginmedia.net [195.182.178.94]
  5    16 ms    14 ms    16 ms  leed-bb-1b-ae5-0.network.virginmedia.net [81.97.48.81]
  6    11 ms    15 ms    17 ms  leed-bb-1a-ae0-0.network.virginmedia.net [62.253.187.185]
  7    17 ms    24 ms    27 ms  popl-bb-1b-as1-0.network.virginmedia.net [62.253.185.238]
  8    18 ms    29 ms    28 ms  tele-ic-5-ae0-0.network.virginmedia.net [213.105.159.117]
  9    22 ms    28 ms    26 ms  linx-gw2.ja.net [195.66.226.15]
 10    22 ms    25 ms    20 ms  ae1.lond-sbr3.ja.net [146.97.35.173]
 11   107 ms   123 ms   107 ms  ae12.lond-sbr1.ja.net [146.97.33.137]
 12   114 ms   107 ms   114 ms  be1.londsh-rbr1.ja.net [146.97.35.146]
 13    21 ms    22 ms    45 ms  janet-web-services.ja.net [146.97.137.170]
 14     *        *        *     Request timed out.
 15     *        *        *     Request timed out.
 16     *        *        *     Request timed out.
 17  ^C

Inbound trace:

Code:

traceroute to d.qasdfdsaq.com (92.235.190.31), 30 hops max, 60 byte packets
 1  core0.f.net.inf.ed.ac.uk (129.215.33.250)  16.471 ms  16.477 ms  16.481 ms
 2  harnoncourt.inf.ed.ac.uk (129.215.33.246)  0.262 ms  0.257 ms  0.246 ms
 3  vlan160.kb7-msfc.net.ed.ac.uk (129.215.160.254)  0.746 ms  0.794 ms  0.921 ms
 4  vlan686.s-pop.eastman.net.uk (194.81.56.57)  0.532 ms  0.605 ms  0.656 ms
 5  so-2-1-0.leed-sbr1.ja.net (146.97.42.45)  6.191 ms  6.267 ms  6.308 ms
 6  as3.lond-sbr1.ja.net (146.97.33.98)  10.451 ms  10.479 ms  10.470 ms
 7  ae13.lond-sbr4.ja.net (146.97.33.134)  10.846 ms  10.836 ms  10.871 ms
 8  ae0.lond-gw-ixp4.ja.net (146.97.35.182)  10.936 ms  10.953 ms  10.934 ms
 9  195.66.224.23 (195.66.224.23)  100.574 ms  96.998 ms  96.992 ms
10  glfd-tmr-1-as0-0.network.virginmedia.net (62.253.185.77)  98.357 ms  98.370 ms  98.410 ms
11  glfd-bb-1b-ae5-0.network.virginmedia.net (213.105.159.45)  102.483 ms  98.387 ms  101.945 ms
12  manc-bb-1a-ae3-0.network.virginmedia.net (213.105.175.145)  109.499 ms  105.057 ms  105.038 ms
13  manc-bb-1c-ae0-0.network.virginmedia.net (213.105.159.86)  108.150 ms  108.685 ms  108.705 ms
14  sgyl-core-2a-ae0-0.network.virginmedia.net (81.97.48.78)  142.906 ms  123.465 ms  123.588 ms
15  sgyl-core-1a-pc200.network.virginmedia.net (195.182.178.89)  121.618 ms  119.853 ms  123.527 ms
16  sgyl-cmts-34-gigaether-140.network.virginmedia.net (81.97.49.150)  116.428 ms  118.243 ms  125.483 ms
17  cpc5-sgyl34-2-0-cust542.sgyl.cable.virginmedia.com (92.235.190.31)  127.333 ms  124.061 ms  127.672 ms

Anyone else seeing this or have any ideas? Nothing's showing up on TBB charts, yet they go through LINX too. And it's not LINX vs TC or TH either.

Ignitionnet · 02-01-2012, 15:21

Seeing it, yes, any idea, I blame the return path as the forward looks fine. I reckon 195.66.224.23 is maxed out inbound or is in strife. Wouldn't be the first time VM have allowed a LINX port to saturate and won't be the last so that's why the first thought is overloaded peering.

Code:

traceroute to www.ja.net (212.219.98.101), 30 hops max, 40 byte packets

1 cpc8-mort6-2-0-gw.croy.cable.virginmedia.com (82.43.108.1) 10.971 ms 33.176 ms 12.508 ms
2 mort-geam-1a-ge115.network.virginmedia.net (81.96.225.25) 42.964 ms 9.697 ms 7.909 ms
3 croy-core-1a-tenge81-490.network.virginmedia.net (62.30.242.41) 13.019 ms 11.734 ms 7.439 ms
4 croy-core-2a-ae2-0.network.virginmedia.net (195.182.178.82) 12.398 ms 55.686 ms 18.481 ms
5 brnt-bb-1a-ae9-0.network.virginmedia.net (81.96.226.1) 9.937 ms 8.553 ms 12.377 ms
6 brnt-bb-1b-ae0-0.network.virginmedia.net (213.105.174.226) 11.571 ms 13.038 ms 23.108 ms
7 brnt-tmr-1-ae5-0.network.virginmedia.net (213.105.159.50) 9.480 ms 15.460 ms 18.334 ms
8 telc-ic-1-as0-0.network.virginmedia.net (62.253.185.74) 40.359 ms 13.004 ms 12.809 ms
9 linx-gw2.ja.net (195.66.226.15) 9.848 ms 11.109 ms 13.205 ms
10 ae1.lond-sbr3.ja.net (146.97.35.173) 9.426 ms 9.019 ms 14.776 ms
11 ae12.lond-sbr1.ja.net (146.97.33.137) 166.128 ms 108.305 ms 111.326 ms
12 be1.londsh-rbr1.ja.net (146.97.35.146) 112.759 ms 96.114 ms 109.629 ms
13 janet-web-services.ja.net (146.97.137.170) 10.466 ms 180.958 ms 13.771 ms

Code:

traceroute to s2.qasdfdsaq.com (178.32.52.252), 30 hops max, 40 byte packets

1 cpc8-mort6-2-0-gw.croy.cable.virginmedia.com (82.43.108.1) 20.740 ms 10.303 ms 23.766 ms
2 mort-geam-1a-ge215.network.virginmedia.net (81.96.225.153) 7.307 ms 9.918 ms 7.536 ms
3 croy-core-1a-tenge81-490.network.virginmedia.net (62.30.242.41) 10.311 ms 9.132 ms 11.651 ms
4 croy-core-2a-ae2-0.network.virginmedia.net (195.182.178.82) 12.905 ms 9.554 ms 8.106 ms
5 brnt-bb-1a-ae9-0.network.virginmedia.net (81.96.226.1) 11.114 ms 17.304 ms 23.627 ms
6 brnt-bb-1b-ae0-0.network.virginmedia.net (213.105.174.226) 11.403 ms 15.318 ms 8.493 ms
7 brnt-tmr-1-ae5-0.network.virginmedia.net (213.105.159.50) 11.085 ms 8.900 ms 7.723 ms
8 telc-ic-1-as0-0.network.virginmedia.net (62.253.185.74) 74.747 ms 10.049 ms 12.689 ms
9 * * * Request timed out.
10 rbx-g1-a9.fr.eu (94.23.122.66) 115.414 ms 106.765 ms 121.598 ms
11 * vss-5a-6k.fr.eu (178.33.100.26) 101.760 ms *
12 s2.qasdfdsaq.com (178.32.52.252) 105.792 ms 140.557 ms 115.563 ms

Code:

traceroute to 195.66.224.23 (195.66.224.23), 30 hops max, 40 byte packets

1 cpc8-mort6-2-0-gw.croy.cable.virginmedia.com (82.43.108.1) 9.035 ms 26.917 ms 11.243 ms
2 mort-geam-1a-ge115.network.virginmedia.net (81.96.225.25) 14.453 ms 8.607 ms 25.676 ms
3 croy-core-1a-tenge81-490.network.virginmedia.net (62.30.242.41) 9.722 ms 10.982 ms 12.246 ms
4 croy-core-2a-ae2-0.network.virginmedia.net (195.182.178.82) 32.250 ms 9.269 ms 10.525 ms
5 brnt-bb-1a-ae9-0.network.virginmedia.net (81.96.226.1) 25.168 ms 13.017 ms 12.506 ms
6 brnt-bb-1b-ae0-0.network.virginmedia.net (213.105.174.226) 9.950 ms 22.397 ms 9.741 ms
7 brnt-tmr-1-ae5-0.network.virginmedia.net (213.105.159.50) 20.487 ms 24.456 ms 15.523 ms
8 telc-ic-1-as0-0.network.virginmedia.net (62.253.185.74) 18.656 ms 9.408 ms 13.513 ms
9 ge0-0.pr2.linx.net (195.66.227.254) 65.178 ms 151.480 ms 27.726 ms
10 ge0-0.tr3.linx.net (195.66.248.126) 42.124 ms 23.369 ms 11.906 ms
11 fe2-1.tr1.linx.net (195.66.248.179) 14.833 ms 20.741 ms 11.336 ms
12 fe1-0.pr1.linx.net (195.66.248.229) 13.844 ms 28.742 ms 20.141 ms
13 195.66.224.23 (195.66.224.23) 255.599 ms 98.604 ms 115.449 ms

TBB doesn't use the bad route so is fine.

Code:

#	Host	Sent	Recvd	Best	Avg	Worst
1 	thinkbroadband-gw2.core-rs2.thdo.ncuk.net	15	15	0ms	3ms	11ms
2 	gi4-3-10-star1.bdr-rt1.thn.ncuk.net	15	15	0ms	8ms	125ms
3 	gi4-47-10-star1.bdr-rt3.thdo.ncuk.net	15	15	0ms	0ms	0ms
4 	linx-gw2.router.ntli.net	15	15	0ms	0ms	0ms
5 	brnt-tmr-1-as0-0.network.virginmedia.net	15	15	1ms	1ms	3ms
6 	brnt-bb-1b-ae5-0.network.virginmedia.net	15	15	1ms	13ms	90ms
7 	brnt-bb-1a-ae0-0.network.virginmedia.net	15	15	1ms	2ms	11ms
8 	croy-core-2a-ae0-0.network.virginmedia.net	15	15	1ms	1ms	2ms
10 	mort-geam-1a-tenge71.network.virginmedia.net	15	15	2ms	2ms	3ms
11 	mort-cmts-06-gigaether-151.network.virginmedia.net	15	15	2ms	2ms	3ms

qasdfdsaq · 02-01-2012, 16:06

Quote:

Originally Posted by Ignitionnet

Seeing it, yes, any idea, I blame the return path as the forward looks fine. I reckon 195.66.224.23 is maxed out inbound or is in strife. Wouldn't be the first time VM have allowed a LINX port to saturate and won't be the last so that's why the first thought is overloaded peering.

Interesting, return path only congestion. I hadn't thought of that. I forget these things are bidirectional sometimes. Was curious why the JANET trace only spiked on the second hop outside VM.

I wonder why today though... Locally my TB graph looks fairly tame overall.

Also I'd assumed OVH's sw1.tc.lon.ovh.net meant telecity but the IP is a LINX peer too, so yes it looks like a VM <=> LINX problem.

thenry · 03-01-2012, 19:36

a few heads eating away a bit of paid bandwidth! pings are shot as a result too.. aahh the joys of paying for services!

Ignitionnet · 03-01-2012, 19:49

Quote:

Originally Posted by qasdfdsaq

Interesting, return path only congestion. I hadn't thought of that. I forget these things are bidirectional sometimes. Was curious why the JANET trace only spiked on the second hop outside VM.

I wonder why today though... Locally my TB graph looks fairly tame overall.

Also I'd assumed OVH's sw1.tc.lon.ovh.net meant telecity but the IP is a LINX peer too, so yes it looks like a VM <=> LINX problem.

There's a LINX PoP in several Telecity facilities.

https://www.linx.net/about/partnerpages/telecity.html