more peering stuff, whats going on.

[Chrysalis]

Ok me and some friend's noticed 2 of us on ntl were getting lag and slow speeds from the .us and and the others also on ntl were fine. So I did some traceroute's from some servers and noticed more priority routing going on with ntl.

it appears the only routes to my connection from america are level3 and cogent both which are performing poorly. Yet when I do traceroutes to www.ntlworld.com and to some ntl connections, backbones from cable & wireless, opentransit and tiscali are used so ntl have other peering routes. I am wondering since I live in a student area which will have high traffic utilisation are ntl sticking the traffic from my ip range onto cheaper transit.

1 41.25.232.72.reverse.layeredtech.com (72.232.25.41) 0.620 ms 0.582 ms 0.561 ms
2 10.1.6.9 (10.1.6.9) 0.716 ms 0.720 ms 0.713 ms
3 216.39.79.213 (216.39.79.213) 0.726 ms 0.564 ms 0.715 ms
4 ohr1-ge-0-1.FortWorthda1.savvis.net (216.39.64.33) 0.714 ms 0.563 ms 0.729 ms
5 bcs2-so-1-2-0.Dallas.savvis.net (208.172.129.229) 1.650 ms 2.143 ms 1.498 ms
6 dcr2-so-6-0-0.Atlanta.savvis.net (204.70.192.69) 21.777 ms 21.781 ms 21.775 ms
7 bcs2-so-6-3-0.Washington.savvis.net (204.70.192.66) 34.880 ms 34.888 ms 35.038 ms
8 cable-and-wireless-americas-operations.Washington.savvis.net (206.24.238.18) 35.194 ms 35.042 ms 35.050 ms
9 so-7-0-0-dcr1.nyk.cw.net (195.2.3.2) 40.396 ms 40.347 ms 40.185 ms
10 so-2-0-0-dcr2.tsd.cw.net (195.2.10.110) 115.381 ms 115.385 ms 115.405 ms
11 so-5-0-0-bcr1.tsd.cw.net (195.2.10.61) 115.700 ms 115.574 ms 115.557 ms
12 ntl1.tsd.cw.net (166.63.211.150) 116.786 ms 116.790 ms 116.784 ms
13 nth-bb-a-so-000-0.inet.ntl.com (62.253.185.97) 116.163 ms 116.039 ms 116.033 ms
14 bir-bb-b-so-200-0.inet.ntl.com (62.253.185.106) 117.894 ms 118.052 ms 118.070 ms
15 brhm-t2core-b-pos31.inet.ntl.com (62.253.185.150) 118.191 ms 118.991 ms 118.067 ms
16 brhm-t2cam1-b-ge-wan33.inet.ntl.com (213.106.231.190) 117.880 ms 117.885 ms 118.033 ms
17 brhm-warw-ubr-6-ge20.inet.ntl.com (213.106.228.214) 125.521 ms 125.263 ms 125.098 ms
18 cpc3-warw6-0-0-cust940.brhm.cable.ntl.com (86.20.247.173) 129.734 ms 126.201 ms 128.022 ms

1 41.25.232.72.reverse.layeredtech.com (72.232.25.41) 0.593 ms 0.571 ms 0.558 ms
2 10.1.6.13 (10.1.6.13) 0.714 ms 0.735 ms 0.559 ms
3 216.39.66.29 (216.39.66.29) 0.558 ms 0.720 ms 0.717 ms
4 216.39.64.17 (216.39.64.17) 0.713 ms 0.563 ms 0.559 ms
5 bcs2-so-1-2-0.Dallas.savvis.net (208.172.129.229) 1.495 ms 1.538 ms 1.656 ms
6 dcr2-so-6-0-0.Atlanta.savvis.net (204.70.192.69) 21.777 ms 21.794 ms 21.777 ms
7 bcs2-so-6-3-0.Washington.savvis.net (204.70.192.66) 35.037 ms 35.045 ms 35.039 ms
8 cable-and-wireless-americas-operations.Washington.savvis.net (206.24.238.18) 35.038 ms 35.046 ms 37.532 ms
9 so-7-0-0-dcr1.nyk.cw.net (195.2.3.2) 39.728 ms 39.722 ms 39.737 ms
10 so-2-0-0-dcr2.tsd.cw.net (195.2.10.110) 115.732 ms 115.787 ms 115.910 ms
11 so-5-0-0-bcr1.tsd.cw.net (195.2.10.61) 115.540 ms 115.545 ms 115.538 ms
12 ntl1.tsd.cw.net (166.63.211.150) 117.101 ms 117.259 ms 117.148 ms
13 win-bb-a-so-500-0.inet.ntl.com (213.105.172.129) 121.508 ms 121.633 ms 121.310 ms
14 win-dc-a-v903.inet.ntl.com (213.105.174.34) 121.479 ms 121.368 ms 121.483 ms

both of those using c&w

and trace to me

1 41.25.232.72.reverse.layeredtech.com (72.232.25.41) 0.477 ms 0.581 ms 0.560 ms
2 10.1.6.13 (10.1.6.13) 0.720 ms 0.564 ms 0.558 ms
3 216.39.79.217 (216.39.79.217) 6.955 ms 0.564 ms 0.559 ms
4 ohr1-ge-0-1.FortWorthda1.savvis.net (216.39.64.33) 0.559 ms 0.721 ms 0.557 ms
5 bcs1-so-3-3-0.Dallas.savvis.net (208.172.131.81) 4.304 ms 36.135 ms 1.808 ms
6 bcs2-so-6-0-0.Dallas.savvis.net (204.70.192.50) 2.121 ms 1.811 ms 1.805 ms
7 dcr2-so-6-0-0.Atlanta.savvis.net (204.70.192.69) 22.089 ms 21.957 ms 21.778 ms
8 bcs2-so-1-1-0.Washington.savvis.net (204.70.192.58) 35.350 ms 47.565 ms 81.063 ms
9 bcs2-so-2-0-0.NewYork.savvis.net (204.70.192.2) 39.874 ms 40.037 ms 39.874 ms
10 bcs1-so-0-0-0.Londonlnx.savvis.net (204.70.192.122) 109.611 ms 109.613 ms 109.629 ms
11 bcr1-so-0-0-0.Londonlnx.savvis.net (204.70.193.113) 109.671 ms 111.334 ms 109.772 ms
12 so-8-2.core2.London1.Level3.net (206.24.169.30) 109.787 ms 109.932 ms 109.767 ms
13 ae-0-55.bbr1.London1.Level3.net (4.68.116.129) 110.234 ms 109.926 ms 110.077 ms
14 ae-1-0.bbr1.London2.Level3.net (212.187.128.46) 110.388 ms 109.968 ms 110.079 ms
15 ge-3-0-0.gar1.London2.Level3.net (4.68.124.62) 110.234 ms 110.240 ms 110.253 ms
16 195.50.91.130 (195.50.91.130) 110.270 ms 110.560 ms 110.568 ms
17 ren-bb-a-so-100-0.inet.ntl.com (62.253.185.165) 121.779 ms 121.630 ms 121.779 ms
18 lee-bb-b-so-600-0.inet.ntl.com (62.253.185.162) 141.481 ms 123.073 ms 150.174 ms
19 lee-bb-a-ae0-0.inet.ntl.com (62.253.187.185) 122.870 ms 123.081 ms 123.030 ms
20 leic-t2core-a-so-000-0.inet.ntl.com (62.253.188.109) 125.216 ms 124.942 ms 125.992 ms
21 leic-t2cam1-a-ge-wan32.inet.ntl.com (82.3.33.9) 125.210 ms 125.061 ms 125.369 ms
22 leic-leic-ubr-8-ge01.inet.ntl.com (82.3.35.150) 125.710 ms 125.270 ms 125.368 ms

my outgoing trace again is crazy, the other's not locked onto level3/cogent go direct to london and out via a shorter route I go to manchester to level3 and then down to london level3 and finally handed over to savvis.

I get the same results with 3 different us isp's and servers and the same sorts of performance differences.

---------- Post added at 18:45 ---------- Previous post was at 18:43 ----------

another

1 colo-03-teb (64.18.131.129) 0.406 ms 0.230 ms 0.236 ms
2 core-01-teb (64.18.128.1) 65.484 ms * 67.582 ms
3 core-01-lga.njiix.net (64.20.32.173) 4.549 ms 0.669 ms 0.583 ms
4 ge-2-1-7-0.nyc33.ip.tiscali.net (213.200.66.221) 0.600 ms 0.555 ms 0.619 ms
5 * * *
6 ntl-gw.ip.tiscali.net (213.200.77.18) 79.218 ms 70.777 ms 82.045 ms
7 pop-bb-b-so-500-0.inet.ntl.com (62.253.184.5) 71.023 ms 70.920 ms 70.931 ms
8 lee-bb-a-so-700-0.inet.ntl.com (62.253.185.237) 76.523 ms 76.398 ms 76.404 ms
9 nth-bb-b-so-000-0.inet.ntl.com (62.253.185.101) 76.424 ms 76.402 ms 76.424 ms
10 pop-bb-a-so-100-0.inet.ntl.com (213.105.172.14) 78.374 ms 78.421 ms 78.518 ms
11 win-bb-b-so-500-0.inet.ntl.com (62.253.185.202) 134.885 ms 78.541 ms 78.660 ms
12 win-dc-b-v900.inet.ntl.com (62.253.188.166) 77.899 ms 77.904 ms 77.945 ms

to me

traceroute to 82.21.24.163 (82.21.24.163), 64 hops max, 40 byte packets
1 colo-03-teb (64.18.131.129) 0.313 ms 0.228 ms 0.218 ms
2 core-01-teb (64.18.128.1) 32.640 ms 3.309 ms 1.417 ms
3 core-01-lga.njiix.net (64.20.32.173) 0.612 ms 0.642 ms 0.586 ms
4 ge-2-1-7-0.nyc33.ip.tiscali.net (213.200.66.221) 0.596 ms 0.546 ms 0.594 ms
5 ge-7-11.car2.NewYork1.Level3.net (213.200.66.210) 0.851 ms 0.678 ms
ge-7-12.car2.NewYork1.Level3.net (213.200.66.214) 0.714 ms
6 ae-1-53.bbr1.NewYork1.Level3.net (4.68.97.65) 1.033 ms
ae-1-55.bbr1.NewYork1.Level3.net (4.68.97.129) 10.280 ms 1.010 ms
7 ae-1-0.bbr1.London2.Level3.net (212.187.128.46) 68.666 ms 68.604 ms 68.799 ms
8 ge-3-0-0.gar1.London2.Level3.net (4.68.124.62) 68.791 ms 68.855 ms 68.689 ms
9 195.50.91.130 (195.50.91.130) 69.282 ms 118.991 ms 69.383 ms
10 ren-bb-a-so-100-0.inet.ntl.com (62.253.185.165) 80.659 ms 80.484 ms 80.640 ms
11 lee-bb-b-so-600-0.inet.ntl.com (62.253.185.162) 100.646 ms 80.230 ms 80.281 ms
12 lee-bb-a-ae0-0.inet.ntl.com (62.253.187.185) 80.406 ms 197.414 ms 80.409 ms
13 leic-t2core-a-so-000-0.inet.ntl.com (62.253.188.109) 82.665 ms 82.598 ms 82.799 ms
14 leic-t2cam1-a-ge-wan31.inet.ntl.com (82.3.33.5) 82.657 ms 82.733 ms 83.660 ms
15 leic-leic-ubr-8-ge01.inet.ntl.com (82.3.35.150) 83.163 ms 83.117 ms 83.301 ms

[AbyssUnderground]

Let me join the club. I have a layeredtech server and have the same problems. Its the fact its a US server and we are in the UK. It will always be bad for gaming.

Ive been arguing with the LT team over this and they are trying to persuade me its perfectly fine. I get around 140-250ms to my server. Its worse in the evenings. Their savvis network is saturated and they wont admit it.

You should go to http://layer0.layeredtech.com and speak to them about it.

EDIT: My pings are 334ms tonight!!!!!!!

[Chrysalis]

its not LT I am complaining about its the route used to my connection, I will be a bit more clear.

The problem exists with many usa providers, infact level3 appears so bad cogent outperforms it.

Anyone not coming in on level3 is fine including ntl people, but the one other guy I know off on the level3 route has the exact same problems as me.

The problem could be level3 itself or ntl's level3 peering. I am also annoyed that ntl dont route all their customers the same.

I did try to traceroute your connection but it seems your website isnt hosted on your cable anymore so I cant see which route is been taken to it. I dont get the constant over 200ms but I am getting a lot of lag spikes and slow transfer speeds. These are happening at places like steadfast sagonet as well tho.

I think you jumped the gun blaming lt if other isp's and users are not getting the lag to your server and go over savvis the problem isnt at lt.

[AbyssUnderground]

its not LT I am complaining about its the route used to my connection, I will be a bit more clear.

The problem exists with many usa providers, infact level3 appears so bad cogent outperforms it.

Anyone not coming in on level3 is fine including ntl people, but the one other guy I know off on the level3 route has the exact same problems as me.

The problem could be level3 itself or ntl's level3 peering. I am also annoyed that ntl dont route all their customers the same.

I did try to traceroute your connection but it seems your website isnt hosted on your cable anymore so I cant see which route is been taken to it. I dont get the constant over 200ms but I am getting a lot of lag spikes and slow transfer speeds. These are happening at places like steadfast sagonet as well tho.

I think you jumped the gun blaming lt if other isp's and users are not getting the lag to your server and go over savvis the problem isnt at lt.
I know its savvis as Ive got people I know from all over the world to do tracert's and they all show problems as soon as it hits savvis. Its not level3 thats the problem. Its the router at the savvis side that level3 jump to. This is why the level3 connection is showing a high ping. Its the savvis router being a pain.

If you wish to ping my house you can try m3ezw.no-ip.com but I think my router blocks the pings. A tracert would be better.

People from sweden, USA itself, france and ireland are all having these ping problems to my server and the cause is savvis. They dont even hit the level3 network. It seems to be worse during the evenings and better during the morning/early afternoon.

I seem to get these pings spikes too, again, its the savvis network at fault. Their 2x 1Gbps connections from the UK to the US seem to be saturated a lot at peak times.

[Chrysalis]

Ok I will address the savvis bit first. You will see one of the traceroutes to myself I pasted is not over savvis at all that one was from justedge it hit tiscali, which it then used all the way to ntlworld.com but to me it went to level3 then to me taking a longer route.

sagonet which I didnt paste is just level3 all the way.

the trace to you from lt is as I expected over opentransit, every single ntl ip I have tested except me and one other guy uses opentransit. Also it is only 120ms to your ubr so I think your problems are on of the 3.

1 - your ubr is causing an extra 80ms lag (very over subscribed) but this isnt likely I think.
2 - your LT switch is faulty or saturated try asking to be moved.
3 - your LT server has a problem.

I think number 2 is most likely since you say everyone has the problem, whilst in my case only 2 of us out of about 20 people I have had testing have problems. I also get on non savvis routes.

Here is the trace to you excluding last hop of course.

traceroute to m3ezw.no-ip.com (82.9.140.38), 64 hops max, 60 byte packets
1 41.25.232.72.reverse.layeredtech.com (72.232.25.41) 0.626 ms 0.576 ms 0.578 ms
2 * * *
3 216.39.79.217 (216.39.79.217) 0.597 ms 0.572 ms 0.564 ms
4 ohr2-ge-0-1.FortWorthda1.savvis.net (216.39.64.41) 0.720 ms 0.570 ms 0.563 ms
5 bcs2-so-3-2-0.Dallas.savvis.net (208.172.131.85) 1.825 ms 1.662 ms 1.654 ms
6 dcr2-so-6-0-0.Atlanta.savvis.net (204.70.192.69) 21.939 ms 22.108 ms 21.782 ms
7 bcs2-so-1-2-0.Washington.savvis.net (204.70.192.62) 35.354 ms 37.390 ms 35.509 ms
8 bcs2-so-2-0-0.NewYork.savvis.net (204.70.192.2) 39.566 ms 39.418 ms 39.566 ms
9 P5-0.NYKCR2.New-york.opentransit.net (206.24.207.30) 39.566 ms 39.419 ms 39.411 ms
10 po10-0.nykcr3.NewYork.opentransit.net (193.251.241.114) 39.567 ms 39.730 ms 39.254 ms
11 po4-0.loncr3.London.opentransit.net (193.251.243.22) 110.549 ms 110.401 ms 110.239 ms
12 po0-0.brsbb1.Bristol.opentransit.net (193.251.243.38) 113.981 ms 113.984 ms 114.133 ms
13 ntl.GW.opentransit.net (193.251.254.146) 115.850 ms 115.941 ms 115.899 ms
14 pop-bb-a-so-010-0.inet.ntl.com (62.253.185.201) 114.136 ms 113.830 ms 113.823 ms
15 nth-bb-b-so-400-0.inet.ntl.com (213.105.172.13) 115.695 ms 115.865 ms 116.008 ms
16 lee-bb-a-so-600-0.inet.ntl.com (62.253.185.102) 120.543 ms 120.534 ms 120.701 ms
17 midd-t2core-a-pos31.inet.ntl.com (62.253.187.90) 122.117 ms 122.424 ms 122.251 ms
18 midd-t2cam1-a-ge-wan31.inet.ntl.com (213.106.237.34) 122.406 ms 122.098 ms 122.092 ms
19 midd-midd-ubr-2-ge10.inet.ntl.com (213.106.238.26) 125.836 ms 126.004 ms 125.835 ms

---------- Post added at 15:29 ---------- Previous post was at 15:23 ----------

justedge to you, also different.

traceroute to m3ezw.no-ip.com (82.9.140.38), 64 hops max, 60 byte packets
1 colo-03-teb (64.18.131.129) 0.483 ms 0.285 ms 0.233 ms
2 core-01-teb (64.18.128.1) 14.351 ms 0.458 ms 0.610 ms
3 core-01-lga.njiix.net (64.20.32.173) 0.611 ms 0.461 ms 0.611 ms
4 ge-2-1-7-0.nyc33.ip.tiscali.net (213.200.66.221) 0.609 ms 0.584 ms 0.611 ms
5 so-0-1-0.lon11.ip.tiscali.net (213.200.80.29) 68.452 ms 68.547 ms 68.455 ms
6 so7-3-0.LONCR1.London.opentransit.net (213.200.77.54) 68.565 ms 68.547 ms 68.568 ms
7 po12-0.loncr3.London.opentransit.net (193.251.242.169) 73.820 ms 73.792 ms 73.823 ms
8 po0-0.brsbb1.Bristol.opentransit.net (193.251.243.38) 77.321 ms 77.170 ms 77.195 ms
9 ntl.GW.opentransit.net (193.251.254.146) 74.072 ms 74.172 ms 74.073 ms
10 pop-bb-a-so-010-0.inet.ntl.com (62.253.185.201) 76.189 ms 76.168 ms 76.069 ms
11 nth-bb-b-so-400-0.inet.ntl.com (213.105.172.13) 76.197 ms 76.171 ms 76.197 ms
12 lee-bb-a-so-600-0.inet.ntl.com (62.253.185.102) 79.317 ms 79.294 ms 79.320 ms
13 midd-t2core-a-pos31.inet.ntl.com (62.253.187.90) 81.069 ms 81.036 ms 80.943 ms
14 midd-t2cam1-a-ge-wan33.inet.ntl.com (213.106.237.46) 81.063 ms 81.037 ms 80.944 ms
15 midd-midd-ubr-2-ge10.inet.ntl.com (213.106.238.26) 81.194 ms 81.414 ms 81.442 ms

I find it very odd that every ip I trace on ntl uses what appears to be bgp but I am routed over level3 or cogent everytime.

[AbyssUnderground]

Its actually none of them. My UBR is undersubscribed as the ping is < 10ms.

I know its savvis or LT's own network because my resellers servers are located all together and in a different part of the DC and on a completely different IP range. I get the same problems to all 8 of his servers.

EDIT: just remembered last week I came across an LT server (saw it in the netcraft toolbar) and I pinged it. Same poor results of 150ms+ pings.

[hammered]

Chrysalis,
I think you may have hit the nail on the head with your original comments. I think this may also be why I experience the same problems as you,with me living in another student area (Loughborough) I think I might do some routing tests over the weekend to see if I get the same results...

[JonathanLH]

Here's my tracert to one of our LT servers (goes a bit weird inside the savvis dc, suspect some firewalls are blocking pings or have them on low priority).
I generally manage about 200-400KB/s per thread from LT.

C:\WINDOWS>tracert lt.mrgigabit.com

Tracing route to lt.mrgigabit.com [72.232.60.213]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms router2 [10.0.0.252]
2 34 ms 7 ms 10 ms nott-grim-ubr-1-ge20.inet.ntl.com [10.189.223.254]
3 15 ms 8 ms 11 ms nott-t2cam1-b-v128.inet.ntl.com [80.4.47.225]
4 14 ms 10 ms 17 ms nott-t2core-b-ge-wan74.inet.ntl.com [80.1.79.201]
5 13 ms 12 ms 12 ms nth-bb-b-so-300-0.inet.ntl.com [62.253.188.37]
6 20 ms 12 ms 11 ms nth-bb-a-ae0-0.inet.ntl.com [62.253.185.117]
7 39 ms 33 ms 21 ms so-0-3-2-0.loncr4.london.opentransit.net [193.251.254.141]
8 16 ms 25 ms 18 ms po3-0.loncr3.london.opentransit.net [193.251.242.217]
9 41 ms 17 ms 29 ms so-6-0-0-0.loncr1.london.opentransit.net [193.251.242.170]
10 16 ms 17 ms 16 ms bcr2-so-4-0-0.londonlnx.savvis.net [206.24.169.157]
11 18 ms 15 ms 17 ms bcs1-so-2-1-0.londonlnx.savvis.net [204.70.193.118]
12 90 ms 19 ms 88 ms bcs2-so-0-0-0.newyork.savvis.net [204.70.192.121]
13 90 ms 87 ms 92 ms bcs2-so-4-0-0.washington.savvis.net [204.70.192.1]
14 101 ms 114 ms 122 ms dcr2-so-7-2-0.atlanta.savvis.net [204.70.192.57]
15 113 ms 128 ms 105 ms dcr2-so-7-2-0.atlanta.savvis.net [204.70.192.57]
16 128 ms 125 ms 146 ms bcs2-so-2-0-0.dallas.savvis.net [204.70.192.70]
17 138 ms 126 ms 125 ms bcs1-so-6-0-0.dallas.savvis.net [204.70.192.49]
18 127 ms 128 ms 128 ms ohr1-pos-4-0.fortworthda1.savvis.net [208.172.131.82]
19 138 ms 435 ms 131 ms 216.39.81.34
20 126 ms 125 ms * 216.39.81.34
21 * * * Request timed out.
22 * * * Request timed out.

and back:

Tracing route to "my house in grimsby :)" over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 209.60.232.72.reverse.layeredtech.com [72.232.60.209]
2 <1 ms <1 ms <1 ms 10.1.6.29
3 <1 ms <1 ms <1 ms 216.39.66.29
4 <1 ms <1 ms <1 ms 216.39.64.9
5 1 ms 1 ms 1 ms bcs2-so-3-2-0.Dallas.savvis.net [208.172.131.85]
6 41 ms 26 ms 21 ms dcr2-so-6-0-0.Atlanta.savvis.net [204.70.192.69]
7 35 ms 35 ms 35 ms bcs2-so-1-2-0.Washington.savvis.net [204.70.192.62]
8 39 ms 39 ms 39 ms bcs2-so-2-0-0.NewYork.savvis.net [204.70.192.2]
9 38 ms 39 ms 39 ms P5-0.NYKCR2.New-york.opentransit.net [206.24.207.30]
10 39 ms 40 ms 39 ms po10-0.nykcr3.NewYork.opentransit.net [193.251.241.114]
11 110 ms 110 ms 110 ms po4-0.loncr3.London.opentransit.net [193.251.243.22]
12 113 ms 113 ms 113 ms po0-0.brsbb1.Bristol.opentransit.net [193.251.243.38]
13 115 ms 115 ms 115 ms ntl.GW.opentransit.net [193.251.254.146]
14 113 ms 113 ms 113 ms pop-bb-a-so-010-0.inet.ntl.com [62.253.185.201]
15 115 ms 115 ms 115 ms nth-bb-b-so-400-0.inet.ntl.com [213.105.172.13]
16 117 ms 117 ms 117 ms nott-t2core-b-pos31.inet.ntl.com [62.253.185.38]
17 233 ms 133 ms 117 ms nott-t2cam1-b-ge-wan41.inet.ntl.com [80.1.79.166]
18 120 ms 119 ms 127 ms nott-grim-ubr-1-ge20.inet.ntl.com [80.4.47.226]
19 129 ms 134 ms 178 ms my house [hidden]

But in NTLs defense, they have great peering into some of the German datacenters. Can pull 10mbit on a single thread! Guess that's because they have part of their network in germany too :)

Feel free to run remote tracerts from any of our servers from www.mrgigabit.com/network.php

[Ignition]

Sorry I don't think the slow responses are due to ntl, and the fact that other customers on that provider using the 'nicer' transits seeing slower responses indicates this.

The ping differences could just be different routes being taken along the fibre rings across the atlantic in the case of Tiscali.

That said there is some routing weirdness on ntl's network which I've documented elsewhere as yet to no response.

It does give the impression that ntl's 'MPLS' network is being configured manually :eek:

BTW Chrysalis why are you complaining about 125ms from Texas and ~80 from New York? I mean wtf dude?

[AbyssUnderground]

Apparently 125ms from the LT DC is normal to the UK...

No im having billing problems with my server, ugh. The forum staff are helping me though.

[Ignition]

Apparently 125ms from the LT DC is normal to the UK...

No im having billing problems with my server, ugh. The forum staff are helping me though.

125ms from Texas to UK is fine, from NY you'd expect sub 90ms.

Just as a reminder traceroutes may not show all hops on networks.

[Paul]

125ms from Texas to UK is fine, from NY you'd expect sub 90ms.The current CF server is in New York and has always had pings around 110ms.

[Chrysalis]

Sorry I don't think the slow responses are due to ntl, and the fact that other customers on that provider using the 'nicer' transits seeing slower responses indicates this.

The ping differences could just be different routes being taken along the fibre rings across the atlantic in the case of Tiscali.

That said there is some routing weirdness on ntl's network which I've documented elsewhere as yet to no response.

It does give the impression that ntl's 'MPLS' network is being configured manually :eek:

BTW Chrysalis why are you complaining about 125ms from Texas and ~80 from New York? I mean wtf dude?

I am not latency is ok, but with lag spikes. It is the speed I have a problem with mainly.

I am very curious whats going on with the transit routing at the moment as to why it is been routed differently to most of the other ntl network nodes.

---------- Post added at 15:34 ---------- Previous post was at 15:30 ----------

here is my outbound route and is over 150ms due to some obscene routing to manchester then london.

many spikes as well due to ubr problems.

Tracing route to web1.imagis.info [72.232.25.42]
over a maximum of 30 hops:
1 7 ms 13 ms 10 ms UBR [10.8.143.254]
2 24 ms 28 ms 12 ms leic-t2cam1-a-ge914.inet.ntl.com [82.3.35.149]
3 29 ms 19 ms 19 ms leic-t2core-a-ge-210-0.inet.ntl.com [82.3.33.6]
4 21 ms 20 ms 17 ms lee-bb-a-so-220-0.inet.ntl.com [62.253.188.110]
5 13 ms 18 ms 25 ms lee-bb-b-ae0-0.inet.ntl.com [62.253.187.186]
6 25 ms 23 ms 55 ms man-bb-a-so-700-0.inet.ntl.com [62.253.185.193]
7 24 ms 25 ms 65 ms 212.187.137.1
8 29 ms 43 ms 23 ms so-6-0-0.mp2.manchesteruk1.level3.net [4.68.113.
114]
9 48 ms 25 ms 36 ms ae-1-0.bbr2.london1.level3.net [212.187.128.57]
10 25 ms 26 ms 27 ms ge-6-0.core2.london1.level3.net [4.68.116.36]
11 41 ms * 29 ms bcr1-so-4-0-0.londonlnx.savvis.net [206.24.169.2
9]
12 41 ms 46 ms 49 ms bcs1-so-1-1-0.londonlnx.savvis.net [204.70.193.1
14]
13 151 ms 104 ms 138 ms bcs2-so-0-0-0.newyork.savvis.net [204.70.192.121
]
14 178 ms 123 ms 112 ms bcs2-so-4-0-0.washington.savvis.net [204.70.192.1]
15 132 ms 112 ms 120 ms dcr2-so-7-2-0.Atlanta.savvis.net [204.70.192.57]
16 136 ms 155 ms 143 ms bcs2-so-2-0-0.dallas.savvis.net [204.70.192.70]
17 133 ms 147 ms 181 ms bcs1-so-6-0-0.dallas.savvis.net [204.70.192.49]
18 199 ms 155 ms 190 ms ohr1-pos-4-0.fortworthda1.savvis.net [208.172.131.82]
19 145 ms 156 ms 145 ms csr1-ve240.fortworthda1.savvis.net [216.39.64.34]
20 132 ms 152 ms 146 ms 216.39.79.214
21 * * * Request timed out.
22 157 ms 191 ms 151 ms 42.25.232.72.reverse.layeredtech.com [72.232.25.42]

Abyss I havent seen a trace from you, you blaming LT but I see nothing from you indicating that.

[Ignition]

Your spikes are all uBR probs :( Can't take any outbound trace you show 100% at face value there's 20+ms variation in the first couple of hops.

The obscene routing looks to be outside of ntl control, they have a transit partner in Manchester advertising that route and that's the best path from Leicester so they take it. Unfortunately it is in London so they'd get there faster via LINX but still, the only way they can fix that is to peer with Savvis who probably aren't interested being a tier 1.

If you say that latency is fine but occasionally speeds are slow no point in me doing traceroutes. Also it's not congestion as congestion means serialisation and serialisation means higher ping times, as you've indicated when showing genuinely congested routes.

I've nothing to indicate that's where the fault is either but if a poster above indicates issues to them from non-ntl people, and you yourself have said that they appear to like to overload switches, well....

6 bcr1-ge-4-3-0.Londonlnx.savvis.net (195.66.224.39) 0.938 ms 0.847 ms 0.861 ms
7 bcs2-so-1-1-0.Londonlnx.savvis.net (204.70.193.122) 1.058 ms 0.929 ms 1.065 ms
8 bcs1-as0-0.Londonlnx.savvis.net (204.70.193.201) 1.192 ms 1.303 ms 1.123 ms
9 bcs2-so-0-0-0.NewYork.savvis.net (204.70.192.121) 80.042 ms 80.321 ms 79.874 ms
10 bcs2-so-4-0-0.Washington.savvis.net (204.70.192.1) 83.409 ms 83.315 ms 83.609 ms
11 dcr2-so-7-2-0.Atlanta.savvis.net (204.70.192.57) 96.786 ms 96.730 ms 96.688 ms
12 bcs2-so-2-0-0.Dallas.savvis.net (204.70.192.70) 114.240 ms 119.220 ms 114.613 ms
13 bcs1-so-6-0-0.Dallas.savvis.net (204.70.192.49) 113.531 ms 113.227 ms 113.710 ms
14 ohr1-pos-4-0.FortWorthda1.savvis.net (208.172.131.82) 114.062 ms 114.631 ms 113.861 ms
15 csr2-ve240.FortWorthda1.savvis.net (216.39.64.35) 112.400 ms 112.386 ms 112.252 ms
16 216.39.79.218 (216.39.79.218) 112.731 ms 112.891 ms 112.200 ms
17 * * *
18 42.25.232.72.reverse.layeredtech.com (72.232.25.42) 115.684 ms 115.693 ms 115.339 ms

[Chrysalis]

Well abyss experience is completely different to mine, he is saying LT have problems full stop. But I have that server and my friend has 2 more. Between the 3 servers most user's generally have no speed problems, the ones that have problems are just me and 1 other guy on ntl who have the level3 routing, his ubr isnt as bad as mine and has fairly good numbers. I agree with you that because my ubr is in such a bad state it makes it harder to diagnose on my own connection which is why I had other people testing.

The manchester routing baffles me altogether, a guy on nottingham which issnt that far from leicester doesnt go through manchester level3, same with another guy in birmingham and one in liverpool, if liverpool can go direct to london then why is leics routing through manchester?

I will see if I can get traceroutes of them so you have the comparison's.

Your own trace shows no problem at lt.

Also another thing which abyss overlooked is that I have this problem with every .us provider that comes in over level3, the cogent ones are ok and I have now managed to get one that comes in on opentransit which is ok as well. The speed symptons are much different to my normal ubr speed related problem,s normally my speed is just generally slow but consistent, over level3 it is generally starting and fast then stalling for a sec or 2 then starting again but slowly speeds up so like a blade of a saw if looking at a speed graph once it hits a good speed it stalls again. The lag spikes could be solely my ubr but I am in dozens of ssh sessions together and I test on another which are nowhere near as bad, again others on lt ssh have no problems.

[JonathanLH]

zigzag is usually software rate limiters from my experience.
We have a 10mbit unmetered box in seattle, it's not set to 10mbit on the network card (god knows why), but is set from a cisco command on a gsr9000 series router. Due to this, a single thread download from say microsoft (also in seattle - with a 1ms ping!) runs at 10mbit on average, but zigzags nicely from about 8mbit to 12mbit.

[Chrysalis]

I wonder if a level3 router somewhere is rate limiting then as this zig zag thing is happening on every transfer I do over level3 transit from the .us. But it is much more aggressive then what you mention, the transfer will be normal for about 30 secs which probably is enough to not affect small downloads and web browsing, then it will stall, start slowly rising and when hits a decent speed suddenly stall again and keep following this momentum. I am running no software rate limiters on either the server or my pc.

[Ignition]

zigzag is usually software rate limiters from my experience.
We have a 10mbit unmetered box in seattle, it's not set to 10mbit on the network card (god knows why), but is set from a cisco command on a gsr9000 series router. Due to this, a single thread download from say microsoft (also in seattle - with a 1ms ping!) runs at 10mbit on average, but zigzags nicely from about 8mbit to 12mbit.

If it were set at 10 on the network card you'd never get 10 due to the overheads. This way you average out a nice 10Mbit.

I doubt Level3 are throttling, I suspect more likely packet loss causing the stall, throttling won't make an outright stall but packet loss will.

[Chrysalis]

I can get around 1.15mB/sec on 10 full duplex. My experience of isp's rate limiting isnt great I prefer good old fashion hardware limits if any limiting has to be done.

[darkone338]

i have a server at ServerMatrix.com (division of theplanet.com) and, despite it being on a 100mbit connection, and being able to transfer at close to 100mbit from other places, I get 4mbits MAX (on my 10mbit)... however interestingly... if I open another session, I can get another 4mbit (of course making my total download as 8mbit/sec)

Avg ping time, is 130ms to Dallas TX.

very strange ;)

[JonathanLH]

exactly the same here from our server at layeredtech, also in dallas, TX i believe

[Chrysalis]

darkone that is normal, from .us you should expect 4-5 mbit per thread due to the rtt.

I was getting around 400kB at start for about 30seconds and it would rise to around 500kB slowly then suddenly drop to about 10kB and then slowly rise again to 300kB+ and drop, repeating.

[JonathanLH]

i've had 10mbit before in 1 stream from the us, just depends on the bandwidth available.
although the rtt does have an impact, the tcp recieve window can offset that.
i knew a formula at one point that told you the max bandwidth for a given rtt and tcp window, but i forgot it :(

[Chrysalis]

I may be mistaken here but you right it is possible but depends on the senders send window as well. I believe if you set a large recieve window which of course should be a multiple of mss, then if the sender has a small send window you dont get the full advantage of the large recieve window.

Also if that download was over http their was every chance it was coming down of a local proxy server.

[Ignition]

i've had 10mbit before in 1 stream from the us, just depends on the bandwidth available.
although the rtt does have an impact, the tcp recieve window can offset that.
i knew a formula at one point that told you the max bandwidth for a given rtt and tcp window, but i forgot it :(

It's easy really. Think about how much phatness you can handle at once, and how often you have to acknowledge data.

RWIN of 2000 for RTT of 100ms = max speed of 40,000 bytes / second if that helps (2000 bytes requiring an ack and can ack 20 times per second, each ack taking 1/2 of RTT)

So it's RWIN * ((1/RTT in ms) * 1000) * 2

[Chrysalis]

some transits will cap transatlantic traffic I have seen it repeatedly.

cogent used to cap at 2mbit but now I think they cap at 5mbit.
level3 cap at 5mbit.
opentransit is at least 10mbit and is the best transatlantic performance I have seen.

telia and abovenet and others I dont know as I got no routes for servers going over that transit.

[JonathanLH]

i've had more than 5mbit through cogent from my german servers to usa ones

[Ignition]

some transits will cap transatlantic traffic I have seen it repeatedly.

cogent used to cap at 2mbit but now I think they cap at 5mbit.
level3 cap at 5mbit.
opentransit is at least 10mbit and is the best transatlantic performance I have seen.

telia and abovenet and others I dont know as I got no routes for servers going over that transit.

That's completely untrue. Transit providers don't cap and don't care about bandwidth usage, more the better from their point of view.

EDIT:

http://www.dslreports.com/speedtests/7401061;493096;596ae795b5089155e8b2fffa6b9fb53e;2. 0;speedtest.cogeco.net/1141036363

dslreports.com speed test result on 2006-02-27 05:32:43 EST:
7227 / 481
Your download speed : 7227 kbps or 903.4 KB/sec.

Your upload speed : 481 kbps or 60.1 KB/sec.

Tracing route to speedtest.cogeco.net [24.226.1.130]
over a maximum of 30 hops:

1 1 ms 1 ms 1 ms home.router [192.168.11.1]
2 * * * Request timed out.
3 8 ms 7 ms 7 ms gr1-stbc-rtr1.homechoice.co.uk [81.1.112.62]
4 9 ms 6 ms 7 ms 81-1-67-48.homechoice.co.uk [81.1.67.48]
5 8 ms 7 ms 7 ms 81-1-67-16.homechoice.co.uk [81.1.67.16]
6 7 ms 7 ms 7 ms 195.50.116.197
7 7 ms 7 ms 7 ms so-1-2-0.gar2.London1.Level3.net [212.113.0.118]
8 45 ms 7 ms 7 ms ae-0-56.bbr2.London1.Level3.net [4.68.116.162]
9 93 ms 93 ms 92 ms ae-0-0.bbr2.Chicago1.Level3.net [64.159.1.34]
10 96 ms 92 ms 93 ms ae-12-51.car2.Chicago1.Level3.net [4.68.101.3]
11 94 ms 93 ms 93 ms 4.79.64.14
12 104 ms 105 ms 105 ms rc2sh-pos13-0.mt.shawcable.net [66.163.77.13]
13 105 ms 105 ms 107 ms ra2sh-ge4-1.mt.shawcable.net [66.163.66.54]
14 105 ms 106 ms 106 ms rx0sh-cogeco-1.mt.bigpipeinc.com [66.244.255.2]
15 107 ms 107 ms 106 ms 24.226.0.102
16 107 ms 106 ms 109 ms cgowave-0-206.cgocable.net [24.226.0.206]
17 106 ms 106 ms 106 ms speedtest.cogeco.net [24.226.1.130]

[Chrysalis]

it may not be them capping themselves then but I have never witnessed above 5mbit on a single thread from usa over the atlantic on some providers. So I have just gone on my own experience (not ntl). Their of course has been times where I have downloaded on ntl at 10meg speeds on what appears to be a us server but in that case it could be a eu/uk mirror, cached content on a proxy so I dont know where its coming from.

Regarding the comment from germany to usa I dont know I havent sent data that way before over cogent. Did you raise rwin on both the 2 servers sending and recieving?