VM Routing and CloudFlare.com

[Ignitionnet]

Originally Posted by Chrysalis

I respect the need to balance across multiple links but of course one would expect an isp to have enough excess capacity cover a route going down in an outage so the fact VM are routing people over america doesnt sit well with me I'm afraid. Using multiple routes is fine, routing people over america to holland is not. I have only had this kind of routing on an isp before once, guess who the isp was?

.
.
.

ntl

nildram, entanet, aaisp, xilo, easynet, freeserve all manage to avoid it.

It's how routing works - the more complex your links to the outside world the higher the chance of 'sub-optimal' routes being optimal according to the EGP.

ntl's network and in turn VM's have a ton of paths out of the network, loads of full table transits and multiple peering points.

[Chrysalis]

ntl have a history of rubbish like this, I remember been routed to france then germany to traffic to the usa years back. Thanks for the attempt at educating me on BGP but BGP as shown in the document you linked ultimately works how its been configured to work. So VM have made a concious decision to prefer US routing over EU routing to a EU destination.

[GazCBG]

I also did a trace from traceroute.org
Use BT and Demon network.

BT being a bigger ISP then Demon I would think.
They both got to NL and I am sure other ISPs do

So where does VM get the idea it is a optimal routing when:
A. Other ISP route to NL, so that a total of 5 different companies I tried and they all route to NL, so that 5 to 1 send to NL.

B. Sending customer(s) to the USA, which takes long 40ms+ to load (I know it not long but loading time counts)

Also I heard hasn't gpt the best of US connections in the evening, it make thing worst?

But some VM customers get NL, I haven't had time to read your link yet Ignitionnet.

So is this VM making a half hearted attempt at saving money

[Chrysalis]

Ignition I dont understand you at times, I know you know its wrong because not too long ago you yourself was moaning about VM's peering situation. But then randomly you will come out and fight their corner. It looks clear for whatever reason VM have weighted the US nlayer link a higher priority than both abovenet and ams-ix, as to why we dont know. Likely financial tho.

[Ignitionnet]

Originally Posted by Chrysalis

ntl have a history of rubbish like this, I remember been routed to france then germany to traffic to the usa years back. Thanks for the attempt at educating me on BGP but BGP as shown in the document you linked ultimately works how its been configured to work. So VM have made a concious decision to prefer US routing over EU routing to a EU destination.

No conscious routing decision made, this is an anycast network, it's not an EU destination it is advertised out of multiple geographical areas.

Do recheck the BGP link I posted, it will make everything make more sense.

In summary according to BGP there's nothing to choose between both paths, so it's entirely down to the internal routing within the VM network to the external paths. The internal VM network will see two equal cost paths to the end destination so will take whatever is its preferred path internally to the transit.

It's quite possible that forcing parts of the network to use the EU node is actually what requires a conscious decision.

[Chrysalis]

BGP works how its configured you know that, each route is applied weighting to work to the isp's preference.

Sorry I missed the anycast part, however this begs the question still how this issue is not affecting other isp's. If we remove all the jitter from pip's trace his is clearly closer so should be a favourable destination for anycast.

How do you know both paths are configured with equal priority?

[Ignitionnet]

Originally Posted by Chrysalis

Ignition I dont understand you at times, I know you know its wrong because not too long ago you yourself was moaning about VM's peering situation. But then randomly you will comne out and fight their corner. It looks clear fowhatever reason VM have weighted the US nlayer link a higher priority than both abovenet and ams-ix, as to why we dont know. Likely financial tho.

Both links use nLayer. Both routes are going to the same IP address 199.27.135.74, neither is being preferred.

The AS Path to that destination is the same length on both, so the routing decision comes down to which route internally within the VM network to get to the transit is the best option, which will vary depending where on the network you are and your proximity to the two links in question.

popl-bb-1a has the following paths to get to nlayer, which is, actually, all the VM network cares about in this case:

popl-tmr-1
amst-ic-1
ams-ix.ae1.cr1.ams2.nl.nlayer.net

popl-bb-1b
popl-tmr-2
tele-ic-2
eqix.xe-3-3-0.cr2.iad1.us.nlayer.net

Each hop will have an IS-IS cost, it chooses the shortest path - Amsterdam.

popl-bb-1b has the following options:

popl-tmr-2
tele-ic-2
eqix.xe-3-3-0.cr2.iad1.us.nlayer.net

popl-bb-1a
popl-tmr-1
amst-ic-1
ams-ix.ae1.cr1.ams2.nl.nlayer.net

Again - per the routing protocol it chooses the shortest path - Equinix.

This is not some manual tuning to avoid costs. I'm sticking up for VM on this because you are wrong with your accusations of manual routing for cost purposes (in this case).

---------- Post added at 23:25 ---------- Previous post was at 23:19 ----------

Originally Posted by Chrysalis

BGP works how its configured you know that, each route is applied weighting to work to the isp's preference.

Sorry I missed the anycast part, however this begs the question still how this issue is not affecting other isp's. If we remove all the jitter from pip's trace his is clearly closer so should be a favourable destination for anycast.

How do you know both paths are configured with equal priority?

Anycast doesn't work like that, it doesn't measure latency, decisions are made by BGP and whatever IGPs the ISPs are using. I have no idea at all of the relevance of latency in pip's case, EIGRP is the only routing protocol that comes to mind that uses latency in routing decisions, VM use an IP/IS-IS/MPBGP network. He goes to Amsterdam because his route towards nlayer goes via popl-bb-1a, so the lowest metric route internally is via Amsterdam rather than going via popl-bb-1b to go via Equinix.

Perhaps VM are the only operator who peer with nlayer in multiple places?

[pip08456]

But I don't have a case!

[Ignitionnet]

Anyway I hope this clears things up. I'm not sticking up for anyone, I'm pointing out that this anomaly, though unwanted, is everything working as intended.

As a final thought why would VM intentionally route to the same location via settlement free peering at New York rather than settlement free peering at Amsterdam?

[GazCBG]

Originally Posted by Ignitionnet

This is not some manual tuning to avoid costs. I'm sticking up for VM on this because you are wrong with your accusations of manual routing for cost purposes (in this case).

I was more asking if VM was doing it to save money.

---------- Post added at 23:33 ---------- Previous post was at 23:29 ----------

I have 1 more questions.

They say they are going to bring London, Paris and Frankfurt DC online using the same setup, will that make any differents, or will it be the same?

[Ignitionnet]

Originally Posted by Big-Gaz

I was more asking if VM was doing it to save money.

Nope, it's a side-effect of how routing works. VM aren't paying a metered rate for traffic either way.

Excuse the technobabble it's kinda a technical issue.

I guess if it's causing a problem reaching out to the guys via the forums or Twitter might be a good move. They can modify routing manually.

---------- Post added at 23:36 ---------- Previous post was at 23:35 ----------

Originally Posted by Big-Gaz

They say they are going to bring London, Paris and Frankfurt DC online using the same setup, will that make any differents, or will it be the same?

Might do, unsure how VM's network is constructed to that detail but if they peer with nLayer in Paris and Frankfurt they'll have another 2 routes to get to that IP address, one of which might be preferable to the US datacentre.

[Chrysalis]

that explanation is making sense so pip just happens to be nearer to the route that has the nl route so for him gets picked.

This may go some way to explaining why VM's network has more routing oddities as well since on a adsl isp everyone tends to go to one central routing point (usually in london) whilst VM have a national network which can drop you off at different points.

[Ignitionnet]

Originally Posted by Chrysalis

that explanation is making sense so pip just happens to be nearer to the route that has the nl route so for him gets picked.

This may go some way to explaining why VM's network has more routing oddities as well since on a adsl isp everyone tends to go to one central routing point (usually in london) whilst VM have a national network which can drop you off at different points.

Correct.

Most ADSL is centralised as their subscribers all terminate on LNS which live in data centres, Easynet's peering is quite centralised on London for historical reasons.

VM have transits running from all over the place, Manchester, Guildford, Winnersh, Docklands, Brentford, etc, etc. The network shifts hundreds of Gbit/s and to be funnelling all of that through London is neither feasible nor intelligent. Routers running with 4 x 40GigE port channels for the sake of centralising would be phenomenally expensive. Getting traffic off the network as quickly as reasonably possible reduces capacity requirements internally as well as keeping routing more available as more exit points.

All that said VM's execution of this strategy can leave a lot to be desired at times.

Not sticking up for anyone, they just happen to be right this time. Nearly everyone is right now and then, I'll be joining the chorus when they next have a routing mishap due to their own traffic management, no fear

[GazCBG]

Ah, now I understand why different VM customers getting the different route.
I thought it was all centalised.

Thanks for the further info

[Ignitionnet]

Originally Posted by pip08456

But I don't have a case!

You are one.

---------- Post added at 14:09 ---------- Previous post was at 14:08 ----------

Originally Posted by Big-Gaz

Ah, now I understand why different VM customers getting the different route.
I thought it was all centalised.

Thanks for the further info

No worries!

This shouldn't make much difference performance wise to be honest - does the page load up fairly quickly regardless?