BT admits DNS outage at weekend
 

Source TheRegister

A major outage that affected millions of their customers and they just went silent with support telling customers who called it was their own equipment. ISP support can be so poor sometimes.

A graph from bgpmon shows the drop in bandwidth during the outage

https://www.cableforum.co.uk/images/local/2014/06/1.png
From here

Re: BT admits DNS outage at weekend
 

That's not a bandwidth graph and it contradicts that the issue was just DNS. Have another look at the label on the y-axis.

Re: BT admits DNS outage at weekend
 

If DNS goes down, bandwidth goes down too as people don't load up their web pages. Still, the graph is BGP router availability rather than bandwidth but this is how bgmonitor tweeted it, which is probably why I read the graph a different way. The link to the tweet was in my previous post but i'll post it below anyway:

Re: BT admits DNS outage at weekend
 

It wasn't *just* a DNS issue. I'd stuck my own router on my BT line on Friday night (configured to OpenDNS) and that failed Saturday morning. Some sites would work while others didn't. It strikes me as being a pretty serious routing issue.

Re: BT admits DNS outage at weekend
 

Same for me and I use OpenDNS. I had to route through a proxy on my dedicated server to access sites like Facebook.

Re: BT admits DNS outage at weekend
 

The lack of info from BT has been pretty bad, just a general statement from a PR guy.

Seen a lot of people saying it was mostly major sites, such as amazon, twitter, facebook, netflix etc which seemed to all have problems while others were ok. Wondering if these sites were just noticed more as they are popular or if it was some kind of routing they do for particular sites.

Re: BT admits DNS outage at weekend
 

It's not router availability it's prefixes being advertised from the AS. Doesn't mean a router went down but that it stopped advertising some prefixes, hence my BT line lost its return path as it was no longer being advertised outside AS2856.

Actually it seemed to stop being advertised internally too. I suspect fat fingers when a configuration change was made to a route reflector; a single router going unavailable shouldn't cause this, there are almost certainly primary and shadow reflectors on the network.

DNS servers per se didn't go down, this was actually a routing issue on the BT network a side effect of which was that DNS became unavailable.

I do not use BT's DNS and had to admin down my Infinity line as both default gateway and the site I use for IP SLA were available :(

You misread the graph, it happens, no need to worry :)

---------- Post added at 12:51 ---------- Previous post was at 12:43 ----------

Just noticed more. Everything from those to Deviantart to CNN were hosed, only the first 5 hops are common throughout.

Tracerting

www.twitter.com[199.16.156.38] ,Maximum hops:30
1 5ms 5ms 5ms 217.32.143.234
2 5ms 5ms 13ms 217.32.144.30
3 13ms 8ms 7ms 213.120.181.214
4 7ms 7ms 7ms 213.120.180.167
5 8ms 9ms 8ms 217.41.169.107
6 6ms 7ms 7ms 109.159.251.121
...
< Completed >


Tracerting

www.amazon.co.uk[178.236.7.220] ,Maximum hops:30
1 5ms 4ms 4ms 217.32.143.234
2 5ms 5ms 18ms 217.32.144.30
3 7ms 6ms 6ms 109.159.244.250
4 11ms 6ms 6ms 213.120.180.167
5 7ms 17ms 8ms 217.41.169.107
6 7ms 7ms 7ms 109.159.251.109
...
< Completed >

www.netflix.com[54.245.104.31] ,Maximum hops:30
1 5ms 5ms 5ms 217.32.143.234
2 5ms 5ms 5ms 217.32.144.30
3 8ms 9ms 8ms 213.120.181.26
4 8ms 14ms 8ms 213.120.180.167
5 9ms 10ms 11ms 217.41.169.107
6 7ms 8ms 7ms 109.159.251.85
...
< Completed >

www.facebook.com[31.13.80.65] ,Maximum hops:30
1 5ms 5ms 5ms 217.32.143.234
2 5ms 5ms 5ms 217.32.144.30
3 9ms 7ms 7ms 212.140.235.130
4 7ms 7ms 7ms 213.120.180.167
5 8ms 8ms 8ms 217.41.169.107
6 6ms 6ms 6ms 109.159.255.159
...
< Completed >

Tracertingwww.rt.com[62.213.85.4] ,Maximum hops:30
1 5ms 5ms 5ms 217.32.143.234
2 5ms 5ms 5ms 217.32.144.46
3 7ms 7ms 6ms 212.140.235.226
4 7ms 7ms 7ms 213.120.180.167
5 8ms 8ms 9ms 217.41.169.107
6 6ms 6ms 6ms 109.159.255.163
...

You get the idea. Before diverging still in BT's 21CN core, indeed still in Yorkshire at hop 6 - just entering BT's core network.

Re: BT admits DNS outage at weekend
 

Yeah, makes perfect sense. I read two articles initially which said it was a DNS problem without mentioning any other issues, so didn't think any further than that.

The fact that BT wouldn't talk to the media or give them any information,(beside the fact it wasn't related to them playing with website blocking filters) you may be right about it being human error.

Re: BT admits DNS outage at weekend
 

Blimey if this was the result of a b0rked change someone's going to get a size 11 in the nads for not following post implementation.

Although if someone was *cough* trying to slip in a sneaky one outside of change control make that a size 13

Re: BT admits DNS outage at weekend
 

Some of the routers on 109.159.25*.* are 10GB and some say XE. Any idea what the XE is?

Re: BT admits DNS outage at weekend
 

There aren't many other things that can cause a properly designed network to suddenly drop prefixes like this. Even if one router did fail the other one would be receiving the prefixes from the access edge, advertising them into the core and in turn the Internet facing routers advertising them onwards.

If primary and shadow both borked, or routes were getting nulled in the network someone screwed up. Friday is a good night to do changes, less interactive business traffic.

https://www.linkedin.com/in/neilmcrae can I'm sure advise :)

EDIT: A swift chat indicates that the issue was due to routing problems between 2 autonomous systems on the BT network. Various services including name, Cleanfeed, etc, are hosted in 5400, customer subnets including the one I'm using are hosted in 2856, and they stopped talking to each other, at one point there was a single route with those two systems in its path. Ooops.

route: 109.144.0.0/12
descr: BT Public Internet Service
origin: AS2856

Re: BT admits DNS outage at weekend
 

xe is usually used to denote 10GbE on Juniper devices

te for Ciscos

Re: BT admits DNS outage at weekend
 

BT Outage, VM DNS Outage, 4 million outages from Microsoft stealing some domain roots and hosing them too.... Been a great week for the internet...

Re: BT admits DNS outage at weekend
 

Mystery solved, great edit! Explains the prefix gap. Guess they would now have added another route with a lower weight or local preference to minimise future failures between the two...

---------- Post added at 14:00 ---------- Previous post was at 14:00 ----------

Cheers for the info!

Re: BT admits DNS outage at weekend
 

I thought it meant Xtreme Elephant.

Re: BT admits DNS outage at weekend
 

Didn't we move away from those telecom trunks some time ago? :angel:

Re: BT admits DNS outage at weekend
 

No, they're still teaching Trunking today!

Re: BT admits DNS outage at weekend
 

Gotta have those trunk ports to satisfy your vlan fetishes

Re: BT admits DNS outage at weekend
 

Those two networks are resiliently directly connected, the issue was what one of them was advertising to the other rather than connectivity issues.

Ah well!

Re: BT admits DNS outage at weekend
 

So, not so much fat fingers as a not properly tested failover mechanism?

Re: BT admits DNS outage at weekend
 

Nah, a series of unfortunate events.

Re: BT admits DNS outage at weekend
 

I'll keep that one in mind next time I hose the network by accidentally bridging the wrong interface... :-P

The sequence in which my fingers fell on the keyboard was merely a series of unfortunate events, sir!

Re: BT admits DNS outage at weekend
 

Wasn't a typo or trunking the wrong VLAN, it was a mistake highlighted by 'a series of unfortunate events'. In itself it would've been okay.