MAJOR NETWORK ISSUE (17 Jan 2012)

[sollp]

Originally Posted by qasdfdsaq

Yes, most sensible departments, including mine, do a combination of all 3 of the above. Although that said, I've seen our whole network taken down more often by human error and critical software bugs than actual hardware failure.

I see no excuse for this level of failure in any major ISP, particularly not one of the largest in the UK. And especially not lingering effects several days on as some people seem to be reporting...

Cleaner unplugging the Rack to hoover, like most Comm's Room's there are extensions off of extension's hanging off a 13amp plug!!

[qasdfdsaq]

Where I work,

a) Cleaners can't enter the server room(s)
b) Rack can't be unplugged accidentally
c) All cabling is neat and tidy, all inter-rack cabling is ducted or under the floor.

I'd expect any serious communications environment to be the same.

[Turkey Machine]

Originally Posted by Peter_

Unless it is posted on the community forum we could not post it on a public forum as it would be classed as commercially sensitive information and could lose us our jobs, not worth it especially as we are awaiting our payout.

Masque & DF & any other VM people who help us out: apologies guys, I hadn't realised Virgin's policy was that far-reaching. We definitely don't wanna lose you as you're one of the most valuable assets to this forum.

---------- Post added at 19:35 ---------- Previous post was at 19:32 ----------

Originally Posted by Sephiroth

I can imagine what's sensitive. The opposition would seize on any information that might lead them to conclude and expose insuffient resilience on VM's part.

As many techies will know, you can algorithmically re-route when there is a node failure; this put pressure on other nodes but is in any case standard everywhere these days.

Plus you can put in physical resilience so that another device takes over with the same IP address in case of failure of a key device. I suspect that didn't happen. If there is a site failure then, of course, this measure is not effective).

The trick is to do proper reliability analysis, identify the potential critical items and design the risk out accordingly. Just so you know, this is one of the day job things I do.

Which is exactly the reason why work want to get out of their current set-up - ever since their take-over, and the lies they told about the networks being kept separate, it's degraded to the point where it's not worth it for them any more. It was convenient at the time because about 2 years ago they wanted to move out of another provider's datacentre.

EDIT: Considering this took out a big chunk of leased line connectivity, I can see a considerable backlash of data contracts being re-negotiated and lost as a result of this.

---------- Post added at 19:39 ---------- Previous post was at 19:35 ----------

Originally Posted by qasdfdsaq

Where I work,

a) Cleaners can't enter the server room(s)
b) Rack can't be unplugged accidentally
c) All cabling is neat and tidy, all inter-rack cabling is ducted or under the floor.

I'd expect any serious communications environment to be the same.

Most of the time this is true, however when many third parties are involved communication does break down.

[MaverickJesus]

It was apparently a cooling failure that caused a core router to overheat, they were updating VM Business support tickets with that explanation shortly after it went tits up.

[Peter_]

Originally Posted by The PIT

Like I said when I tried it I couldn't get through. So it doesn't how many posted or what was put on a banner once the site was over whelmed.

The information was here though.

I never even noticed and I was online for the whole time but never looked at any forum until hours later and saw all the posts, rather surprised to say the least.

Now if I had been in work it would have worked wonders for my AHT.

Originally Posted by qasdfdsaq

Where I work,

a) Cleaners can't enter the server room(s)
b) Rack can't be unplugged accidentally
c) All cabling is neat and tidy, all inter-rack cabling is ducted or under the floor.

I'd expect any serious communications environment to be the same.

Absolutely.

[Sephiroth]

... and last time I entered a Virgin Media data centre server hall, so it was.

[qasdfdsaq]

Originally Posted by Sephiroth

... and last time I entered a Virgin Media data centre server hall, so it was.

Hey did you notice if their network was running on tits?

[Sephiroth]

Originally Posted by qasdfdsaq

Hey did you notice if their network was running on tits?

Qasi

In the spirit of your question, it was all up!

[qasdfdsaq]

xD

[sollp]

Originally Posted by qasdfdsaq

Where I work,

a) Cleaners can't enter the server room(s)
b) Rack can't be unplugged accidentally
c) All cabling is neat and tidy, all inter-rack cabling is ducted or under the floor.

I'd expect any serious communications environment to be the same.

Indeed you would, but after visiting many that aren't because of the few things I have put down.

[Tuvoc]

Originally Posted by Peter_

I never even noticed and I was online for the whole time but never looked at any forum until hours later and saw all the posts, rather surprised to say the least.

Strangely my service seemed totally unaffected, I was somewhat surprised to hear of it the next day. If it was network-wide, I don't understand...

[Peter_]

Originally Posted by Tuvoc

Strangely my service seemed totally unaffected, I was somewhat surprised to hear of it the next day. If it was network-wide, I don't understand...

Not everyone was affected including myself and many others.

[Chrysalis]

Network wide doesnt necessarily mean 100% affected.

[qasdfdsaq]

IIRC just about everything going via Leeds => Poplar was broken.