re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Mine continues to improve...

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

I've been back OK since about 20 minutes ago. Previously, only some websites were loading.

I'm on the Hayes UBR.

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Good stuff :)

Must be a major routing fault somewhere. Glad I wasn't in this evening :D ;)

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

It's been ****'d for days no one bothered to read my post... Everynight latency goes mental, speeds half and sites don't work.

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

You just know someone out there is going to google that and then set up a site :erm:

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

All appears to be ok here in Lincolnshire now.

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

You may have a local fault unrelated to this. If you still have it after this, then you'll need to report it.

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Going to be a bit embarrassing for them. Nationwide outage.
Phones can't cope.
Support forum doesn't load or is unusable.
Status page doesn't load.
Shows how pathetic their support is really.

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Not really, you can't have 4 million support staff sitting waiting for your call or capacity for that much traffic to a single webpage.

Thankfully issues like this are rare. Something like this will be a CSO (Critical Service Outage) and tend to be resolved with a few hours.

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Thank the Lord for Cable Forum ;)

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Yes, when a few core network switches have a major problem they normally know about it quite quickly!

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Yes :)

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Yes a great help and has broken all records into the process.:)

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Yup headed off here first port of call.
You may need new servers in the future as you've probably gained a few people. :)

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

The access problem never hit RG41 if my circuit is anything to judge by.

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

UPDATE from VM's Facebook page:

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Region 06 is now up and running. Thank goodness for 3g and tethering.

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Got my connection back in South Gloucestershire at around 20:20.

So that's about 0.5 hours with a very unreliable connection and 2.5 hours completely down.

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

We're sorry that you've had a brief period of intermittent connection problems tonight, these were related to a routing hardware issue.

Our engineers have brought our services back on line and we're working to resolve any remaining problems at the highest priority.

We're very sorry for any inconvience caused tonight.

Thanks

http://community.virginmedia.com/t5/...973439#M111264

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Yes but a simple "If you are calling about the current service outage, we're aware of it already" would massively reduce the number of support staff required.

And a lot of sites have a "backup" high volume, low complexity system for reporting major outages - e.g. reverting to a single line of text instead of failing completely.

Having a contingency plan to deal with major outages is all part of being a major service provider.

---------- Post added at 20:40 ---------- Previous post was at 20:39 ----------

Actually I was getting a "Site too busy" response from Cable Forum most of the time while VM's own forums were slow, but functional.

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Funnily enough my 02 connection is like grease lightening tonight.
Perhaps half the customers have jumped ship and brought virgin down

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

My modem went off at midnight last night for an hour which usually means VM working somewhere and as it came back with a different IP I thought it was re-segmentation. However performance was below par and pings longer than usual.

Early this afternoon as surfing started to fall apart and changes in DNS servers made no difference I called India and spent almost 4 hours of moronic basic checks. Although I stated right from the onset that it had all the hallmarks of a rapidly deteriorating network issue he still insisted that I needed an engineer to change the Superhub.

I find it incredibly frustrating that VM use an outsourced support system which allows a slowly unfolding problem to deteriorate into a countrywide melt-down through downright incompetence. What a waste of an entire afternoon\evening dealing with a script driven individual who would not listen and called back three times hoping that I would think he was the specialist he was supposedly going to get to ring me.

If VM pay the people in India nothing, they get what they pay for, but if they pay the anything they are not getting value for money by any measure. Just my luck to get the same guy who said his supervisor was out but would ring me on his return (two months ago). This time he went the extra mile (hundreds) but ran out of skill level with 15 minutes. I guess his last promised call was forgotten about because the network problems I had been insisting upon were actually a reality.

FTTC has at long last reached within half a mile and we must be due soon. It is shame that at some point in time somebody in retentions will get hassle from me derived from the level of incompetence displayed by their Indian counterparts.

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Ahh seems to have come back up now.

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

it is Down here in keighley this is from my mobile

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Spoke too soon died again I'll stick with 02 who never seem to go down.

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Its fine here in Warrington

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

My O2 went down twice during the year I was with them.

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

The phone lines couldn't cope with the amount of attempts to call, so failed before they could connect. There was a service message applied as soon as the outage was identified.

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

No loss of connection in Liverpool.

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Didn't notice any problems throughout.

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

I've not noticed any problems here .. Medway area.

On the other hand I've not been online that much , so I may have missed it!

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Terrible here in Droylsden

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

As far as I am concerned, all is back up and running... (Area 21)

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Awful in Stockport (SK6) - was working OK but now very random again.

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Give it time, I've read of more Be* issues in the past year than should be of a 21st Century DSL network.

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Its working at about 75% at the moment here in Leeds not back to 100% yet

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Easily done, provided you can actually connect to the phone system rather than the NTS service Virgin use for their 150 and 08454541111 numbers.

If you had a few hundred thousand customers bombarding your support number for answers wouldn't you sweat a little? It's the reason most have automated menus on high-capacity servers to deal with that load, evidently this was just a little too much for the poor PBX to handle!

An alternative for the 503 Server Error if you will, like some sites' cute 404 errors. This'd be fine if their internet service hadn't degraded to the point that the error was worth a pretty page for.

I'm not sure hardware failure comes under that list of first things to check. Evidently something big was up with their internet service when leased lines fail as well - it must have been a very major switch failure within their core network for customers to have still been connected at the UBRs but not routing the traffic, even the VOD failed and that's supposed to be internal.

God bless Paul M's ability to build a stable forum to take the load of 5000+ angry cable customers wanting answers. This place is non-profit and survives on donations. :)

By the way - slow is better than "unavailable". At least if it's slow you can get somewhere. ;)

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

I must admit that when I found this site unavailable due to an overloaded server I assumed Alan Fry had posted another of his legendary 'plans'..... :D

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

All back up and running now though. :)

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Actually the forum did bomb out a few times with 'too many connection' warnings from vBulletin. These usually happen if there is a really high load on the server.

I can understand it on a site like this, but for VM's site to get 503 errors isnt acceptable. They could easily host the site in a cloud failover distribution, the costs would be negligible for VM to do so.

---------- Post added at 21:58 ---------- Previous post was at 21:54 ----------

Sounds a bit like mine. Woke up this morning with a new IP, service fine for most of the morning however was noticably slower this afternoon, with regular drops and server not found notices. Did a bunch of worldwide ping and speed tests and didnt find any problems. Shortly after this most sites failed to load. I thought it may be OpenDNS at first as I've got out network set to use them, however obviously it wasnt.

VOD wasnt affected in any way however.

(St Albans AL4 via Hemel)

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Have you read about the fault on the network on the community forum instead of making assumptions.

CLICK ME

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

service back to its usual 60% now.

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Not noticed any issues in Newcastle

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Lol.

I'm glad I missed all the fun, was at work till it was all fixed.

So anyone know what the fuss was about? Someone mentioned routing hardware failure (which is pretty embarrasing, as a major ISP should have backup routes on just about everything) and someone else mentioned an aircon failure in Poplar?

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

It sounds like this is either fixed or getting fixed - good news from my perspective. I was in work when the failure hit (Virgin Media leased line) and had all sorts of complaints - I worked out when I could not call Virgin it must be a Virgin fault but good to know it should have gone away before I get back into the office tomorrow! At home I haven't seen any disruption (VM cable) - but that might be because I was in work dealing with complaints during the major blackout!

Good work Virgin for getting it fixed, fingers crossed it doesn't happen again any time soon!

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Wrong completely. Have you *ANY* idea how much bandwidth actually costs for ISPs? (clue - it's not cheap. Try £10 per MegaBIT-per-second as a rough minimum, with 100 MegaBIT being the minimum size for bandwidth charges, not to mention transit costs (LINX/LoNAP for UK traffic and Cogent et al [expensive] for the worldwide traffic)).

Your domestic 50Mb connection you pay for is a contended service. If you have a 50Mb leased line you pay the high premium to have that service switched on, all the time, and bandwidth reserved exclusively for you. :)

---------- Post added at 22:59 ---------- Previous post was at 22:57 ----------

Backup routes yes, hardware failure is rare, and when it does happen it requires an engineer to go to the rack(s) in question and manually switch the hardware. That takes time because you have to do it slowly, carefully and properly. Not ham-fistedly so some script-kiddie in his mum's basement can continue his Warcraft campaign.

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Failover is usually soft in my world - the backup protocols are all there on Cisco kit. A SPOF is not how VM's network is designed otherwise you'd see everyone routing through that point in the traceroutes.

The engineer goes to the racks to replace or reset the kit in question after reading the logs and when the emergency change manager decides would be the best time.

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

How is this at all relevant to 503 errors on VM's website?

Internal access to VM's website has nothing to do with transit or peering, they wouldn't have to pay anyone for anything. In the worst case all they'd have to do is rent a £30/month server in someone else's datacentre to serve up "Yes, the site really is down" messages. £30 a month really is negligable for a company the size of VM.

Dunno about yours or VM's setups, but in our environment backup routes (and pretty much backup everything else) kick in automatically. Last major outage we had on our primary route, nobody even noticed except the net-ops guys, even support hadn't heard a thing from either side till I told them.

Having to manually fail-over faulty hardware in this day and age is pretty backwards.

In any case, the failure of a single router or any individual piece of hardware should not be able to cause anything as severe as this. Loss of an entire datacentre due to aircon failure however, could be a justifiable cause, though quite what was going on with the A/C would raise a few questions in itself.

[Edit]
Yeah, what Seph said.

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

We have a number of limits in place to stop anything bringing the site down completely. One of them cuts the forum to that site busy message if the server load exceeds a preset value. That limit is normally set to about 6.00 (the server normally runs at about 1.50 when busy). Today the load hit over 9.00 at one point, so that "safety valve" kicked in until the load fell again. I actually rasied it at one point to let more people on.

Too many connections actually has no connection with server load. Its purely down to the connection limit on mysql. Again we have this set such that the whole thing cant run away with itself and die.

We hit record concurrent guest and member figures, and survived very well. :cool:

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Not sure if it's related or not but my hub has been resetting itself for the last few hours now, several times while reading this thread..

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Still off here this morning looks like virgin are in the fubar.

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

All fine in Warrington.

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

I only seem to struggle getting connected to Digitalspy is that a bad thing, it takes 2 or 3 attempts.

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

My connection seems to have been alright for the last hour now, and i didn't have any issues yesterday.

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

And fine here in Plymouth.

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

I work this morning to find that my Tivo had the error network access so im guessing there is an issue on some parts again

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Loss of some tv channels in preston and tivo network error

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

loss of some tv channels here as well, and tivo network availability message being displayed, harlow, essex.

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

It's better here this morning to an extent but some websites are still patchy, one moment they load, the next they time out!

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

definatly not fixed

My colocated server still totally unreachable from my vm connection, but reachablr from my vodafone one.

Approximately 80% of websites still have timeout/loading issues, unless they are a "major" website (eg, google).

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Started OK this morning but patchy again now and taking a while to load sites.

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Mine had a slight issue at around 7pm (Walthamstow)

http://www.thinkbroadband.com/ping/s...18-01-2012.png

So glad I've got a backup Sky ADSL line. My router worked wonderfully and switched onto the ADSL connection with Virgin had issues and switched back when Virgin was up again.

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Hi, No TV channels this morning when I turned things on at 7.15am here in my town in West Yorks (Dewsbury), it just sat there stuck with the Virgin Media logo on screen till 8.10am when the tv came back on, not browsed channels yet. Internet is working, VM site is a bit slow to react if your on there looking around. I understand there are still issues.

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Very poor show from Virgin people a still having problems yet if you check the status for your area it says everything is fine.

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

I'm ok I think

https://www.cableforum.co.uk/images/...2012/01/48.png

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Respect :)

Fine comparison to VM's website, which did just run away with itself and die, with no error message or warning.

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Things seem to be OK here (Surrey)...

https://www.cableforum.co.uk/images/...2012/01/47.png

But it did take me 4 attempts to get this speed test done so some things are still not back entirely to normal.

The first two attempts it just hung on "searching for server based on ping" and the third time I manually selected a server and it "failed to load the latency file", all issues I have never had before.

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Seems fine but last night kept getting AP:52 when connecting up new V+ HD box

Then 2 hours later it worked

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Things do not look OK in Surrey.

The fault IS the inability to get to sites and the ping and latency issues are part of the problem. What you outline in your last two paragraphs is the fault:)

I have spoken to faults, at last, and they do still have issues with no known remedy time but as I was asked to exercise patience I will not be holding my breath waiting for this fix and its ADSL for me until the issue is resolved.

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

No problem in this part of Surrey (Camberley) readings, on wireless, as follows:
Ping 17ms
Download 19.61 (on 20mb service)
Upload 1.95
:)

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

I am in Bisley so we are in the same part of Surrey that seems OK.

I think we come under Guildford as a VM "area".

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

I think the beginning of happy days might have returned or at least look promising.

I can now instantly load the sites which an hour ago were no go so we may be on the road to normal service.

:):):):)

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

I am very suprised not to find a write-up of this on "the register", they usually have a pop at virgin, at every possible opportunity.

re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

PWND.

This is really really bad.

Re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

What lovely use of the English language! Is this website "The Sun" of the IT world?

Re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Except, its not 3hrs, they are still titsup. Google works, slowly, but google images just times out, as do many websites still.

Re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

No, that's a very technically specific term, used correctly by the journo :p:

Re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

I'm concerned VM's network appears to be run on tits.

Re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

"on" or "by" ?

Sorry, couldn't resist .

Re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

That too.

Re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

M41, I am getting a tad fed up now. I want my normal connection back, not this claggy one.

Re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

I have to say, it seems ok here in Nottingham

(I'm going to regret that .... MUST get some work done)

Re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

I didn't notice anything as I was in bed having a nap.
Just been reading about it on the bbc news site, love the comment 'dear virginmedia, have you tried switching it off and then back on again? love, the uk' (something like that)

I plan to be awake for the next unsheduled outage ;)

Re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

no on demand in harlow, essex again, tivo availability message being displayed again!

Re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

I found this amusing :D

https://www.cableforum.co.uk/images/...2012/01/47.jpg

Re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Hahahaha :D

Re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

If Darren isn't already a member on here I vote we invite him to join. :)

Re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Where abouts are you in Harlow? I live near the Shark Pub, Little Parndon.

Re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Seems to be okay here now unless my router is automatically rerouting.
Still were problems with virgin at work when I left for home which supplies a guest network for parts of the campus. They raised a support ticket over packet loss.
Sadly it seems Virgin have been a little economic with the truth saying it was only three hours I guess they need lessons on how to tell the time.

---------- Post added at 20:02 ---------- Previous post was at 19:57 ----------

The only problem with this comment is that you couldn't access that site either most of the time.

Re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

I haven't had any problems myself.:erm:

Re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

24 hours after the apocalypse and Virgin still are none the wiser as to what failed. This can't be good...

Re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Had a minor outage on Tivo on demand (Sidenote: Why when the network is down can you not access your own damn recordings?! That seems stupid!) which lasted no more than 30 mins.

Other than that, been pretty much rock solid all day. (AL4 St Albans via Hemel 50meg)

https://www.cableforum.co.uk/images/...2012/01/45.png

Re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Actually they found the problem. It was Richard Branson uploading new data to his spaceship.

Re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Not at all. If you look at the rest of the site, you'll find some very well written articles, mostly on the subject of IT (although they frequently branch out into other areas). One thing you will notice is that The Register (or El Reg as it prefers to call itself) has a very definite (and sometimes dirty) sense of humour which frequently comes out in the writing.

Re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Well the were more posts on this subject on that forum than the were on this forum, does that not tell you something.

Re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Yes we do. but it's business sensitive ;)

Re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Why would it be commercially sensitive? I can accept that you may be under an NDA, but to say what failed is commercially sensitive is just ludicrous. Unless ofc it'd be an admission that VM had failed to pay their LINX peerage fees (joke).

Either way, it still doesn't appear to be properly fixed. Since the outage, I'm having pretty bad lagspikes fairly regularly, like one every 2 hours, lasting about 5 minutes each, rendering services unusable during the lagspike.

Re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

Unless it is posted on the community forum we could not post it on a public forum as it would be classed as commercially sensitive information and could lose us our jobs, not worth it especially as we are awaiting our payout.

Re: MAJOR NETWORK ISSUE (17 Jan 2012)
 

I'm guessing that means it wasn't some sort of switch failure then...is the 'business sensitive' information good or bad news?