IT outages are reported across the world
 

https://www.bbc.com/news/live/cnk4jdwp49et

https://www.cableforum.uk/board/atta...1&d=1721378210

More details here

https://www.thetimes.com/uk/technolo...rike-b3n2h5gmc

Update - if anyone is travelling today (by air), probably worth checking the airport website you are flying from - this is a banner on Leeds Bradford

Re: IT outages are reported across the world
 

Sky news came back around 9am.

But they have no banner or clock at the bottom of the screen.

Re: IT outages are reported across the world
 

https://downdetector.com/?_gl=1*1al7...OTE3Ni4wLjAuMA..

Pretty alarming

Re: IT outages are reported across the world
 

Anyone know how Liz Truss's first day at Microsoft is going?

Re: IT outages are reported across the world
 

Re: IT outages are reported across the world
 

Apparently, cause identified and fix issued.

Someone is going to get their wrists slapped.

Re: IT outages are reported across the world
 

My doctors surgery last Monday failed to put an urgent script for my dads medication by Thursday.


If they haven't done it my sister will raise hell and report the surgery to the appropriate, who will blame this IT outage.

---------- Post added at 11:27 ---------- Previous post was at 11:26 ----------


Trouble is they test these updates on test systems, but the real world is completely different.

Re: IT outages are reported across the world
 

Coincidentally quite a few financial institutions and others are currently undergoing "planned maintenance" on their websites when you try to log on. Admirable that they're turning a huge outtage into an opportunity to do some housekeeping :erm:

Re: IT outages are reported across the world
 

Hope no one has any Crowdstrike shares :)

Re: IT outages are reported across the world
 

:D

Re: IT outages are reported across the world
 

I blame the Labour Government.

Re: IT outages are reported across the world
 

From what i heard it did not affect personal PC's running windows. I have been using Linux for the last 2 months on my main PC and it only affected me via services that i need to connect to that were on windows machines.

Re: IT outages are reported across the world
 

My work please has over a 100 pcs and equipment affected.

Re: IT outages are reported across the world
 

Quelle surprise....

Re: IT outages are reported across the world
 

It is de rigueur on this forum to blame the Government for everything. I am hoping to boost my popularity and reputation by getting in first.

Re: IT outages are reported across the world
 

:LOL:

Re: IT outages are reported across the world
 

Don't worry it couldn't happen to your password manager !

I read on CD earlier people on Win11 were getting BSODs,,,

I hope everything is better now!!

Re: IT outages are reported across the world
 

Faulty update to Anti-Virus software.

The update was released by Crowdstrike overnight, so most systems failed this morning.

Basicially they crashed with the good old Blue Screen of Death.

The fix apparently involves one file.

Re: IT outages are reported across the world
 

In all seriousness though, shouldn't we have seen at least one Government minister providing reassurance on this global catastrophe? Millions of people affected (many with no idea about the impacts) and not even a peep out of the Government :mad:

Perhaps they've forgotten they are now in charge.

Re: IT outages are reported across the world
 

I’m reading that to fix it you have to boot into safe mode and delete a single .sys file. But also that there is no remote fix, because without safe mode the PC just goes into an endless loop of reboots. So corporate users big enough to have in-house IT support will get up and running again soon enough, but private and small business users are screwed unless they buy support or are confident enough to find and follow online instructions. Many will be able to do this, many more will not.

I predict there will be a class-action lawsuit against Crowdstrike before the week is out.

---------- Post added at 17:07 ---------- Previous post was at 17:03 ----------

Not being a windows expert (or even a windows user) I can’t vouch for the veracity of this, but I am seeing multiple posts across the interwebs saying the manual fix is:

1. Boot Windows into Safe Mode or WRE.
2. Go to C:\Windows\System32\drivers\CrowdStrike
3. Locate and delete file matching "C-00000291*.sys"
4. Boot normally.

Re: IT outages are reported across the world
 

That looks reasonable.

Re: IT outages are reported across the world
 

I am surprised that any global updates are not pushed out in waves to assess impact of initial deployments.

Re: IT outages are reported across the world
 

Well that was a day and a half !!!!

---------- Post added at 20:22 ---------- Previous post was at 20:21 ----------

That is the fix from crowdstrike and it works just fine

Re: IT outages are reported across the world
 

Contactless payments were fine at Troon which was the extent of my IT use for the day.

Does seem slightly worrying that a minor error can have such significant consequences.

Re: IT outages are reported across the world
 

Not jealous much.

Although, at halfway a couple of my bets are still live.

Re: IT outages are reported across the world
 

More to the point, you would think they would check a machine still works after being updated (and rebooted) before pushing it out to anyone.

Re: IT outages are reported across the world
 

A reboot wasn’t required to trigger, the csagent.sys file faulted in non paged and put machines into a boot loop

Our biggest headache was walking people on physical laptops through deleting the file as bitlocker caused some fun

AVD & w365 we just restored

Servers we automated the fix

Re: IT outages are reported across the world
 

I must admit as a Linux user this was me yesterday :)

https://x.com/i/status/1814235168363987221

Re: IT outages are reported across the world
 

Yes, of course, pre-roll out testing is a given and I do not understand why this was not done for a enterprise-grade production deployment.

What is flawed is the lack of a phased roll out to assess impact of new delivery. I am guessing that this update came via a Crowdstrike update process rather than Microsoft Update. This is why Microsoft is not really getting the heat for this.

I do not think Linux or MacOS would have been taken down this easily although I may be wrong.

Re: IT outages are reported across the world
 

Same here.:)

Re: IT outages are reported across the world
 

Please tell me you don’t cosplay like a PGA pro when you go to these things … Glasgow Central was full of utter tools getting trains down to Troon the other day :rofl:

Re: IT outages are reported across the world
 

:D

Happy to confirm a hoody, shorts (since it was still warmish for Scots) and a fairly comfortable pair of trainers was my attire.

The other “official” outfit seemed to be the tracksuit of a previous Open or Ryder Cup in the last two decades.

Re: IT outages are reported across the world
 

I'd guess it was done, then something else went wrong during the world-wide deployment. Such as a file got corrupted during transfer, or someone accidentally published the wrong file.

Re: IT outages are reported across the world
 

quite a shocking amount of computers that were affected according to MS and that is only the ones I would suspect have telemetry on as well


https://www.bbc.com/news/articles/cpe3zgznwjno
now also the Spammers are trying to take advantage :O



Thankfully at my work we have it only on servers which thankfully for a change isn't my responsibility so I was scot free :D



but I know of some ex colleagues that suffered hard

Re: IT outages are reported across the world
 

Apparently not: https://twitter.com/Perpetualmaniac/...76668095754753

Re: IT outages are reported across the world
 

That doesn't explain how it got released

Re: IT outages are reported across the world
 

I remember back in Windows 95 you could get programs to run by putting a link to the exe.

Someone did this to shutdown, so it immediately shutdown on booting

Yes well alot of peoples dont have good judgement anymore.....

Im glad it was fixed........ I dont think Ihave ever seen so many businesses networks not working like I did on friday!! (You dont realise how many are hooked up to the internet)

Re: IT outages are reported across the world
 

It doesnt really explain much at all.
A "Null Pointer Exception" is very common, and easily caused (usually some sort of programming error).

In this case, the contents of the C-00000291.sys file they tell you to delete caused a crash in the code using it (csagent).
This means the error appears to be the csagent code, not the deleted file - it should handle bad data without crashing, but it didnt.

[Anyone who knows much about windows may also wonder why a file that basically seems to be data has a .sys extension, it really should not].

Wow I did a search Paul,I have 103 files with the .sys extention.....

28 are duplicates of another one....

Re: IT outages are reported across the world
 

Too many of these 'outages' happening lately. Banks etc. Think we all know who to blame. Either we don't get an explanation or its blamed on things like a software update. Propaganda at work.

Re: IT outages are reported across the world
 

it's starmageden :shocked:

Re: IT outages are reported across the world
 

Ummm, Do enlighten us. :erm: