Coronavirus (OLD) - Page 416

1andrew1 · 05-10-2020, 11:59

More info on those data issues.

Quote:

A technical glitch which meant almost 16,000 Covid-19 cases went unreported has been slammed as "shambolic" after it emerged that it led to delays in tracing contacts of people with coronavirus. The number of UK Covid cases surged by 22,961 on Sunday after it was revealed that thousands of cases were not included in daily reports due to some data files reporting positive test results exceeding the maximum file size.

https://www.msn.com/en-gb/news/uknew...rtan-dhp-feeds

Sephiroth · 05-10-2020, 14:09

Quote:

Originally Posted by Pierre

I wouldn't rush to it either. Vaccinate the at risk groups as a priority anyway. The vast majority of the population don't need the vaccine anyway.

My current thinking is that I wouldn't touch the vaccine.
I'm afraid of the auto-immune system effect. I get flu if I take the flu jab (so I've stopped doing that). I've covered this a few pages ago.

---------- Post added at 11:17 ---------- Previous post was at 11:08 ----------

Quote:

Originally Posted by 1andrew1

More info on those data issues.

https://www.msn.com/en-gb/news/uknew...rtan-dhp-feeds

Quote:

A technical glitch which meant almost 16,000 Covid-19 cases went unreported has been slammed as "shambolic" after it emerged that it led to delays in tracing contacts of people with coronavirus. The number of UK Covid cases surged by 22,961 on Sunday after it was revealed that thousands of cases were not included in daily reports due to some data files reporting positive test results exceeding the maximum file size.

How can a file exceed the maximum file size (oxymoron)? Unless the receiving software has a maximum file size constraint. And how would that happen? Unless they were on 16 bit software it would have to have been a deliberately introduced constraint.

Far more likely that the receiving software didn't have dynamic table sizes.

Obviously there is a design problem with the back-end software but if they don't employ me to validate all this, what can you expect!!

---------- Post added at 13:09 ---------- Previous post was at 11:17 ----------

Quote:

Originally Posted by Sephiroth

My current thinking is that I wouldn't touch the vaccine.
I'm afraid of the auto-immune system effect. I get flu if I take the flu jab (so I've stopped doing that). I've covered this a few pages ago.

---------- Post added at 11:17 ---------- Previous post was at 11:08 ----------

How can a file exceed the maximum file size (oxymoron)? Unless the receiving software has a maximum file size constraint. And how would that happen? Unless they were on 16 bit software it would have to have been a deliberately introduced constraint.

Far more likely that the receiving software didn't have dynamic table sizes.

Obviously there is a design problem with the back-end software but if they don't employ me to validate all this, what can you expect!!

Further to the above, the PM said on TV that the "data had been truncated".

This would mean that the back-end software did not mitigate the failure mode of a system file size limit, which is c. 8 PetaBytes in W10. However, I've seen hospitals in my local trust still using Vista where the maximum is 2TB (on 32 bits).

**Damien** · 05-10-2020, 14:18

Appranrtly they had the database in Excel. Not sure if that means they had some of interface on top of Excel or where literally just using Excel itself.

**Maggy** · 05-10-2020, 15:14

If only we could get a track and trace system going.

papa smurf · 05-10-2020, 15:35

Quote:

Originally Posted by Maggy

If only we could get a track and trace system going.

We have a fine tradition of making a complete hash of anything to do with data in this country.

1andrew1 · 05-10-2020, 15:48

Quote:

Originally Posted by papa smurf

We have a fine tradition of making a complete hash of anything to do with data in this country.

Which country might that be?

papa smurf · 05-10-2020, 15:50

Quote:

Originally Posted by 1andrew1

Which country might that be?

England mostly.

**Hugh** · 05-10-2020, 17:01

Quote:

Originally Posted by Damien

Appranrtly they had the database in Excel. Not sure if that means they had some of interface on top of Excel or where literally just using Excel itself.

What the whatting what?

Quote:

The problems are believed to have arisen when labs sent in their results using CSV files, which have no limits on size. But PHE then imported the results into Excel, where documents have a limit of just over a million lines.

The technical issue has now been resolved by splitting the Excel files into batches.

https://www.dailymail.co.uk/news/art...el-glitch.html

Glitch, my arse - who in (insert appropriate diety’s here) name uses a million+ line spreadsheet, besides complete amateurs? I spent most of the last 20 years getting home-grown systems off spreadsheets and migrated to ‘proper’ IT systems, because the spreadsheets tend to have no documentation, ownership, or resilience.

This is NOT an IT glitch. It’s using a spreadsheet to do the job of a database. It’s not fit for purpose. It’s trying to hammer a nail in with a roll of sandpaper; the tool doesn’t suit the job.

1andrew1 · 05-10-2020, 17:25

Quote:

Originally Posted by Hugh

What the whatting what?

https://www.dailymail.co.uk/news/art...el-glitch.html

Glitch, my arse - who in (insert appropriate diety’s here) name uses a million+ line spreadsheet, besides complete amateurs? I spent most of the last 20 years getting home-grown systems off spreadsheets and migrated to ‘proper’ IT systems, because the spreadsheets tend to have no documentation, ownership, or resilience.

This is NOT an IT glitch. It’s using a spreadsheet to do the job of a database. It’s not fit for purpose. It’s trying to hammer a nail in with a roll of sandpaper; the tool doesn’t suit the job.

It's more World-leading cut and paste than World-Leading Track and Trace.

nomadking · 05-10-2020, 19:00

It would've had to be a spreadsheet rather than a database, simply because people are much more familiar with spreadsheets than databases.

MS Access isn't a standard part of MS Office.

Sounds like the data was originally in a database, but had to be sent out in a more user friendly format.

**Damien** · 05-10-2020, 19:16

Quote:

Originally Posted by nomadking

It would've had to be a spreadsheet rather than a database, simply because people are much more familiar with spreadsheets than databases.

MS Access isn't a standard part of MS Office.

Sounds like the data was originally in a database, but had to be sent out in a more user friendly format.

I would argue MS Access is also not up to the job but Excel certainly isn't.

There are other ways to present data including exporting into spreadsheets from a database. It sounds like here the master version of the data was in Excel and was being appended too via CSV files.

nomadking · 05-10-2020, 19:43

Quote:

Originally Posted by Damien

I would argue MS Access is also not up to the job but Excel certainly isn't.

There are other ways to present data including exporting into spreadsheets from a database. It sounds like here the master version of the data was in Excel and was being appended too via CSV files.

That apparently wasn't the problem. It wasn't a case of more than 1m rows, but more than 64K rows or 1400 cases.

Link

Quote:

The issue was caused by the way the agency brought together logs produced by the commercial firms paid to carry out swab tests for the virus.
They filed their results in the form of text-based lists, without issue.
PHE had set up an automatic process to pull this data together into Excel templates so that it could then be uploaded to a central system and made available to the NHS Test and Trace team as well as other government computer dashboards.
The problem is that the PHE developers picked an old file format to do this - known as XLS.
As a consequence, each template could handle only about 65,000 rows of data rather than the one million-plus rows that Excel is actually capable of.
And since each test result created several rows of data, in practice it meant that each template was limited to about 1,400 cases. When that total was reached, further cases were simply left off.
Until last week, there were not enough test results being generated by private labs for this to have been a problem - PHE is confident that test results were not previously missed because of this issue.

downquark1 · 05-10-2020, 19:45

Dominic Cummings is particularly keen on filling the civil service with physicists. I can tell you mistakes like that would never happen. Although spelling mistakes and delays may go up.

nomadking · 05-10-2020, 19:49

Quote:

Originally Posted by downquark1

Dominic Cummings is particularly keen on filling the civil service with physicists. I can tell you mistakes like that would never happen. Although spelling mistakes and delays may go up.

Usual nonsense. As if he would've had any say in hiring developers for PHE.

They were possibly hired before he came onto the "scene".

Too much hiring on the basis of bits of paper, rather than core underlying ability.

downquark1 · 05-10-2020, 19:52

Quote:

Originally Posted by nomadking

Usual nonsense. As if he would've had any say in hiring developers for PHE.

They were possibly hired before he came onto the "scene".

Too much hiring on the basis of bits of paper, rather than core underlying ability.

I didn't say he did anything. I am saying what he would have done if he was in charge.