[Sephiroth]

I've been digging into the way the algorithm was tested.

https://unherd.com/2020/08/how-ofqua...lgorithm-test/

The normal way to test a predictive algorithm is to see how good it is at predicting the past. That is, you run the program for the previous year and see how well its predictions match what happened in real life. OfQual did that for 2019, but because teachers in previous years were not asked to rank students, OfQual could not use 2019’s teacher-generated rank orders for a test run.

Instead, it used the rank order that emerged from the 2019 exam results. Which is like showing you can predict the results of a horse race by including data about the order in which the horses crossed the finish line in that same race. “If a test uses aspects of the same data that it is trying to predict, then it results in a false sense of security,” says Nason.

Even by including some of the data they were trying to predict, OfQual found their accuracy in predicting exact grades ranged from two thirds for History to one in four for Italian. For most non-language subjects, over nine in 10 students would be within one grade of the true result, but 3% of Maths students (for example) missing a fair result by two grades or more adds up to a lot of teenagers. Over 10% of Further Maths students, ironically the only ones who can understand the tortuous workings of the algorithm that betrayed them, would be over a grade away from a fair result.

Remember, the politicians called the algorithm "robust" when they either had no idea of its robustness or they were lying - most likely the former in which case someone must resign on grounds of incompetence.