Retired Heidie explains how the Pisa system has been gamed

By Alasdair Macdonald

I remember in the first few years of PISA and looking at the samples of students in the countries which were ‘top of the league’. These countries presented no students who had additional needs. Others presented mainly children outwith the two lowest socioeconomic deciles.

With regard to children from Scotland, a number of the questions were on topics which were no longer in the curricula in Scottish education.

Much is known of the concomitant factors which lead to higher outcomes. For example present only girls for examination, or present no children from single parent families, present only children whose mothers have a number of Higher grades or university degrees, etc.

An example of gaming the system related to the ‘derived grade procedure’ which used to be applied to results after the papers had been marked but before grades had been determined by the examiners. When schools present children for examination, they have to provide a list of all candidates in rank order according to the evidence the school has, such as preliminary examinations. If the ranking order from the school matches closely the ranking order produced by the actual exams, the school is deemed ‘concordant’. So, if, for example, in a group of ten consecutive student 8 attain the expected grade, the other two are automatically awarded the expected grade even though their actual performance on exam day was lower. In the private sector because of selection by affluence they have large numbers of children being presented compared to a local authority school in an area of low socioeconomic status. Because the numbers are large, its rank order is deemed statistically significant and those within the group whose marks were lower are automatically given the same grade as their peers. But in the low socioeconomic group the sample size is small and is not deemed statistically significant, so pupils in the group whose exam marks are less than expected are not automatically upgraded and have to submit evidence which, in most cases is rejected.

For example Posh Academy presents 24 pupils expected gain an A pass and 18 do so on the first making then the other four are upgraded to A. At Ragged Secondary 8 are expected to get an A, and 6 do, the other two do not get upgraded automatically because the sample size is smaller. Evidence for appeal is submitted, and, in almost all cases is deemed insufficient. Having graded such exams I confirm that most evidence submitted whether from Posh Academy or Ragged Secondary IS insufficient. But, in most cases Posh Academy does not need to submit evidence. So, there is an incentive to place a few pupils who are unlikely to to get an A in with certain A passes, on the expectation of automatic upgrading. Thus, the affluent schools can game the system to inflate their pass rates.

This was made clear in the diet of 2000 or 2001, when the Scottish Exam Board made an arse of things (to use the technical term!) The Derived Grade Procedure was unable to be applied, so, there were no automatic upgrades and the posh school pass rates were pretty much the same as those for the local schools. Cue outrage by the right-wing press and BBC Scotland!


Discover more from Talking-up Scotland

Subscribe to get the latest posts sent to your email.

Leave a comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.