Pisa – 33% of Scotland’s schools took part but only 4.7% of England’s schools were confident enough to do so – Scotland did far better

I’m grateful to AR and Haggis Hunter https://twitter.com/thistlefarmer5 for alerting me to this.

The England PISA sample consisted of 201 eligible schools having at least one pupil in this age range. In England, 3,852 pupils from original sample schools and 911 pupils from replacement schools participated. Pupils in participating schools that did not participate are not replaced. 159 agreed to participate, along with a further 32 replacement schools, but 16 schools withdrew before data collection. Data was therefore collected from 143 schools in the original sample and 32 replacement schools. Of this total, nine original sample schools and one replacement school were omitted from the response rate adjudication process, leaving a total of 134 schools from the original sample and 31 replacement schools.

chrome-extension://efaidnbmnnnibpcajpcglclefindmkaj/https://assets.publishing.service.gov.uk/media/656dc3321104cf0013fa742f/PISA_2022_England_National_Report.pdf

So, in England, in the end, 165 schools and 4 763 pupils took part.

The Scottish participation, from only a tenth of the pupil population and schools, was:

117 secondary schools participated in the survey, representing a school response rate of 96.4 per cent; exceeding the OECD’s minimum standard of 85 per cent. 3,257 students took part giving a weighted student participation rate of 79.4 per cent, slightly below the student participation technical standard of 80 per cent. 

https://www.gov.scot/publications/programme-international-student-assessment-pisa-2022-highlights-scotlands-results/

There are 357 secondary schools overall in Scotland so a sample of 32.8% was used.

Only 4 763 pupils in 165 schools took part in England. There are 3 458 secondary schools in England so only 4.7% of them took part. Were they then representative or just maybe, the best performing?

There are roughly 10 times as many 15 year-olds in England yet the sample-size was less than twice.

The Pisa results in England and Scotland are simply not comparable and had the English sample of schools been similarly representative and not so suspiciously selective, we can only guess that Scotland would have performed far better than England.

19 thoughts on “Pisa – 33% of Scotland’s schools took part but only 4.7% of England’s schools were confident enough to do so – Scotland did far better”

  1. Hey where’s that flunky from Dundee Uni who usually pops up on a full moon to criticise Scottish schools Scottish children Scottish people anything Scottish really including our PISA results , actually he’s probably sleeping

    Liked by 1 person

    Reply

  2. The unionist media are not interested in your statistical shenanigans. They have got data which shows ‘Scotland bad’ and that is their raison d’etre.

    Liked by 4 people

    Reply

  3. Hi John, there is an extension attached to your link to the PISA report for England and I have removed it to make it easier for people to download. 👍

    PISA 2022 – National Report for England

    Page 8, Executive Summary, Introduction (the pdf has 229 pages!)

    Click to access PISA_2022_England_National_Report.pdf

    “When reading this report, it is important to keep in mind that England’s sample of
    participating pupils may not be entirely representative of all 15-year-old pupils in England. This is, to some degree, always the case with international studies such as PISA, but in this case the sample for England did not meet 2 of the 82 PISA Technical Standards. Analysis of the characteristics of the pupils who participated revealed that the final sample had somewhat higher academic attainment on average than the general population and a somewhat lower proportion of pupils who had been eligible for free school meals in the past 6 years. In other words, higher performing pupils may be overrepresented in the final sample and some of the PISA results may therefore be somewhat higher than they might otherwise be. This issue was also a challenge for some of the other participating education systems including several OECD countries. Given that the sample may not be entirely representative of the population, caution is required when interpreting the analysis that is presented in this report, though this does not necessarily translate directly to a particular score being a certain number of points higher than its ‘true’ value, and the OECD itself makes no adjustments to the scores in any education system in which some of the PISA’s Technical Standards were not met.”

    Liked by 6 people

    Reply

      1. Worth sharing..

        https://schoolsweek.co.uk/pisa-2022-rise-in-maths-but-warning-over-inflated-results/

        “England’s secondary school pupils have risen up the international rankings for their attainment in maths, but there has been little comparable improvement in reading and science.

        And this year’s landmark Programme for International Student Assessment (PISA) results are also likely to overinflate scores for England as too few schools took part in the study.”

        continues

        “Too few students from England took part in the study – meaning the results could be up to eight points too generous, as more higher-performing pupils took part.

        This big caveat led to academics urging caution on interpreting the findings.”

        Liked by 1 person

        Reply

        1. Are the international TIMMS maths exercises still happening or have they been subsumed into PISA ?
          Was amused some years ago when Cameron appointed Carol Vorderman as his maths expert, to study attainments in different countries. Ms V prompty found (2014) that France didn’t take part in TIMMS.
          On a last flippant note, the only time I ever looked at a set of TIMMS results, the German students scored 1.1% more on the TIMMS tests and were on average 1 year, 1 month older than the Scots. Appealed to my simple innumerate mind.

          Like

          Reply

    2. What interests me also is that if 73 countries are now taking part, there is a possibility that the %age includes countries with SLA lower than 15, eg India (14 since 2010) so that 15 year olds still in school are likely to be unrepresentative in terms of ability and/or motivation.

      Like

      Reply

  4. This goes to show that “the union” is unequal and completely corrupt.
    The media are complicit, always have been and always will be.
    The Ununited Kingdom is a sham!!!
    Vote Yes and whether you like it or not the SNP are thee biggest party in Scotland
    therefore the only one’s currently able to achieve Independence for this rich
    country………everyone needs to do there bit to get the truth out!!!

    Liked by 2 people

    Reply

  5. I agree anonymous
    Vote SNP for Scottish independence because at present there really is no other viable option
    Even if you don’t like SNP for whatever reason there is clearly going to be self destruct if you vote for anyone else or don’t vote so don’t shoot yourself in the foot put aside your policy anger aside the only thing that counts , is independence or even the chance of independence

    Liked by 4 people

    Reply

  6. As a school teacher (long since retired) I was involved in this annual “testing” programme. Our sample size was largely taken from S2 pupils so most were only 14 years old or had just turned 15.

    Liked by 1 person

    Reply

  7. Now, I have no real wish to give more profile to the OECD’s latest PISA (Program for International Student Assessment) results. There are many reasons to ‘moderate’ the importance accorded to PISA as an input to educational policy evaluation and design (search TuS with the term ‘PISA” for more on this) – an importance hyped especially it seems in Scotland by political opponents of the governing party and by their allies in an oppositional news media.

    However, regular TuS readers will be well aware of the evidence of Labour Party politicians in Scotland who are relentless in their negativity towards our public services. They will be aware of the negativity even when performance in Scotland is substantially better than in Wales where their Labour Party colleagues are in government,

    So in short – and with no particular wish to have a pop at Mark Drakeford’s Welsh Government – this is too much to resist. It’s an article by the data analysis company, Statista (6 December 2023) of the PISA results for Wales entitled: ‘Welsh Teens Left Behind as UK Education Gap Widens’

    ‘Welsh students are falling further behind their peers in the United Kingdom at school, according to the newly released 2022 results of the Program for International Student Assessment (PISA) by the Organization for Economic Co-operation and Development (OECD).’

    ‘The national averages for performance across maths, science and reading were lower in Wales than in England, Scotland and Northern Ireland, highlighting a regional education gap that persists in the UK.

    ‘Wales was also the only country of the four to drop below the OECD averages, which were calculated based on the test results of 690,000 students across 81 countries and economies. And it seems that this gap is getting wider: Where the national scores dropped in all three core subjects across the four UK regions, the greatest declines for each subject were in Wales.’

    ‘Students performed the worst in maths in each UK country in 2022 and it was also the subject with the greatest fall in attainment per country since 2018. This mirrors a global trend, as the OECD nations analysed showed an average decline of 15 points in maths since 2018.’

    ‘.. since the last PISA exams in 2018, students in Wales performed on average as if they were a full year behind in maths, just under a year behind in reading and three-quarters of a year behind in science.’

    Message to Labour in Scotland? Governing is difficult, ask Mr Drakeford and his ministers!

    And as to reasons for the PISA results in Wales? ‘Plaid Cymru’s education spokesperson, Heledd Fychan MS, has cited youth poverty rates leading to high pupil absenteeism, school budget deficits and a recruitment crisis in the education sector among the reasons for the lower results. “Despite the hard work and dedication of an overstretched workforce, the pupil attainment gap is widening and we cannot ignore the link between poverty and [Tuesday’s] disappointing results”, she said.’

    However, there is a wider context: ‘In the OECD report, other reasons for the observed declines across many OECD countries include interruptions from school closures during the Covid-19 pandemic as well as the longer-term issues of teaching quality and lack of resources.’

    We also learn from the same article: ‘Since the 2022 PISA tests were taken, Wales has already started rolling out a new Curriculum for Wales.’ By the way, for this this new curriculum the Labour-led Welsh Government has chosen to follow the principles of Scotland’s Curriculum for Excellence.

    Unsurprisingly, a BBC Wales news article online has Lindsay Paterson, professor emeritus of education policy at Edinburgh University, stating: ‘the decline in Scottish performance corresponds with the implementation of the Curriculum for Excellence schools model.’ One day the BBC will have a eureka moment and discover that Scotland has more than one academic with an interest in curricular design … or maybe not!

    Liked by 3 people

    Reply

    1. On PISA results for Wales, the National Education Union in Wales (NEU Cymru) issued a statement on 5 December which included a cautionary remark:

      ‘Comparison is difficult. We are particularly concerned that these results do not represent the full cohort of learners in many of the countries represented. Whilst this is understandable, in the context of the pandemic, it makes comparisons less meaningful.

      “Here in Wales, we have the new Curriculum for Wales, which none of the cohort tested in PISA have been taking. If the WG wants to take anything from these results, it is an opportunity to ensure that they have a qualifications system which does not focus on tests and time-limited exams but gives young people a real chance to show what they can do.’

      And in Scotland, there is a statement from the EIS: “This PISA test data, although limited in what it can say about the quality of learning within any education system, telling only a fraction of the story, reflects aspects of the hard of work and professional commitment of teachers amidst the most challenging of circumstances in the wake of the pandemic and continuing social inequality.

      “Across the OECD, average scores in reading, maths and science are lower, this pointing to the detrimental impact of the Covid pandemic on learning and the shortcomings of governmental responses to education recovery.

      “Scotland’s schools and teachers continue to strive to provide the best possible learning experiences for all young people but increasingly they are swimming against a tide of cuts, which threatens now to be a tidal wave UNLESS THE SCOTTISH GOVERNMENT MAKES DIFFERENT DECISIONS ON SPENDING AND REVENUE RAISING TO PROPERLY FUND EDUCATION.” (With my emphasis)

      Anyone found a ‘magic money tree’ growing in Scotland yet?

      Returning to Labour-run Wales, this from a BBC Wales article on education prompted by the PISA 2022 results:

      Source: BBC News website ‘Pisa: Welsh curriculum not full answer to improve school tests – minister’. (https://www.bbc.co.uk/news/uk-wales-politics-67636205 )

      Jeremy Miles, Labour’s education minister in Cardiff says that school “budgets are under significant pressure” and an “INCREASED LEVEL OF PUBLIC SPENDING RIGHT ACROSS THE UK” WAS NEEDED TO HELP RAISE PERFORMANCE. The BBC article follows his remarks immediately with this: ‘On Monday, Sir Keir Starmer said anyone expecting an incoming UK Labour government “to quickly turn on the spending taps” would be disappointed.’

      Liked by 2 people

      Reply

    2. The BBC is now under the control of Starwar…..it comes in cycles, Labour Tory Labour Tory etc…., I have seen this over too many years to mention……self preservation of the licence fee…….I can only presume that Starwar has had a back room meeting with them and promised they can continue to rip off Scotland.

      Like

      Reply
  8. Pingback: A better metric for judging a nation’s education system – Talking-up Scotland

Leave a comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.