Thursday, 31 May 2018

The changes ahead of us, the changes behind

Stop the world, I'm getting off

In the four years since the removal of levels primary schools have had to contend with a staggering amount of change to almost every aspect of the accountability system. No longer are pupils achieving Level 4 or Level 5; now they are described as meeting 'expected standards' or achieving a 'high score' (in a test) or 'working at greater depth' (in writing). Those that don't make the grade in a test have 'not met expected standards', whilst in writing they're defined as 'working towards the expected standard'. P-scales cling on (for now) but sitting above those - and below the main curriculum assessments described above - sit a series of ‘interim’ pre-key stage assessments including 'foundations for the expected standard' (KS1 and KS2), 'early development of the expected standard' (KS2 only), and 'growing development of the expected standard' (KS2 only). Key measures include percentages achieving expected standards and high score/greater depth (KS1 and KS2), and average scaled scores (KS2 only - the DfE didn't collect the KS1 test scores).

Progress measures have also changed. The old 'levels of progress' measures are obviously dead, but value added remains. Now sensibly 'zero-centred', the 'new' progress measures involve a smaller number of prior attainment groups derived from a KS1 APS baseline in which maths is somewhat controversially double-weighted. The number of prior attainment groups has already changed from 21 in 2016 to 24 in 2017 and may or may not do so again. We also have a complicated system of nominal scores which are used to calculate the progress of those pupils below the standard of tests at KS2, and these scores also changed between 2016 and 2017. And very soon we’ll run out of levels. How progress will be 
measured from 2020 onwards, when the first cohort without KS1 levels reach the end of KS2, is anyone's guess. It may well require nominal scores to be retrospectively assigned to the 'new' KS1 assessments.

The changes to progress measures also meant changes to floor and 'coasting' standards with value added thresholds replacing levels of progress medians, and a change to the rules so that now being below the attainment component and just one out of three progress thresholds spells trouble; previously schools would have to be below all four.

In the last four years primary schools have therefore had to cope with changes to the programmes of study, assessment frameworks, nomenclature, writing moderation, test scores, attainment and progress measures, coasting and floor standards; and of course, there was that failed attempt at implementing a reception baseline in 2015. It's a huge amount of upheaval in a short space of time.

And in the next four years, it's all set to change again.

2017/18
The main change this year involves the assessment of writing at KS1 and KS2, which becomes more 'flexible' having been a supposedly 'secure fit' over the last 2 years. I say supposedly, because it could be argued that that assessment and moderation of writing has been fairly flexible up to now anyway. Increased flexibility and discretion is welcome but is likely to lead to even more confusion.

Another important change this year is this capping of extreme negative progress scores, limiting them to the score of the bottom 1% for each prior attainment group. This should help mitigate some of the issues relating to negative outliers in the current progress system. 

2018/19
Things really kick off this year with some welcome and some not so welcome changes. First is the removal of statutory teacher assessment of reading and maths at KS2, a fairly pointless exercise where teachers state whether or not pupils have met expected standards only to have their assessment usurped by the test score. There are plenty of pupils assessed by teachers as having not met expected standards who then go and score 100 or more on the test, and vice versa. It's the test score that rules and so collecting the teacher assessment seems fairly pointless. 2019/19 marks the end of that process.

Also this year we'll see the removal of the interim pre-key stage assessments and the start of a phased withdrawal of p-scales (starting with P5-8), to be replaced with a new system of numerical standards.  

2019/20
This is probably the most controversial year of all with the rollout of the times tables check for year 4 pupils, and the start of a large scale voluntary pilot of the reception baseline assessment. There is concern about both assessments but it is the latter that is understandably getting most of the attention. It involves assessing pupils shortly after they start in reception in order to provide a baseline for future progress measures. The assessment designed by NFER will involve a series of activity-based, table-top tasks that will generate a standardised score; and this score will be used in much the same way as the KS1 scores in the current KS1-2 progress measure.

This also the year that our first cohort (current Year 4) of pupils with new KS1 assessment data reach the end of KS2, which means a new methodology probably involving a new system of yet to be invented KS1 point scores. Prepare to learn progress measures all over again.

And finally we have part 2 of the phased withdrawal of p-scales, with p1-4 being removed in this year. These apply to non-subject specific study and will be replaced by '7 aspects of engagement for cognition and learning'.

2020/21
Following the pilot of the reception baseline in September 2019, this year will see the full national rollout to all schools with a reception year. This first cohort of 'baseliners' will reach the end of KS2 in 2027 and until then it's business as usual (sort of) with progress measured from KS1 (somehow). 

This could also be the year we see changes to the Early Years Foundation Stage profile, with descriptors underpinning early learning goals (ELG), moderation arrangements, and statutory assessment processes all in scope for an overhaul. There is particular focus on revising 'the mathematics and literacy ELGs to ensure that they support children to develop the right building blocks for learning at key stage 1' (pages 5-11).

2021/22
I can't seem to find anything scheduled for this year. I must have missed something.

2022/23
The year we could be waving goodbye to statutory assessment at key stage 1, but only if the reception baseline gets off the ground (because we can't have cohorts without a baseline for progress measures). With every silver lining.........

2026/27
This the year the first cohort of reception baseliners reach the end of KS2, which means another revision of progress measures with new calculations, and new prior attainment groups to get your head round. Unless you work in a junior or middle school, in which case this is the year you've possibly been waiting for. The recent announcement by the DfE that they do not intend to measure the progress of pupils in non-all-through primary schools (i.e. infant, first, junior, and middle schools) from 2027, instead making these schools responsible for 'evidencing progress based on their own assessment information', is welcome, but it does beg the question: why not all schools? There is also the fact that infant and first schools will have a statutory responsibility for administering a baseline which they will have no real stake in. There are many questions to answer but 9 years is a long time in education.

Other changes
The Secretary of State recently announced that floor and coasting measures will be scrapped in favour of a single measure aimed at identifying schools in need of support. A consultation will be carried out on future measures but needless to say this change can't come soon enough.

That's it: a rundown of the main changes we will face over the next few years. No doubt I've missed something vital so please let me know. In the meantime, don't let the system get you down.

Friday, 4 May 2018

Test score upload and progress analysis in Progress Bank/Insight

Andrew Davey at Insight (@insightHQ; www.insighttracking.com) has been busy building a very neat, intuitive interface for the quick uploading of standardised test scores into Progress Bank and Insight, and analysis of the data.

As stated previously, the aim of Progress Bank is to provide schools with a simple, online system that will capture any standardised test data from any provider and measure progress between any two points. The sort of data that could be uploaded and analysed includes:
  • NFER tests
  • GL progress tests
  • CAT4
  • PUMA/PIRA
  • STAR Assessment
  • KS1 scaled scores
  • KS2 practice SATS results
  • KS2 actual SATS results
  • Reception baseline scores
Ultimately, we want to be able to build up enough data to enable the calculation of VA between any two points. This will involve a DfE-style calculation whereby pupils are placed into prior attainment groups based on a previous assessment, and their score on a following assessment is compared to the average score of pupils in the same prior attainment group. This could be from reception to KS1, or from KS1 to KS2, or from Y1 autumn to Y5 spring, or Y3 entry to Y6 SATS (useful for junior schools). In theory, if we get enough data, we can measure progress between any two points. The progress scores will be shown for pupils, key groups and cohorts, for reading and maths (and possibly SPaG if you are testing that too). By measuring progress using standardised tests, it is hoped schools will stop reinventing levels and use teacher assessment purely in the classroom, for formative purposes.

Until we reach the point where we have enough data to calculate VA, we will instead track changes in standardised scores or percentile rank for cohorts, groups and pupils (bearing in mind that standardised scores do not always go up, and no change is often fine). 

The system involves a three step process:
  1. Upload CTF (it is secure and GDPR compliant)
  2. Upload test scores
  3. Analyse data 
It is fairly quick to do. Once a CTF file has been uploaded, users can then upload test scores via a simple process that allows them to copy and paste data onto the screen.


Then paste the names and chose the order of surname and forename. This will enable the system to match pupils to those already in the system:



Then validate the data. Any pupils that don't match will be flagged and can be matched manually.


we can then select the assessment for which we want to upload scores for this particular cohort:


and add the scores on the next screen, again by copying and pasting from a spreadsheet:



That gets the data into the system (you can retrospectively upload data for previous years and terms by the way) and all we need to do now is analyse it. This is done via a simple pivot table tool within the system. The following screen shows summary of year 5's NFER tests scores for autumn and summer term broken down by key group. There are various options to select cohorts, start and end points, assessments, columns and rows values, and cell calculations. Note that the progress column currently shows change in standardised score, and the plan is to move that to a VA measure when enough data is available.

 
And finally, by clicking on a cell, we can drill down to pupil level; and by clicking on a progress cell we can access a clickable scatter plot, too.


Red dots indicate those pupils whose scores have dropped, and green dots show those whose scores have gone up. Clicking on the dots will identify the pupil, their previous and current test scores, and progress score between the two points selected. The colours are not intended to be a judgement, more an easy way to explore the data.

That's a quick tour of the Progress Bank concept, as it currently stands. The upload tool is already available to Insight users, and the pivot table report will be rolled out very soon. Progress Bank, featuring data upload, pivot tables and scatter plots, will be launched as a standalone tool in the Autumn term, for those schools that just want to capture and analyse their standardised scores without the full tracking functionality of Insight. It will therefore complement existing systems, and provide a quick and simple way of generating progress scores for Ofsted, governors and others.

Prices to be announced. 

More info and register your interest at www.progressbank.co.uk