Thursday, 15 September 2016

The progress loophole of despair

Is it possible that there are certain situations where a school would benefit more from a pupil actually scoring less on a test? Having spent probably too much time playing with the KS2 pupil ready reckoner tool (click here and click 'how Ofsted and DfE analyse your data' to download), it would appear so. But this flies in the face of the core purpose of value added measures - that all progress counts - and the key statement in the DfE's Primary Accountability guidance: 'It is important that schools are held to account and given recognition for the progress made by all of their pupils'. So, how can we have a loophole where certain pupils are excluded, even if they've taken the test? And what are the implications for future assessment arrangements if this loophole remains open?

Having played with the pupil ready reckoner tool, I've found a number of interesting scenarios and they involve pupils assessed as HNM or PKS (pre-key stage, which includes BLW, PKF, PKE and PKG):

1) Pupils assessed as PKS who do not sit the test
These pupils obviously have no test score and are instead assigned a nominal score as defined in the accountability guidance: BLW = 70 points; PKF = 73 points, PKE = 76 points, PKG = 79 points. These points are for progress measures only and are not used in the average scores measure. These pupils will be counted in the denominator for the percentage threshold measures.

2) Pupils assessed as PKS who do sit the test but do not gain enough marks to achieve a scale score
Naively, I assumed that any pupil assessed as PKS would not sit the test because they were working below the level of the test, but that is certainly not the case. There are enough examples in the datasets I've seen to suggest it's actually quite common. In situations where 'PKS' pupils sat the test but did not achieve enough marks to be awarded a scale score, they are instead assigned the same nominal scores according to their PKS code as detailed above. In the ready reckoner tool, if 'no scale score awarded' is selected in the first box, users are prompted to 'Please enter a KS2 Teacher Assessment if the test outcome is B (working below the standard of the test) or if the pupil entered the test but did not achieve a scaled score'. PKS codes are acceptable values and the nominal score (i.e 70-79) is awarded in place of the scale score. Again, these points are for progress measures only and are not used in the average scores measure. These pupils will also be counted in the denominator for the percentage threshold measures.

3) Pupils assessed as PKS who sit the test and gain enough marks to achieve a scale score
This scenario  is quite simple: the pupil achieves a scale score, which supplants the nominal score assigned to the PKS code; and because the lowest scale score is 80, the pupil makes at least a single point gain in their individual progress score (the highest they can get if they don't score on the test is 79). However, unlike the nominal scores assigned to PKS codes, this score is also included in the average score, so whilst the school makes an almost negligible gain in terms of progress, its average scores are likely to take a big hit. These pupils are therefore included in progress measures, average scores and percentage threshold measures.

4) Pupils assessed as HNM who achieve a scale score
These pupils are not working below the level of test but are assessed as not having met the expected standard. These pupils take the test and the vast majority achieve a scale score - there is a wide range of scores for HNM pupils with many achieving low scores (i.e. below 90). These scores are used in the progress measures and average scores. This is all fairly normal.

5) Pupils assessed as HNM who do not gain enough marks to achieve a scale score
Here's the loophole. Unlike, PKS pupils, this group are not assigned a nominal score of any sort. When 'no scale score awarded' is selected in the pupil ready reckoner tool, and HNM is then selected in the teacher assessment box, as prompted, instead of the nominal scores awarded to PKS codes, we are instead confronted with a red box stating 'pupil excluded'. So a PKS pupil that sits the test but does not achieve enough marks to get a scale score is assigned a nominal score from 70-79, but HNM pupils receive no such score and are excluded. The impact of this, particularly on a small school's data can be enormous. Those nominal scores for PKS codes can have a major negative impact on progress measures, but if pupils were instead assessed as HNM and had not scored on the test, they'd have been excluded and the negative impact would disappear.

What concerns me is that schools may start to be strategic in their assessments if this loophole remains open. Yesterday I saw an example of a pupil that had been assessed as HNM, who gained just enough marks to achieve the lowest scale score of 80. The pupil ended up with individual progress scores of around -12, which has a profound impact on the school's overall progress measures. If the pupil had achieved just a few less marks, then no scale score or nominal score would have been awarded and the pupil would have been excluded from the progress measures. The -12 would disappear.

In another school, a PKE pupil had been entered for the test but failed to achieve a scaled score. In this case the nominal score of 76 is awarded, and again there is a big impact on VA due to the large negative deficit. If this pupil had been assessed as HNM then no nominal score would have been awarded, the pupil would be excluded from progress measures and there would be no major, negative impact on the school's progress measures.

The worst case scenario is that schools assess pupils as HNM rather than PKS and enter them for the test knowing/hoping they don't score, thus ensuring there is no negative impact. If they do score on the test then at least the score will be higher (i.e. 80+) than the nominal score they would be have been awarded for the PKS code (79 maximum). So, either they score and there is some benefit, or they don't score and there is an even greater benefit. I'm sure (I hope) this wouldn't happen but we currently have a ridiculous situation where schools could gain from entering pupils for a test they may not be able to access, award them an incorrect teacher assessment, and hope they don't score.

The simplest solution is to award a nominal score of 79 points to those 'HNM' pupils who take the test but do not achieve enough marks to be awarded a scale score. That would solve this particular problem but considering the big negative deficits associated with PKS nominal scores and the considerable impact these have schools' progress measures, there are still serious question marks over the whole methodology.

Monday, 12 September 2016

VA Calculator and Floor Standards tool: links to download

Warning: this blog post contains no opinions or rants whatsoever.

I recently shared my VA calculator and floor standard tool on Twitter and thought it'd be useful to put the links into a blog so people can find them easily.

PLEASE NOTE: the tools are stored in my One Drive so don't edit the online versions (think I set them to view only but just in case). Please download and save them to a local drive before getting to work, otherwise everyone will be able to see your data and i'll have to delete it. Also, it's likely that the links will be blocked on school PCs so you might need to access at home.

So, here are the links:


Click here to download the VA calculator

I've updated the VA calculator to include an Actual vs Estimated results page. You can now use this to get an idea of percentages 'expected' to achieve EXS and high standard for current cohorts. It will also calculate actual results once you have entered Y6 test scores and TAs; and compare actual to estimates.

Many people have asked me for the password for this spreadsheet, which is understandable. The spreadsheet is still locked but to save you the hassle of contacting me, the password is 'primary'. PLEASE don't unlock and edit the online version.

Note: new version calculated low/middle/high bands


Floor standards tool can be downloaded here

Please read the notes on the first tab of both tools for instructions and get in touch (@jpembroke) if you have any questions or comments.

Have fun!

Thursday, 1 September 2016

Update on primary floor standards and progress measures

To coincide with the performance tables checking exercise, the DfE have released an updated Primary Accountability guidance document and additional note on the new progress measures. These documents contain useful information on how progress measures are calculated, and provide more detail on floor standards.

Schools that have downloaded their checking exercise data will find their progress measures in the summary sheet and will note that there are three figures for each of reading, writing and maths, e.g. 2.5 (-4 to 5.6). The first figure (out of brackets) is the progress score: negative indicates that the cohort made less than average progress, 0 indicates average progress, and a positive score indicates more than average progress. The other two figures (in brackets) form the confidence interval, which dictates whether progress is significantly above or below, or in line with average.

How to tell if progress is significant or not
Take note of the confidence interval in brackets beside your progress score:
  • If the first figure in brackets (the lower part of the confidence interval) is positive then your progress is significantly above average. For example: 4.6 (1.7 to 7.4)
  • If the second figure in brackets (the upper part of the confidence interval) is negative then your progress is significantly below -3.2 (-5.9 to -0.5)
  • If the figures range from negative to positive (i.e. they straddle 0) then data is not statistically significant. Your progress score is either positive but not significantly so (e.g. 0.8 (-1.5 to 3.2); or negative but not significantly so (e.g. -1.3 (-2.8 to 0.3).
Floor standards
The updated accountability guidance defines the much anticipated thresholds for 'sufficient progress' as follows:

Reading: -5
Writing: -7
Maths: -5

If 65% or more pupils achieved the expected standard in reading, writing and maths then these sufficient progress thresholds do not come into play. That school is above floor. If fewer than 65% of pupils achieved the expected standard in reading, writing and maths then the school's progress scores will be compared against these thresholds. In this situation, the school needs to match or exceed all thresholds to be in the safe zone. However, there are certain circumstances where floor standards do not apply:
  • there are fewer than 11 eligible pupils at KS2
  • fewer than 50% of pupils have KS1 assessments that can be used to establish prior attainment groupings
  • There is insufficient KS2 attainment information because there are fewer than 6 pupils with results in a particular subject.
In addition, please note that if your school falls below the 65% EXS attainment threshold and only falls below one of the three 'sufficient progress' threshold then you would need to be significantly below average progress in that subject, not just below, which provides an extra cushion (see footnote on p6 of main guidance) although it is hard to see how any school in scope for floor standards could be anything but significantly below if they fall below the above thresholds. All this tweaking is no doubt to ensure that we end up with the promised 6% of schools below floor

As for coasting, we have no new information. The guidance states that the DfE 'plan to announce the 2016 progress thresholds, which will be key in determining whether a school meets the 2016 part of the coasting definition, in the autumn when we lay the coasting Regulations in Parliament. We will add this information into this guidance once the Regulations have been laid.' So, it looks like progress thresholds for coasting will be different to those used in the floor measures and will probably be tougher (higher).

Nominal scores for teacher assessments

The guidance provides detail of the nominal scores, which will be assigned to teacher assessments for purposes of progress measures. Nominal scores for writing are as follows:
  • Working towards the expected standard: 91
  • Working at the expected standard: 103
  • Working at greater depth: 113
and for those working below the standard of the test (pre-key stage):
  • Below standard of interim pre-key stage standards (BLW): 70
  • Foundations for the expected standard (PKF): 73
  • Early development of the expected standard: 76
  • Growing development of the expected standard: 79
There is no mention of those that sat the tests but failed to score (i.e. HNM). I assume there are very few such children, but it is an anomaly that needs to be cleared up. Let me know if I've missed something.

It is important to note that the above scores are for purposes of progress measures, to ensure that progress measures reflect the progress made by all pupils. The scores will not be reported or used elsewhere, including in the average scaled scores that will feature in RAISE and performance tables.

Prior Attainment Groups (PAGs)
For future reference, the DfE provide a useful lookup table, which provides KS2 estimates for each KS1 prior attainment group. There are 21 groups in total, ranging from pupils with KS1 APS between 0 and 2.5, up to the highest prior attainment group comprising pupils with KS1 APS of 21.5 or higher (i.e. at least one subject at KS1 was Level 4). Prior attainment group 12 (those with KS1 APS between 15 and 15.5 - the 2B'ers) have KS2 estimates for reading, writing and maths of 100.6, 100.7, and 101.5 respectively. These scores represent the 2016 average scores for pupils with the same start point nationally (i.e. the average scores for pupils in the same prior attainment group).

It is encouraging to see that the methodology now takes note of p-scales at KS1, rather than lumping all 'below level 1' pupils into one group and coding them as 'W' (working below). This welcome development results in much greater differentiation of prior attainment groups and will consequently lead to a fairer, more refined progress measure. From now on, pupils that were P6 at KS1 will not have the same 'expectations' for KS2 as a pupil that was P8, and that's a good thing.

Oh, and one more thing: national 'average' progress is always 0. Adding up all the +/- differences between actual and average score comparators will always result in 0. There is no longer any such thing as expected progress.

Pupils working at a higher standard
Quick mention of this measure, which we've been aware of for some time now (it was originally announced on a government legislation website, and more details came out in the first version of the Primary Accountability document, published in January). We now know the the high standard threshold is 110 for reading, maths and EGPS; whilst in writing it is obviously 'working at greater depth' (GDS). The DfE Statement of Intent shows that the performance tables (and RAISE) will show percentages achieving the high standard in each individual subject, but the headline measure is the combined one. This will involve those pupils that achieved 110+ score on the reading and maths tests and achieved GDS in writing (i.e. combined). EGPS scores are not part of the headline measure (this year at least). National and LA data can be found in the recently published SFR, by the way. 

That's enough for now. Still lots to get through so no doubt I'll be blogging again in the near future.