Monday, 6 June 2016

The ticking time bomb of KS2 VA

I guess you could call it 2020 vision. This issue of calculating KS2 VA in 2020 has been bugging me for some time, ever since I discovered that the DfE would not be collecting KS1 test scores this year, just the teacher assessments. I found this decision rather odd: why go to the bother of providing raw score to scaled score conversion tables if they are not going to use the scaled score? In fact, why bother with the tests at all if the teacher assessment is paramount? There are no thresholds for 'working towards' (WTS) or 'greater depth' (GDS) - just 'working at' (EXS) - so, in terms of 'informing' the teacher assessment, they are fairly limited, useless even. Surely there will be plenty of children scoring, say, 97 who meet all the 'working at' criteria set out in the interim assessment frameworks, so how does the score inform the judgement?

I digress. My main concern is the calculation of KS2 VA in 4 years time, for this current Year 2 cohort. I'm struggling to work out how the DfE can do it effectively, and if they do what I think they'll have to do, then the decision not to collect the KS1 scores this year seems even more bizarre. (I bet they change their mind next year, by the way).

So, let's get back to basics: what is value added? Hopefully everyone gets this by now but it's worth taking some time to nail it.

Value added involves comparing a pupil's attainment (in this case at KS2) against that of other pupils nationally with the same prior attainment (in this case at KS1). I've already blogged about how this will work over the next 4 years when pupils will have levels at one end and scaled scores at the other but, essentially, if a pupil achieves a KS2 score of 109 and their KS1 APS is 16, then we are interested in how that score of 109 compares against the national average KS2 score for pupils that had the same KS1 APS of 16.

This differs from Contextual Value Added (CVA), which involves comparing a pupil's KS2 attainment against that of other pupils with the same prior attainment and similar socio-economic characteristics in similar schools. CVA offers a more like-for-like comparison, which many would argue is  more fair. It certainly helps some schools whilst hammering others.

So value added is all about identifying and grouping pupils by their prior attainment; and pupils with the same prior attainment will have the same 'expectations' for KS2 (we should call them estimates or benchmarks really, not expectations). The more refined the grouping, the fairer (but more complex) the measure is. Now, historically, and for the next 4 years, prior attainment takes the form of levels at KS1, of which there are 6 options (7 if you include Level 4!) - W, 1, 2c, 2b, 2a, 3 - across three subjects (reading, writing and maths). There are 216 possible combinations of these 6 outcomes across the three subjects so that's 216 possible prior attainment groups (some are highly unlikely I admit). Now, the basis of KS2 VA is KS1 APS and of course many of the these combinations of levels at KS1 will produce the same APS (e.g. 2c, 2b, 2a and 2a, 2b, 2c both result in APS of 15). But, under the old methodology, which applied an adjustment based on relative differences in pupil's KS1 attainment in reading and maths, these pupils would be treated as different, and would therefore have different KS2 VA estimates. 

This year (i.e. for this current Y6) the methodology has changed - as I blogged about here - and now VA estimates are entirely based on KS1 APS, which means that all pupils with same KS1 APS, regardless of differences in individual subjects, will have the same estimates to reach at KS2. The change to the KS1 APS calculation, in which maths is double weighted, mitigates the issue outlined in the blog somewhat, but this change in methodology will result in far fewer prior attainment groups - 34 by my calculation (43 if you include L4 - rare at KS1). Whilst this is a substantial reduction on the 200+ possible outcomes previously, it does bring KS2 in line with the more complex progress 8 measure used at GCSE, which has 36 fine level prior attainment groups. 

So for the next 4 years, pupils' KS2 scores will be compared against the national average KS2 score for pupils in the same KS1 prior attainment group. A pupil with a KS1 APS of 16 will have a different KS2 estimate to a pupil with a KS1 APS of 16.5. Despite the change, and some concerns, the measure is still quite refined.

But what about 2020, when the current year 2 pupils reach KS2? What happens then? My previous concerns, regarding the change in VA methodology this year, pale alongside my concerns about accountability measures in 4 years time. Yes, it's some way off but I have two burning questions:

1) How will prior attainment groups be established for this cohort?

2) How many prior attainment groups will there be?

Going back to the start of this post, the decision not to collect KS1 scores this year strikes me as rather odd. As discussed above, current VA methodology requires pupils to be placed into prior attainment groups so their final attainment can be compared against pupils of supposedly similar ability. These prior attainment groups are based on KS1 APS and KS1 APS is the average of the scores in 3 subjects (or 2 now that they are combining reading and writing into an overall KS1 English score). Regardless of the calculation, it requires scores to calculate an average, and the DfE haven't collected them this year. So, in order to calculate an average score to generate the prior attainment groups will the DfE resort to assigning nominal scores to the KS1 teacher assessments, much like they are doing for writing at KS2 this year? Something like this perhaps:

BLW: 70
PKF: 80
WTS: 90
EXS: 100
GDS: 110

How else can we work out an average for reading, writing and maths on which to establish prior attainment groups? I can't add up EXS, EXS and WTS and divide by three! And if this is what's going to happen, why not just collect the scores in the first place?

The big worry is this will result in far fewer prior attainment groups than we currently have. Adopting the same KS1 APS calculation scheduled for use in the VA measure this year, the above scoring across three subjects results in 17 distinct APS outcomes - 17 prior attainment groups. Yes, I could tweak the above nominal scoring to get a few more options but quite frankly it's making my head hurt; and it would still result in a major reduction in the number of prior attainment groups.

The thing to bear in mind is that fewer groups means more pupils and greater variation within each group; and that all pupils within a particular prior attainment group will have the same VA estimates. So, let's fast forward to 2020 and imagine we have a year 6 pupil who was EXS, EXS, WTS in reading, writing and maths back at KS1. If VA continues as a measure, their scores in KS2 tests will be compared against the national average KS2 scores for EXS, EXS, WTS pupils. Blunt to say the least.

This will represent a steady erosion of the VA measure over a 10 year period, starting with the removal of CVA in 2010, with its near infinite number of prior attainment groups, to VA based on mean average KS1 APS with adjustments to account for differences in reading and maths (200+ groups), to the latest methodology involving a change in APS calculation and removal of reading and maths adjustments (34 likely groups), to 2020 based on broad teacher assessments (potentially fewer than 20 groups). So, in 4 years time, will VA be a fair and useful measure or just a very blunt instrument? Will we really be comparing like-for-like?

As we know, this year's teacher assessment arrangements are interim. Considering the problems these arrangements will cause in terms of progress calculations in 2020, I fully expect KS1 scores to be collected next year. Why they didn't do so this year is a mystery, and possibly a decision the DfE will end up regretting. 



If only they had 2020 vision.


No comments:

Post a Comment