Published in The Australian, Tuesday 15 March 2011
The new version of My School is a step in the right direction as it includes some information about student progress, but more changes are required.
The method of calculating school performance indicators has not changed. It is still prone to errors and bias against schools in poorer neighbourhoods. This should be replaced by value-added measures that calculate the contribution schools make to student progress.
My School is an important step forward: it increases transparency and the information available to families, schools and policy-makers, facilitating key objectives such as school improvement, accountability and choice. But the benefits are restricted if the measures of school performance are inaccurate. Principals and teachers can’t improve schools based on incorrect measures, schools should not be held to account on biased performance indicators and families can’t make informed decisions about which schools to choose for their children.
Performance is measured on My School by comparing the average score of each school’s students in the National Assessment Program – Literacy and Numeracy tests. Comparisons are also made between schools in “like-school groups”, which are based on a ranking that is supposed to reflect the extent to which students’ socioeconomic background influences their performance. This is an improvement on focusing only on raw test scores. It is also a method that has consistently been shown to incorrectly label schools as over or underperforming.
The problems stem from the difficulty in measuring the impact of students’ background. Whenever you try to measure school performance, you are trying to isolate the impact of the school from all the other factors that influence student achievement. This is not easy, as it is not all about money. In fact, the parents’ level of education and occupations are often more important, and a variety of other factors – such as linguistic background, the amount of home reading and numerous aspects of their environment – influence children’s education. These are all very difficult to measure.
As we have seen in recent months, these difficulties are magnified when the data used to create like- school groups is incomplete or distorted. Poorer schools are erroneously placed in the same group as wealthier schools, increasing the likelihood they will be labelled incorrectly as low performing. Hence, the reliance on like-school groups to measure performance increases the risks of producing biased results, particularly for schools in poorer suburbs and regional areas.
For example, consider a school in Melbourne labelled on My School as underperforming in Year 9 numeracy. The average Year 9 numeracy score in the school was 618, compared with an average of 651 in the like-school group. However, the school had made above-average progress in the group. Students’ numeracy scores had increased, on average, by 58 points, compared with 44 points in the like-school group.
In this school, the performance of students in their first year of secondary school was 47 points below the average in the like-school group. Schools should not be considered similar if they have such large differences in student performance in the first year of secondary school. This is why this school has been labelled as underperforming despite making more progress with its students.
This would not happen with value-added measures of school performance, a method that compares only students who have the same initial level of performance.
Value-added measures emphasise students’ progress rather than their background. The data and methodology have been shown to be fairer and more reliable.
Research in numerous countries has proved the greater accuracy of value-added measures. That is why teachers, school associations and education unions in other countries support their introduction. This was also recommended by the recent Senate inquiry into NAPLAN administration and reporting.
It is unfortunate the new version of My School has not taken this approach. It is equally important to know students’ literacy and numeracy levels, and the progress they are making. My School should publish both, but schools’ performance should be judged on the contribution they make to students’ progress. It will make My School more effective, end disagreements about rankings in like-school groups and increase accuracy as well as fairness to schools.