Comparing Schools Essay

This report provides advice on the collection and reporting of information about the performances of Australian schools. The focus is on the collection of nationally comparable data. Two purposes are envisaged: use by education authorities and governments to monitor school performances and, in particular, to identify schools that are performing unusually well or unusually poorly given their circumstances; and use by parents/caregivers and the public to make informed judgements about, and meaningful comparisons of, schools and their offerings. Our advice is based on a review of recent Australian and international research and experience in reporting on the performances of schools. This is an area of educational practice in which there have been many recent developments, much debate and a growing body of relevant research. Our work is framed by recent agreements of the Council of Australian Governments (COAG), in particular, at its meeting on 29 November 2008: C OAG agreed that the new Australian Curriculum, Assessment and Reporting Authority will be supplied with the information necessary to enable it to publish relevant, nationally-comparable information on all schools to support accountability, school evaluation, collaborative policy development and resource allocation. The Authority will provide the public with information on each school in Australia that includes data on each school’s performance, including national testing results and school attainment rates, the indicators relevant to the needs of the student population and the school’s capacity including the numbers and qualifications of its teaching staff and its resources. The publication of this information will allow comparison of like schools (that is, schools with similar student populations across the nation) and comparison of a school with other schools in their local community. (COAG Meeting Outcomes) Our work also has been framed by the recently endorsed MCEETYA Principles for Reporting Information on Schooling (see Section 1. 4). Before summarising our specific recommendations, there are some general conclusions that we have reached from our review of international research and experience. The specific recommendations that follow are best understood in the context of these general conclusions: †¢ Vigilance is required to ensure that nationally comparable data on individual schools does not have the unintended consequence of focusing attention on some aspects of the purposes of schooling at the expense of other outcomes that are as important but not as easily measurable. Parents/caregivers and the public are interested in a broad range of information about schools, and nationally comparable data should be reported in the context of this broader information. †¢ Although it has become popular in education systems in some other parts of the world to use statistical models to develop ‘measures’ of school performance and to report these measures publicly in league tables, we believe that there are very v Reporting and Comparing School Performances  sound technical and educational reasons why school measures of this kind should not be used for public reporting and school comparisons. †¢ Related to this point, we are not convinced of the value of reporting ‘adjusted’ measures of student outcomes publicly. Measures of student outcomes should be reported without adjustment. †¢ To enable the comparison of unadjusted student outcomes across schools, we believe that a ‘like-schools’ methodology should be used. This methodology would allow parents/caregivers, the public, and education systems to compare outcomes for schools in similar circumstances. †¢ While point-in-time measures of student outcomes often are useful, it is difficult to establish the contributions that teachers and schools make to point-in-time outcomes. In general, measures of student gain/growth across the years of school provide a more useful basis for making judgements about the value that schools are adding. †¢ Measures of gain/growth are most appropriately based on measurement scales that can be used to monitor student progress across the years of school. The NAPLAN measurement scales are an example and provide educational data superior to that available in most other countries. Consideration should be given to developing national measurement scales for early literacy learning and in some subjects of the national curriculum. †¢ Initially reporting should build on the understandings that parents and the public have already developed. For example a school’s NAPLAN results should be reported in forms that are consistent with current NAPLAN reports for students. Although much work needs to be done in defining the most appropriate measures, the principle should be to build on the representations of data that are already familiar to people. Recommendations Our report makes the following specific recommendations: student outcome measures †¢ Nationally comparable data should be collected on the literacy and numeracy skills of students in each school, using NAPLAN (Years 3, 5, 7 and 9). †¢ Nationally comparable data should be collected on the tertiary entrance results of students in each senior secondary school. These data could be reported as the percentage of students achieving tertiary entrance ranks of 60 or above, 70 or above, 80 or above, and 90 or above (calculated as a percentage of the students achieving tertiary entrance ranks). †¢ Nationally comparable data should be collected on the percentage of students in each senior secondary school completing Year 12 or equivalent; the percentage of students applying to all forms of post-school education; and the percentage of students completing VET studies. vi Reporting and Comparing School Performances †¢ Nationally comparable data should be collected on the achievements of students in core national curriculum subjects (English, mathematics, science and history), beginning in 2010. National assessments could be developed initially at Year 10. †¢ Nationally comparable data should be collected on the early literacy learning of children in each primary school. These assessments will need to be developed and should be administered upon entry to school and used as a baseline for monitoring progress across the first few years of school. physical and human resources †¢ Nationally comparable data should be collected about sources and amounts of funding received by each school, including all income to the school from State and Commonwealth governments, as well as details of fees payable by parents, including those that are mandatory and any voluntary levies that parents are expected to pay. †¢ Nationally comparable data should be collected on the numbers and qualifications of teaching staff in each school. Basic data would include academic qualifications, details of pre-service teacher education, and details of any advanced certification (eg, Advanced Skills Teacher; Level 3 Teacher). student intake characteristics †¢ Nationally comparable data should be collected on the socio-economic backgrounds of students in each school. Data should be based on information collected at the individual student level, using at least parental occupation and, possibly, parental education levels, under the agreed MCEETYA definitions. †¢ Nationally comparable data should be collected on the percentage of students in each school of Aboriginal and/or Torres Strait Islander background under the agreed MCEETYA definition. †¢ Nationally comparable data should be collected on the percentage of students in each school identified as having a language background other than English (LBOTE) under the agreed MCEETYA definition. †¢ Nationally comparable data should be collected on the geo-location of each school using a 3-category scale: metropolitan, provincial, and remote. †¢ Nationally comparable data should be collected on the percentage of students in each school with special educational needs. A nationally agreed definition of this category will need to be developed. like-school comparisons †¢ In reporting student outcome data for a school, data for like-schools should be provided as a point of comparison. Like-schools will be schools in similar circumstances and facing similar challenges. †¢ In determining ‘like-schools’, account should be taken of the percentage of students with Indigenous backgrounds, the socio-economic backgrounds of the students in the school, and the percentage of students from language backgrounds other than English. vii Reporting and Comparing School Performances †¢ For each school separately, like-schools should be identified as the schools most similar to that school on the above characteristics (rather than pre-defining a limited number of like-school categories). †¢ Work should commence as soon as possible on the development of an appropriate like-schools methodology. public reporting †¢ For the purpose of providing public information about schools, a common national website should be used to provide parents/caregivers and the public with access to rich information about individual schools. †¢ The national website should provide information about each school’s programs, philosophies, values and purposes, provided by the school itself, as well as nationally comparable data, provided centrally. †¢ Nationally comparable student outcome data should, wherever possible, provide information about current levels of attainment (ie, status), gain/growth across the years of school, and improvement in a school over time. †¢ The complete database for each state/territory should be made available to the relevant state/territory departments of education and other employing authorities, enabling them to interrogate data for their schools and to make judgments about school performances using aggregated data and national summary statistics. We believe that almost all nationally comparable data collected centrally could be reported publicly. The exceptions would arise when the public reporting of data may have negative and unintended consequences for schools. For example, we can envisage negative consequences arising from the reporting of the socio-economic backgrounds of students in a school, or of the financial circumstances of struggling, small schools (both government and non-government). We also believe that data reported publicly should be factual data about a school, and not the results of secondary analyses and interpretations that are open to debate (eg, value-added measures). viii Reporting and Comparing School Performances 1. INTRODUCTION In education, good decision making is facilitated by access to relevant, reliable and timely information. Dependable information is required at all levels of educational decision making to identify areas of deficiency and special need, to monitor progress towards goals, to evaluate the effectiveness of special interventions and initiatives, and to make decisions in the best interests of individual learners. The focus of this  paper is on the provision and use of information about individual schools. The starting point is the observation that relevant and reliable information about schools is required by a range of decision makers – including parents and caregivers, school principals and school leadership teams, system managers and governments, and the general public – all of whom require dependable information that they can use to maximise opportunities and outcomes for students. 1. 1 Audiences and Purposes  Parents and caregivers require valid and reliable information to evaluate the quality of the education their children are receiving, to make informed decisions in the best interests of individual students, and to become active partners in their children’s learning. They require dependable information about the progress individuals have made (the knowledge, skills and understandings developed through instruction), about teachers’ plans for future learning, and about what they can do to assist. There is also considerable evidence that parents and caregivers want information about how their children are performing in comparison with other children of the same age. And, if they are to make judgements about the quality of the education their children are receiving, they require information that enables meaningful comparisons across schools. School leaders require reliable information on student and school performances for effective school management. Research into factors underpinning school  effectiveness highlights the importance of the school leader’s role in establishing an environment in which student learning is accorded a central focus, and goals for improved performance are developed collaboratively by staff with a commitment to achieving them. School managers require dependable pictures of how well students in a school are performing, both with respect to school goals for improvement and with respect to past achievements and achievements in other, comparable schools. Governments and system managers require dependable information on the performance and progress of individual schools if they are to exercise their responsibilities for the delivery of quality education to all students. Effective management depends on an ability to monitor system-wide and school performances over time, to gauge the effectiveness of special programs and targeted resource allocations, to monitor the impact of policies, and to evaluate the success of initiatives aimed at traditionally disadvantaged and underachieving sections of the student population. Accurate, reliable information allows system managers to measure progress against past performances, to identify schools and issues requiring special attention, to target resources appropriately, and to set goals for future improvement. 1 Reporting and Comparing School Performances 1. 2 Forms of Information Because there are multiple audiences and purposes for information about schools, the forms of information required for effective decision making are different for different stakeholders. Parents and caregivers require a wide range of information, including information relating to their immediate needs (eg, Is the school easily accessible by public transport? Does it have an after-school program? What fees and/or levies does it charge? ); the ethos of the school (eg, What evidence is there of bullying/harassment? What are the espoused values of the school? Do students wear uniforms? What level of discipline is imposed? Who is the principal? ); their child’s likely educational experience (eg, Who will be my child’s teacher next year? Will they be in a composite class? How large will the class be? Does the school have a literacy intervention program? What extra-curricular activities are provided? ); and the school’s educational results (eg, Does the school achieve outstanding Year 12 results? ). School leaders require other forms of information, including information relating to staffing and resources (eg, What resources are available for music next year? How many beginning children have special learning needs? ); the effectiveness of initiatives (eg, Is there any evidence that the extra class time allocated to literacy this year made a difference?); and academic results (eg, How many Year 5 students did not meet the minimum performance standard in Reading? Have our results improved since last year? Are we still below the state average? How did last year’s Year 12 results compare with those of the neighbouring school? ). System managers and governments require still other forms of information, including information to monitor system-wide trends over time, to evaluate the effectiveness of attempts to raise standards and close gaps, and to identify schools that are performing unusually well or unusually poorly given their circumstances. In general, the schoollevel information required by system managers and governments is less fine-grained than the information required by parents, teachers and school leaders. Figure 1 displays schematically various forms of information that could be made available about a school, either publicly or to specific audiences (eg, system managers). The forms of evidence represented in Figure 1 are: A: student outcome measures that a school could choose to report Most schools report a wide range of information about the achievements of their students to their school communities. This information is reported in school newsletters, local and community newspapers, school websites, and at school events. The information includes details of Year 12 results, analyses of postschool destinations, results in national mathematics and science competitions, language certificates, awards, prizes, extra-curricular achievements, community recognition, and so on. Most schools take every opportunity to celebrate the achievements of their students and to announce these achievements publicly. 2 Reporting and Comparing School Performances Figure 1. Forms of information that could be made available about a school B:a sub-set of student outcome measures on which it is agreed to collect nationally comparable data Within the set of student outcome information that might be reported for a school, there could be a sub-set of outcomes on which it was agreed to collect nationally comparable data. A reason for identifying such a sub-set would be to ensure some common measures to facilitate school comparisons – within a local geographical area, across an entire education system, nationally, or within a group of ‘like’ schools. Inevitably, nationally comparable data would be collected for only some of the outcomes that schools, parents and communities value. Performances on common literacy and numeracy tests in Years 3, 5, 7 and 9 are an example of nationally comparable data currently in this category. C. physical and human resources measures that a school could choose to report Schools provide information in various forms and to various audiences about their physical and human resources. Information of this kind includes details of staff qualifications and teaching experience, staff turnover rates, school global budgets, computers and other technology, newly constructed facilities, bequests, results of fundraising drives, and so on. Some of this information may be reported to the school community; some may be kept confidential to the school, education system or government departments. D: a sub-set of physical and human resources measures on which it is agreed to collect nationally comparable data Within the set of physical and human resources measures reported for a school, there could be a sub-set of measures on which it was agreed to collect nationally comparable data. For example, there have been recent calls for greater consistency and transparency in the reporting of school funding arrangements (Dowling, 2007; 2008) and for more consistent national approaches to assessing and recognising teacher quality (Dinham, et al, 2008). 3 Reporting and Comparing School Performances E. student intake measures that a school could choose to report Most schools have considerable information about their students. For example, they may have information about students’ language backgrounds, Indigenous status, socio-economic backgrounds, learning difficulties and disabilities. This information usually is reported only within education systems or to governments and is not reported publicly, although schools sometimes provide information to their communities about the range of languages spoken by students in the school, the countries from which they come, the percentage of Indigenous students in the school and the school’s special Indigenous programs, or the number of severely disabled students and the facilities and support provided for these students. F: a sub-set of student intake measures on which it is agreed to collect nationally comparable data. Within the set of student intake characteristics reported for a school, there could be a sub-set of measures on which it was agreed to collect nationally comparable data. Some progress has been made toward nationally consistent definitions and nationally consistent data collections on student background characteristics. G. all other information that a school could choose to make available Beyond information about student outcomes, student backgrounds and their physical and human resources, schools provide a range of other information to the communities they serve. 1. 3 Nationally Comparable Data Acknowledging the many purposes and audiences for information about schools, and the various forms that this information can take, the specific focus of this paper is on the collection and reporting of nationally comparable data for the purposes of evaluating and comparing school performances. In other words, the focus is on categories B, D and F in Figure 1. We envisage three broad uses of such data: †¢ use by parents and caregivers in judging the quality of educational provision and in making informed decisions in the best interests of individual students; †¢ use by school leaders in monitoring a school’s improvement and benchmarking the school’s performance against other, comparable schools; and †¢ use by education systems and governments in identifying schools that are performing unusually well or unusually poorly given their circumstances. As noted above, these three stakeholder groups are likely to have different needs. The ways in which nationally comparable data are analysed, combined and reported may be different for different purposes. We see the process of reaching agreement on the core data that should be available about a school as a national collaborative process, and see little value in arriving at different conclusions about these data for different parts of the country. 4 Reporting and Comparing School Performances 1. 4 Principles for Reporting The Principles for Reporting Information on Schooling (see pages 6-7) adopted by the Ministerial Council for Education, Employment, Training and Youth Affairs (MCCETYA) provide an important point of reference for any proposed collection and use of nationally comparable data on schools. These principles recognise the multiple audiences and purposes for information about schools, the need to collect broad evidence about student and school performances, and the desirability of monitoring intended and unintended consequences of reporting information on schools. Australian governments have undertaken to ensure that data provided for the purposes of comparing schools are reliable and fair and take into account the contexts in which schools work. Governments also have undertaken not to develop simplistic league tables of school performances. 1. 5 Structure of Paper This paper first considers the kinds of nationally comparable data that might be collected about schools for the purposes outlined above. We draw on national and international research and experience, attempt to anticipate the likely requirements of different audiences, and take into account what measures currently exist and what additional measures might be desirable in the future. Each of the three data categories in Figure 1 is considered in turn: †¢ †¢ †¢ student outcome measures physical and human resources measures student intake measures (sections 2-3) (section 4) (section 5) We then consider alternative ways of evaluating and comparing school performances. Two broad methodologies are discussed: †¢ †¢ the direct comparison of student outcomes the construction of measures of school performance (section 6) (section 7) Finally, we consider issues in reporting publicly on the performances of schools: †¢ †¢ audiences and purposes for reporting options for public reporting on schools (section 8) (section 9) 5 Reporting and Comparing School Performances MCEETYA PRINCIPLES FOR REPORTING INFORMATION ON SCHOOLING There is a vast amount of information on Australian schooling and individual schools. This includes information about the educational approach of schools, their enrolment profile, staffing, facilities and programs, and the education environment they offer, as well as information on the performance of students, schools and systems. Different groups, including schools and their students, parents and families, the community and governments, have different information needs. The following principles provide guidance on requirements for information on schooling, including the types of information that should be made readily available to each of the groups noted above. These principles will be supported by an agreed set of national protocols on the access to and use of information on schooling. Good quality information on schooling is important: FOR SCHOOLS AND THEIR STUDENTS. Principle 1: Schools need reliable, rich data on the performance of their students because they have the primary accountability for improving student outcomes. Good quality data supports each school to improve outcomes for all of their students. It supports effective diagnosis of student progress and the design of quality learning programs. It also informs schools’ approaches to provision of programs, school policies, pursuit and allocation of resources, relationships with parents and partnerships with community and business. Schools should have access to: †¢ Comprehensive data on the performance of their own students that uses a broad set of indicators †¢ Data that enables each school to compare its own performance against all schools and with schools of similar characteristics †¢ Data demonstrating improvements of the school over time †¢ Data enabling the school to benchmark its own performance against that of the bestperforming schools in their jurisdiction and nationally FOR PARENTS AND FAMILIES. Principle 2: Information about schooling, including data on the performance of individuals, schools and systems, helps parents and families to make informed choices and to engage with their children’s education and the school community. Parents and families should have access to: †¢ Information about the philosophy and educational approach of schools, and their staffing, facilities, programs and extra-curricular activities that enables parents and families to compare the education environment offered by schools †¢ Information about a school’s enrolment profile, taking care not to use data on student 1  characteristics in a way that may stigmatise schools or undermine social inclusion. †¢ Data on student outcomes that enables them to monitor the individual performance of their child, including what their child knows and is able to do and how this relates to what is expected for their age group, and how they can contribute to their child’s progress †¢ Information that allows them to assess a school’s performance overall and in improving student outcomes, including in relation to other schools with similar characteristics in their jurisdiction and nationally. 1 Any use or publication of information relating to a school’s enrolment profile should ensure that the privacy of individual students is protected. For example, where the small size of a school population or of a specific student cohort may enable identification of individual students, publication of this information should be avoided. 6 Reporting and Comparing School Performances FOR THE COMMUNITY. Principle 3: The community should have access to information that enables an understanding of the decisions taken by governments and the status and performance of schooling in Australia, to ensure schools are accountable for the results they achieve with the public funding they receive, and governments are accountable for the decisions they take. Students are an important part of our society and take up a variety of roles within it after leaving school. The community is therefore a direct and indirect consumer of the product of our schools, as well as providing the means of public funding. Information about schools in the public domain fulfils the requirement that schools be accountable for the results they achieve with the public funding they receive, including relative to other ‘like’ schools; it should also give the community a broad picture of school performance and a sense of confidence in our school systems. The community should have access to: †¢ Information about the philosophy and educational approach of schools, and their staffing, facilities, programs and extra-curricular activities that enables the community to compare the education environment offered by schools. †¢ Information about individual schools’ enrolment profile, taking care not to use data on student characteristics in a way that may stigmatise schools or undermine social inclusion †¢ National reporting on the performance of all schools with data that allows them to view a school’s performance overall and in improving student outcomes, including in relation to other schools with similar characteristics RESPONSIBLE PROVISION OF SCHOOLING INFORMATION Australian Governments will ensure that school-based information is published responsibly so that: †¢ any public comparisons of schools will be fair, contain accurate and verified data, contextual information and a range of indicators to provide a more reliable and complete view of performance (for example, information on income, student body characteristics, the spread of student outcomes and information on the value added by schools) †¢ governments will not devise simplistic league tables or rankings and will put in place strategies to manage the risk that third parties may seek to produce such tables or rankings, and will ensure that privacy will be protected. †¢ reports providing information on schooling for parents and families and the community will be developed based on research on what these groups want to know and the most effective ways the information can be presented and communicated. FOR GOVERNMENTS Principle 4: Governments need sound information on school performance to support ongoing improvement for students, schools and systems. Government also need to monitor and evaluate the impacts (intended and unintended) of the use and release of this information to improve its application over time. Good quality information on schooling enables governments to: †¢ analyse how well schools are performing †¢ identify schools with particular needs †¢ determine where resources are most needed to lift attainment †¢ identify best practice and innovation in high-performing schools that can be mainstreamed and used to support improvements in schools with poorer performance †¢ conduct national and international comparisons of approaches and performance †¢ develop a substantive evidence base on what works. This will enable future improvements in school performance that support the achievement of the agreed education outcomes of both the Ministerial Council for Education, Employment, Training and Youth Affairs and the Council of Australian Governments. 7 Reporting and Comparing School Performances 2. STUDENT OUTCOMES Information about the outcomes of a school’s efforts is key information for parents and caregivers if they are to judge the quality of educational provision; for school leaders to monitor a school’s performance and improvement; and for education systems and governments to identify schools in need of additional support. However, schools work to promote many different kinds of outcomes for their students. For some schools, an important objective is to improve school attendance rates. For others, assisting students to make successful transitions into the workforce is a high priority. Some schools are more focused than others on supporting the social, spiritual and emotional development of students. Still others measure their success in terms of entry rates into highly sought-after university courses. Decisions about the outcomes to be reported publicly for schools are important because they influence judgements about how well individual schools are performing. This is particularly true when education systems and governments attempt to construct ‘measures’ of school performance: Perverse incentives can arise when the [school] performance measure has both a large impact upon actors and focuses on an aspect of schooling that does not reflect the true or overall purpose and objectives of schools. Unfortunately, this can be common in school performance measures if the performance measure is too narrowly defined. (OECD, 2008, 26).