Building teacher assessment capabilities to improve instrument quality and teacher marking reliability

Teacher assessment literacy is critical in improving assessment instruments and teacher judgments on the standards of student achievements on those instruments. The discussion in this paper focusses on the value of new system of ‘Endorsement’ of assessment instruments before use with students and ‘Confirmation’ of teacher judgments in marking student responses to those. This involves the 20,000 teachers of Grades 11 and 12 in Queensland, Australia. In particular, this paper examines the significant increase in teacher professionalism that results from enhanced assessment literacy, and the far more detailed feedback (feed-forward) to students about how to not only improve the quality of future works, but actually monitor the standards of those works during their production. It lets students into the ‘secret’ of marking (Sadler, 2002, 2007) using explicitly stated criteria that align directly with the subject syllabus objectives (conceptualised from Marzano & Kendall, 2007) - typically 6 – and the standards descriptors for each mark level in those. This also provides students with the opportunity to meet achievable challenges set for them (Willis, ca 2018). So, what is ‘Endorsement’ and ‘Confirmation’, and how why are they practiced? The quality and comparability of assessment tasks and marking in a school-based assessment system where teachers both set and mark assessment instruments is critical. We have to trust teachers to do that well (Stobart, 2015), and in Queensland, the inter-marker reliability on a five-point scale is around 0.9 (external exams are typically about 0.65 – personal communication, P. Newton of Cambridge Assessment, 2008). Instead of statistical moderation of school-based assessment results using the externally set and marked assessment instrument in each subject - a practice in many jurisdictions - Queensland ‘front-ends’ quality instruments and teacher marking using rigorous processes. In each General subject (there are 69 General subjects such as ‘Aerospace’ Systems’ and ‘English’ and 25 Applied subjects such as ‘Building & Construction Skills’ and ‘Business Studies’ - see QCAA, 2018a), there are three internal summative assessment instruments (worth 25 marks each) and one external (worth 25 marks). Teachers mark the internal assessments using the instrument-specific marking guide (ISMG) for each provided in the relevant subject syllabus. Before use in the classroom for summative assessment (contributing to exit mark) with Grade 12, QCAA Assessors (teachers employed part-time by the QCAA), assessment instruments from each school in each subject undergo a process of Endorsement. This is to ensure that “all assessments provide sufficient opportunities for students to demonstrate syllabus requirements and to build teachers’ capacity to develop high-quality assessments” (QCAA, 2018b). The three Grade 12 internal summative assessment instruments are endorsed about four months in advance of students undertaking them so that there is time for any required edits or amendments of the assessment instrument by the school and the QCAA Assessors to check that the quality is satisfactory in accord with the Principles of Assessment (QCAA, 2017) and specific subject syllabus requirements. Once students complete the internal assessments, teachers mark the student responses using the ISMG provided by the QCAA in the relevant subject syllabus. The comparability of student achievements is then achieved through a Confirmation process. There are just on five hundred schools teaching approximately 110,000 students in Grades 11 and 12 across the State. To achieve comparability of results between schools State-wide (and, to some extent within schools too), moderation involves the checking of the marks awarded out of 25 for each instrument. Indeed, they are checked down to the level of each objective so that data is obtained, and rectifications made should a teacher/school be too soft or too hard in marking. That is, on each objective assessed in an instrument as well as overall out of 25 for that instrument. In this process QCAA Assessors check the school’s marking for both accuracy and consistency by selecting stratified samples of student responses. So, for example, if half of the students studying English got say between 20 and 22 out of 25 for an instrument, then half of the sample would be from that mark range. The QCAA Assessors match the evidence in each student’s response selected to the instrument’s ISMG in the syllabus. So, while schools are responsible for internal moderation processes, it becomes very obvious with the Confirmation process if there is potentially an issue within the school (such as marking inconsistencies between teachers), for the Head of department or Principal to address through, for example, further marker training and professional development in assessment. In discussing Confirmation, the QCAA (2018c) emphasises that: The Confirmation process is informed by the attribute of reliability. Schools will be confident that the Confirmation process ensures: • student work is judged using the ISMG • judgments are valid and reliable • students’ final subject results are credible • judgments are comparable across schools and the state. Using the ISMG, teachers will mark student assessment and internally moderate judgments within or across syllabus cohorts, to ensure consistency of results. This will continue building teachers’ skills and abilities in making judgments about student responses. Overall, teacher professionalism and capabilities are further enhanced through the rigorous processes of Confirmation and Endorsement. In addition, the detailed nature of the ISMGs results in feedback to students, in the context of the specific requirements of the assessment instrument, that is both very detailed and comprehensive. Indeed, with Endorsement and Confirmation as teachers build their assessment literacies and capabilities, students similarly have the opportunity to meet achievable challenges and monitor the quality of their responses to assessment instruments during their production.