QA News June 2010 - Issue 68

• Chief Executive's foreword
• Leaders of the future
• Qualification design rules out for consultation
• University Entrance being reviewed
• Marking External Assessment (MEA) Pilot
• Annual Report on NCEA and New Zealand Scholarship
• Moderator-teacher agreement rates rising
• Excellence in unit standards
• Adult Education and Training Qualifications
• Check out National Qualifications Services
• Quality Assurance for Mātauranga Maori
• Feedback positive on kaiako and whānau workshops
• NZQA to contribute to OECD education review
• Top Art on tour

Moderator-teacher agreement rates rising

In 2009 the overall moderator-to-teacher agreement rate for student work was 83% at the level of credit (i.e. at the Not Achieved or Achieved boundary), and 76% at the level of the grade. The latter figure for 2008 was 72%.

The agreement rate gives a broad indication on whether or not teachers are assessing at the national standard for those standards selected for moderation. It represents all types of standards at each level of achievement.

Brent Logan, Manager of Secondary Moderation, says “We aim to achieve a year on year improvement in the agreement rate.  Due to the nature of standards-based assessment there is a degree of subjectivity and professional interpretation involved so we will never achieve 100% agreement.” [see below]

Since 2009 the Secondary Moderation team has established a range of initiatives to help improve assessment judgements being made by teachers.  The most significant has been to offer hundreds of subject specific workshops throughout the country for teachers to work together discussing student work with national moderators.  Other initiatives include subject specific web pages, clarifications documents, reports and annotated exemplars of student work at the grade boundary. 

“It can take a year to see the benefits of these initiatives,” explains Brent. “This is because of the time lag involved between raising professional understanding, further assessment taking place and when materials are then requested for moderation. Most of the materials moderated in 2009 were for assessment that was undertaken in 2008. Moderation is a post-assessment feedback process.”

The Quality Assured Assessment Materials (QAAM) trade mark that will be applied to assessment materials that meet the national standard.

NZQA plans to introduce further consistency guidelines for moderators and a Quality Assured Assessment Materials (QAAM) trade mark to ensure that assessment materials are at the national standard before they are used in schools.

“We have already noticed a steady improvement in teacher-moderator agreement rates over the last year and we fully expect this to continue to improve. From 2010 we are selecting a more representative sample of standards for external moderation by NZQA.  This will give us a better idea of what the overall agreement rate is for all standards rather than just for those we have identified issues with,” Brent says.

The moderators provide feedback in their moderation reports on how teachers can better judge student work against the national standard. “We strongly encourage teachers to regularly check the subject specific web pages on the NZQA web site and to attend a secondary moderation best-practice workshop to discuss issues directly with our National Moderators. Many of those who attended these workshops in 2009 have described them as the best professional development they have had in many years”.

Details of these can be found on the subject specific web pages or http://www.nzqa.govt.nz/subject name.

Limitations of agreement rates

There are limitations to applying the agreement rate to all assessment judgements being made by teachers. Until July 2007 student work was not randomly selected. This took a year to fully implement and the random selection of student work will not be a feature of moderation data until 2010 moderation is reported next year. Until this point, teachers were asked to send in that evidence that was borderline or which they had the greatest difficulty making judgements.  This meant it was not representative of all assessment judgements.  Also, the standards selected for moderation are not representative of all standards.  In the past, School Relationship Managers have targeted for moderation those standards where teachers were having the most difficulty with judgements or any new standards where agreement rates were likely to be lower.  So the agreement rates reported to date very likely under-represent the true level of agreement between teachers and moderators.