SecQual

Marking External Assessment (MEA) Pilot
S2010/012 - 20 Apr 2010

In the 2010 examination round, a small number of level 1 standards will be marked using a standards-based scoring method for collecting and collating evidence. Candidates will continue to receive an N, A, M, or E grade on their results notices. However, their examination booklet will show the scores used in assigning that grade, with the final score shown on the front cover. The judgement statement will show the cut-score for each grade and enable candidates to check their result.

The methodology used in the pilot has been endorsed by NZQA’s external advisory group, the Technical Overview Group Assessment (TOGA). Members of this group include Professors Gary Hawke (Chair), Terry Crooks, Cedric Hall, John Hattie and Geoff Smith, and Associate Professor Alison Gilmore.

Background

Item response theory (IRT) has been used in NZQA since 2006 across all subjects to analyse the effects of question quality on candidate performance in examinations. As a result of this research, improvements have been made to the format of questions, beginning with a move in 2008 to scaffolded items, each having a single Not Achieved, Achieved, Merit or Excellence grade. An item may be a single question or several related questions. All items provide opportunities to reach Achieved, Merit and Excellence. This change was incorporated in the majority of papers in 2009, with the remainder expected to incorporate the change in 2010.

Alongside this research, a project was set up to consider using scores to assign grades. Initially, scores were assigned to N, A, M and E to arrive at an appropriate grade. This rough estimate showed that the score-based method may be a more reliable way of assigning grades than the current sufficiency method and full trials were begun in 2007.

The assessment schedules were written with scores assigned to nine levels of performance, based on the criteria in the standard, as follows:

The numbers correspond to levels of evidence as shown in the assessment schedule e.g. M6 corresponded to evidence for high Merit performance. Marking panels working in March/April 2008 and 2009 re-marked copied examination papers using the score-based schedules. The original papers were unaffected and returned to candidates as usual. Benchmarking by the panel leader and senior marker was used to set the appropriate level of candidate response for each score for an item using ‘guinea pig’ papers. The scores across all items in a paper were then aggregated. Towards the end of marking, the panel leader and senior marker then held a standard-checking meeting, called the cut-score meeting, where samples of candidate work were used to make a final judgement of performance against the standard. The cut-score meeting is an additional stage of benchmarking which is very effective because it takes place late in the marking process when the markers are very familiar with the student answers.

These trials showed greater accuracy in assigning grades using score-based marking. Surveys of the experienced markers involved indicated that once used to the system, markers preferred the score-based system and found it straightforward to use.

Marking External Assessment (MEA) Pilot 2010

The marking pilot this year will be a live pilot. The standards involved will all be at level 1 and have been selected because they cover a range of types of subjects and cohort sizes. The MEA pilot will use information gathered from the previous trials and will use a schedule with the same nine levels of performance.

The MEA marking process will have five key differences from the present marking process:

• the collection of evidence is score-based
• scores are aggregated to give a total score across a paper
• markers record the aggregate score for each candidate
• the cut-score phase is the final standard checking phase
• when the cut-scores are entered the grade is generated automatically from the database.

The candidate results notice will not be affected by this marking change.

The marking on the returned papers will, however, appear different:

• each item will have an overall grade and score, e.g. a high M will be an M6, a low A will be an A3
• the aggregate score will appear on the front of the paper but not the grade
• the grade will appear on the candidate’s results notice.

The judgement statement, as published on the NZQA website in January, will show the cut-score for each grade boundary. This will enable candidates to check their grade has been generated correctly.

Benefits expected from score-based marking include:

• closer alignment of marking to the standard
• greater accuracy at grade boundaries
• reduction in year-by-year variability
• increased transparency.

Questions and Answers

Questions and answers about the MEA Pilot are available below.

MEA QA doc (DOC, 47KB)

MEA QA pdf (PDF, 18KB)

Enquiries

Please refer any enquiries relating to this circular to your School Relationship Manager.

School Relationship Manager
Secondary Assessment and Liaison
Telephone: 04 463 3000
Fax: 04 463 3113
Email: firstname.lastname@nzqa.govt.nz