Presentation prepared for a sub-committee of the Education and Science Select Committee
By Karen Van Rooyen Chief Executive Officer New Zealand Qualifications Authority
16 March 2005
Contents: Introduction |
The Presentation below is also available in Word (  )or PDF (  ) format. |
Introduction
This inquiry has been called to investigate the way examinations are set within the secondary system and to look, in particular, at whether bias has been injected into any recent exams. I will be making some introductory comments and then Kate Colbert, who is Group Manager Secondary Education, will be describing in more detail the way external examinations within the NCEA system are set. Then I will be happy to take your questions.
The impetus for this inquiry came from concerns about a 2004 NCEA history exam and a 2004 NCEA economics exam. In the case of the history exam the concern was that it denigrated one of the political parties currently represented in parliament. In the case of the economics exam the concern was that an ideological bias was being displayed.
I have examined both the history and economics exams in question. I am satisfied that they both meet the standards that NZQA requires. In my judgment, neither exam is biased or reveals either malice or prejudice. Both exams were valid.
The process of exam setting begins with the Ministry of Education, which determines the curriculum and set the standards. NZQA then writes and marks exams, which assess what students, know and can do relative to those standards. Thus the correct question is did the exams in question fairly test the students' knowledge relative to the particular standards. I believe that they did.
Economics exam
The 2004 Level 3 Economics exam included the questions "The New Zealand government provides 'free' education at state secondary schools. Explain why this results in a better resource allocation than the free market," and "Explain why using 'free market' policies causes income inequality". As a result, an anti-market bias was alleged.
These exam questions were prepared as mechanisms for assessing students' achievement with a curriculum and standard. The curriculum and the achievements standards are developed by the Ministry of Education, after wide consultation with secondary and tertiary experts in all relevant disciplines.
In this case, these questions were in response to a syllabus that includes a section called "Allocation via the Public Sector". The relevant part requires that students "Describe the characteristics of public and merit goods and identify the reasons why the market will not, or cannot, supply them in sufficient quantities". Likewise the relevant Ministry of Education Standard is headed "Describe and illustrate resource allocation via the public sector to compensate market failure".
NZQA has no doctrinal or philosophic opinion on these matters either way. Our only function is to assess what students know and can do, in terms of the standard. The question did that and similar questions in bursary exam papers have appeared in the past.
The type of question will change only if the curriculum itself changes. The issues raised appear to relate to the current content of the curriculum, rather than the examination framework.
History Exam
The history exam that caused concern was a Level 1 History 2004, which included a reference to a 1970s "National Party Member of Parliament not sympathetic to Maori concerns". In addition, an illustration in the exam of that hypothetical MP was said to look like the current leader of the National Party.
Firstly, the art work was in no way intended to satirise any individual. We would never do that. Insofar as the picture reveals anything, it is merely that, in the mind of the person who drew it, a National Party MP in the 1970s might have been tall and well dressed. With the gift of hindsight, it would have helped if we had added a moustache or a full head of hair, but that is after the event.
Regarding the exam question itself, the students were being asked to write from the point of view of the protagonists during a difficult and turbulent period of New Zealand's history. The question was not about the views of the National Party or National MPs now. This was the period of land marches, the Bastion Point controversy, and emerging Maori nationalism.
Academic rigour demands that aspects of New Zealand's past should be scrutinized dispassionately. History should not be tidied up, rewritten or sanitized. The fact is that in the period being studied most of the time National was in power, and to study the period the students needed to consider the point of view of a person who opposed Whina Cooper and Eva Rickard – not just any person, but someone who could influence events as a member of the government of the day.
The 1970s was a period of collision between the majority culture and Maori in New Zealand. We shouldn't shy away from that fact or get precious about things that are different in terms of the way we view the world today.
NZQA makes no judgment about whether the views of those involved then were right or wrong. The question did not invite students to denigrate or judge anyone. It did not ask the students to say whether any of the protagonists at the time were right or wrong – rather it asked them to argue from their points of view. Indeed, were a student to argue cogently or even convincingly from the point of view of that hypothetical National MP, they would get a high mark.
The person who wrote the history exam has provided a note to the Qualifications Authority explaining the thinking behind that question and the wording. It is annexed to this paper, as it was to an earlier submission to the full Education and Science Select Committee. It shows that the question was written with great care.
Kate Colbert will now explain the processes by which we set exams, the point being that these are not ad hoc.
Exam Setting Processes
Kate Colbert,
Group Manager Secondary Education,
NZQA
The process by which we set exams is rigorous. Considerable effort and care goes into examiner selection, examiner training and then exam setting and extensive cross-checking. Chronologically, it works as follows.
In the June or July of the year preceding the examination we advertise for Examiners and Material Developers and by July or August they are appointed.
Material Developers act as drafting assistants to Examiners.
For Examiners and Material Developers we seek people with
- subject knowledge and expertise in terms of the Ministry of Education Standards being assessed
- a generic knowledge and understanding of standards based assessment
- regional and national subject involvement
- knowledge or experience of the Treaty of Waitangi, Te Reo Maori and Tikanga Maori
- an ability to work to stringent deadlines
- an ability to implement and maintain appropriate administrative and communications systems.
Typically these people are working teachers, ex-teachers or university academics.
All new Examiners and Material Developers are required to go through a training procedure. Within the training the requirement to avoid bias is made very clear.
From September through to January exam papers are drafted. Note that an exam paper is not a complete exam. It is a set of questions to match Ministry of Education Standards, which, later on, are brought together and compiled as actual exams.
Then the exam papers are sent to "Material Critiquers" – a second pair of eyes – who scrutinize them to ensure that each question
- is aligned to the relevant Ministry of Education Standards and assessment specifications
- has clear instructions and language
- is set at the right level
- exhibits consistency between evidence, assessment and the MoE standards
- can be done in time, relative to the exam as a whole.
The Material Critiquers provide feedback to the Examiners.
Out of this process, a final draft of exam papers is submitted to NZQA's 12 National Assessment Facilitators – a third set of eyes - at the latest by May. They check the submitted material and NZQA Editors – a fourth set of eyes - format the various papers into the first proof of the actual exams.
All exam papers are then sent to separate Independent Checkers – a fifth set of eyes – who are subject experts. The Independent Checkers do the exams without any background material, as if they are students, and then provide answers and feedback as to anything in the exams that was confusing or ambiguous.
As a result of these processes further amendments are made, and after another round of checking within NZQA, in July to September exams are finalized for printing.
In aggregate, this is a process that takes considerable time and "many sets of eyes". As the line manager directly responsible, I have confidence that this process is robust and has as many cross checks as one could legitimately expect, relative to the time and cost involved. I have looked at the two exams in question and I am confident that in their preparation due process was followed and the appropriate care was taken.