Presentation prepared for the Education and Science Select Committee

By Karen Van Rooyen Chief Executive Officer New Zealand Qualifications Authority

16 February 2005

Contents: Presentation prepared for the Education and Science Select Committee Process for Scholarship, 2005 Qualifications Authority Processes All Available Subjects - Grade Fluctuations in NZ's Pass/Fail Education System 1997-2001 which were not scaled in those years Response To Controversy Over Race Relations Question In 90212 'Examine The Perspectives Of People In An Historical Setting' (History 1.4) Examination Paper 2004

The Presentation below is also available in Word ( )or PDF ( ) format.

Presentation prepared for the Education and Science Select Committee

Mr Chair, the formal purpose of this committee today is to examine the Annual Report, the financial statements and the overall performance of the New Zealand Qualifications Authority during the 2003-2004 financial year. No doubt the committee will also want to consider more recent and one-off events, but before I come to them, let me first give you an overview in terms of those formal matters.

The New Zealand Qualifications Authority story of recent years is one of the implementation of massive change - implementation that, albeit with some difficulties, has been hugely successful. In particular, we saw by the end of 2004 and into the early part of this year, the complete NCEA system come into being. Now NCEA is operating in years 11, 12 and 13, or as it used to be called, 5 th, 6 th and 7 th form.

Behind this is a profound shift from a system of assessment that was based on ranking students to one that seeks to assess what each student knows and can do. As has been said elsewhere, in the old days education divided those who succeeded from those who failed, in a series of steps, culminating in a small elite going to university.

Arguably, though it is moot, this was appropriate for the much simpler economy that we had in the 1950s, 60s and 70s, but it would not be appropriate now.

Now we have a much more multi-layered economy. Most people now go on to some kind of tertiary education, either in a tertiary institution or the workplace. For a multi-layered economy we need a multi-layered education system and a multi-layered system of qualifications and assessments.

The central feature of this new system is that it is standards based. So, firstly, the Ministry of Education sets a standard for a particular qualification. Teachers then teach it, and then we, NZQA, set exams or other assessment procedures and for external exams we arrange the marking. By that we find out if the students can meet that standard, and then that's the result. We don't using scaling. The information that the student has, and which he or she can present to an employer, is real.

And, of course, here lie some of the difficulties that have attracted the headlines in recent weeks. But I will come to that in a moment.

Just briefly, the New Zealand Qualifications Authority has met the operational challenges that this has presented. With a permanent staff of a little over 250, the work has been done.

For example, in 2004 we marked 2,000,000 papers. And we mail each paper back to the students so they can learn from their mistakes, enjoy their successes, and, if they wish. litigate the results. To my knowledge no other western country does that.

In all this, my staff have been magnificent, and I thank them. They have not wavered. Looking to the next few years, I am seeking a period of consolidation. We have stretched our capacities to the limit to get the work done. That carries risks, which now need to be mitigated. My programme, looking forward, is about maturing the organisation and building further its internal capacities.

And I am looking forward to NZQA being a little less newsworthy, as the new assessment system gains acceptance, and over time people see that it is sensible and it works.

Now I want to turn to the specific subjects are of interest right now, or have been of interest in recent months.

Website

Last year an issue arose in regard to the Qualification's Authority's website and a press statement which was critical of the National Party's education spokesman. The statement was one of a number of press statements by non-NZQA people that we had placed on our website because they made constructive points about NCEA. We failed to notice that this particular press statement also made some critical comments to which Mr English understandably took exception, when they appeared on the website of a government agency. As soon as Mr English made his concerns known, the press statement was taken off our website, and our error was acknowledged. We have reviewed our website policy, and concluded that from now on only press statements from NZQA itself will be posted on our website.

Economics exam

The New Zealand Qualifications Authority has been criticized for what some see as bias in two of its exams.

The first of these was an economics exam. At issue was the fact that a 2004 Level 3 Economics exam included the questions "The New Zealand government provides "free" education at state secondary schools. Explain why this results in a better resource allocation than the free market," and "Explain why using 'free market' policies causes income inequality".

These exam questions were prepared as mechanisms for assessing students compliance with a curriculum and standard that NZQA does not set. NZQA is responsible for assessment, not what's taught.

In this case, these questions were in response to a syllabus which includes a section called "Allocation via the Public Sector". The relevant part requires that students " Describe the characteristics of public and merit goods and identify the reasons why the market will not, or cannot, supply them in sufficient quantities".

The questions that caused concern were assessing what students knew in terms of the standard. NZQA has no doctrinal or philosophic opinion on these matters either way. Our only function is to assess what students know and can do, in terms of standards that others in the education sector set. The question did that.

History Exam

The history exam that caused concern was a Level 1 History 2004, which included a reference to a 1970s "National Party Member of Parliament not sympathetic to Maori concerns". There was also concern about an illustration of that hypothetical MP which was said to look like the current leader of the National Party.

Firstly, I want to assure this committee that the art work was in no way intended to satirise any individual. We would never do that.

Regarding the exam question itself, the students were being asked to write from the point of view of the protagonists during a difficult and turbulent period of New Zealand's history. NZQA makes no judgment about whether the views of those involved were right or wrong. The question did not invite students to denigrate or judge anyones' views - rather it asked them to argue those views. The question was about an historical period. It was not about the views of the National Party or National MPs now.

The person who wrote the exam question has since provided a note to the Authority explaining the thinking behind that question and the wording. It is annexed to this paper. Clearly the question was written with great care.

Scholarship

And, finally, Mr Chairman, I would like to discuss New Zealand Scholarship.

The essential point about the 2004 New Zealand Scholarship results is that significantly fewer students achieved Scholarship than in recent years. In addition, that diminished achievement was most marked in the sciences.

The degree of variability in National Scholarship was more than NZQA expected, unlike in levels in 1, 2 and 3 NCEA, where the variability is as expected.

By way of background, there is always variability whenever the total numbers of students sitting an exam change from year-to-year and when the exam changes from year to year. This is illustrated in a table attached to this paper, which shows variability year-by-year for the old School Certificate.

Nonetheless, the variability in Scholarship was more than was expected. NZQA will be working with the Ministry of Education and the school sector to ensure that New Zealand Scholarship in 2005 produces less variability than in 2004.

The Government has announced interim measures to ensure that students who did very well in 2004, but did not achieved Scholarship, still gain recognition. The Government has also announced various reviews, which I welcome. These will help us, and by us I mean the education sector, sort this out.

There is a great deal more that needs to be done to determine why this variability occurred. But is there anything that at this stage we can say about reasons?

Some key points set the scene. New Zealand Scholarship is actually a new qualification, even though it carries the same name as its predecessor. For the previous ten years or so Scholarship was pegged to the best results in bursary. In 2004 for the first time in quite a while it was a stand-alone series of exams. In its new format, it was identified as a very rigorous testing of those students destined for university study at the highest level.

More fundamentally, New Zealand Scholarship is new because, like NCEA, and indeed the National Qualifications Framework, it is standards based. So there's no scaling. The results are real - we have adjusted marks up or down to get a predetermined result.

Thus what's happened this time is that at this level what young people know and can do has been "unmasked".

There are five questions that need to be asked, these being:

• Was the curriculum sound?
• Were the standards appropriate?
• Did the teaching provide what was required?
• Were the exams written correctly relative to the standard?
• Was the marking sound?

All these will be reviewed, but let me give some preliminary thoughts.

We have to ask were the standards in some subjects too tough? This is not to blame anyone, but it is one of the things we need to know.

Aside from a monetary reward, the benefit from Scholarship in some courses, mostly the sciences, is that success confers an entitlement to skip stage one at university and go straight to stage two.

The standard would not be appropriate if it was so easy that it pushed students into stage two university who then couldn't cope. Likewise, the standard would be at fault if it was so hard that it was keeping out students who could do stage two.

Likewise, we need to look at how schools prepared students for this exam. The new Scholarship standards require students to synthesize and interpret what they have learnt. Merely knowing the subject is not enough. This is qualitatively different from other less demanding exams and assessments, and these are skills required for senior careers in science.

And then we have to look at the writing and marking of the exams, NZQA's direct responsibility. Both processes involve rigorous cross checking to mitigate risk. Charts explaining how both these processes work are attached. As they reveal, the process is multi-staged and involves very careful scrutiny and cross checking.

That said, did the exams match the standard and was the marking appropriate? We need to do more work to answer that definitively, and that work is under way now. NZQA itself will be commissioning an external review of our exam setting and marking, to be undertaken by international experts.

The events of recent weeks have been very difficult, but the virtue of standards-based assessment is that it forces us to confront these issues, and then to do better, and we will.

Mr Chairman, it has been especially useful to lay these matters out before this committee and the public. I now welcome the committee's questions.

Qualifications Authority Processes

Setting of examinations

• Exam writers with tertiary experience contracted - positions advertised nationally
• All contractors trained by NZQA National Assessment Facilitators (NAFs)
• Exams drafted
• Exams independently critiqued for:
  • Alignment to assessment specifications for standard
  • Clarity of language and instruction
  • Assessment tasks set right level
  • Consistency between evidence, assessment, and standard
• Exam reviewed and revised
• NZQA editors check exams for layout, language, terminology
• Exam checked by NZQA NAF responsible for scholarship subject
• Exams tested/trialed by independent checker
• Independent checker information to NZQA editor
• Information to NAF, Examiner,critiquer
• Examiner/critiquer confer to resolve any issues
• Exam paper updated
• Editor, examiner and NAF recheck
• Paper approved.

Marking of examinations

• Subject panel leaders and markers recruited (secondary and tertiary) and trained by NZQA National Assessment Facilitators (NAFs)
• All panel leaders and markers trained by NZQA
• Exams occur
• Papers to markers
• Panel leaders and markers meet and consider candidate responses and determine levels of sufficiency (standard setting meeting)
• NZQA NAF meets with panels and markers in 'target areas' re levels of sufficiency (standard setting meeting)
• Exemplars identified to demonstrate levels of performance
• Markers use exemplars to establish marking standards
• Marking occurs
• Marking checked by panel leader and check markers (approx 10%)
• Panel leader checks check markers
• Results submitted to NZQA
• Scripts submitted.

All Available Subjects - Grade Fluctuations in NZ's Pass/Fail Education System 1997-2001 which were not scaled in those years

Note: These figures have been taken from the relevant year's edition of Secondary Qualifications Statistics, which is publicly available.

Also, if individual questions were looked at there would be greater variability. There would have been similar variation in Bursary, but it was hidden by scaling.

Response To Controversy Over Race Relations Question In 90212 'Examine The Perspectives Of People In An Historical Setting' (History 1.4) Examination Paper 2004

In the light of recent political and media comment on the above question I think it appropriate to inform the Authority of the context and rationale behind the question. I will firstly outline the parameters in which I was working from an examination setting point of view, and then will outline why the content of the question itself was considered valid and appropriate.

Examination context

The Achievement Standard requires students to "describe and explain perspectives and related actions of people from the setting". Furthermore, the Explanatory Notes make it clear that questions must be answered 'in role', in other words, answering from the point of view of the person concerned.

• To give students ample opportunity to demonstrate their ability to do this, examiners in 2002 and 2003 had settled on a pattern of asking for three different perspectives, and related actions.
  • A constraint for the Examiner is that not only does a context have to be chosen that reflects three defined perspectives (remembering that this is entry level History), but that there must be enough content behind each example to not only describe the perspective, but to state consequent actions that demonstrate it.
• The set topics for the setting of contexts are in the Appendix to the Achievement Standard. This topic is 'Race relations: New Zealand Māori and Pākehā 1912-1980'.
  • Each set topic is divided into three sections each headed by a focussing question. In 2002 the context was chosen from the first section; in 2003 it was chosen from the second. So in 2004 it was decided to select a context from the third section.
  • The third section, in full, states:

    (3) What were the issues in race relations 1960-1980?

    • Race Relations in the 1960s and 1970s
      • land, language and culture, education and employment
      • the emergence of Māori urban leadership

  • So from within the above section a context had to be chosen that involved three people representing three perspectives and related actions. It was from this that it was decided to focus on land as it appeared to be the most clearly defined.
• Whina Cooper and Eva Rickard were chosen for two of the individuals because:
  • their perspectives were well-known,
  • they are always taught in the topic, and
  • they had clearly associated actions that backed up their perspectives on Māori land (the Land March and Raglan Golf Course protest respectively).

Ironically, the other well known and commonly taught case study in this topic is the Bastion Point occupation. This example was rejected for this question partly because it was led by Joe Hawke, later to become a Labour Party MP. It was thought that this could be seen as politically contentious, especially when set up against the 'National MP'.

The choice of 'A National Party Member of Parliament not sympathetic to Māori concerns'

Having chosen two Māori activists, a third person had to be chosen that clearly represented a different perspective, and could have actions to relate to the perspective.

Given that through this period there were no pressure groups that overtly represented an opposing viewpoint it was decided to choose a member of the government of the period. This would be someone who supported the status quo and was opposed to the changes sought by Cooper and Rickard.

There were several ways this individual could have been described:

• A named politician of the period - eg. Ralph Hanan, Duncan McIntyre, Keith Holyoake, Norman Kirk, Matiu Rata etc

This option was rejected because:

• These individuals are not focused on in the customary way the topic is taught by History teachers. This would make it excessively difficult for candidates.
• There was a risk that candidates would focus on race relations issues other than land if one of these politicians was chose

For example, Hanan was known for his implementation of the Hunn Report which was more to do with social and educational reform, McIntyre was known for his rapport with Māori leaders and making steps towards official biculturalism, Rata was known for his role in establishing the Waitangi Tribunal in 1975, and later his break with Labour to form Mana Motuhake in 1979, Ben Couch for his support of rugby contacts with South Africa and so on

It must be remembered that the focus of the question was land - not race relations generally.

• 'A member of the government in the 1960s and 1970s'

This option was rejected because:

• There were noticeable differences in perspective between the National and Labour governments of this period on some Māori land issues.

For example:

• Labour opposed the 1967 Māori Affairs Amendment Act passed by the National government.
• Labour tentatively welcomed, if not fully endorsed, the Māori Land March in 1975,
• it established the Waitangi Tribunal in 1975,
• it opposed the Muldoon government's actions on Raglan Golf Course and Bastion Point.

To create an assessment schedule for this would have been problematical because there were just too many options within it. A clearly contrasting perspective to Cooper and Rickard could not be guaranteed from students at this level.

• A 'Pākehā who was unsympathetic to Māori concerns'

This option was rejected because:

• Although they could probably give the perspective, what actions could candidates give to support it? This would not be History, it would be retrospective speculation or guesswork.

• 'A National Party Member of Parliament'

This option was rejected because:

• There was a risk that it could be seen as generalising all National Party MPs of the period with the same perspective, and supportive of all the actions relating to Māori land undertaken by the National governments of the 1960s and 1970s. It assumes that if you were a National MP you had a set and given perspective. This is clearly unsatisfactory.

• 'A National Party Member of Parliament not sympathetic to Maori concerns'

This proved the most satisfactory option and was chosen because:

• It directed candidates to consider someone who had a different perspective to Cooper and Rickard.
• No single individual could be identified.
• It stated that this was 'a' National MP who was clearly likely to have a different perspective to the other two individuals named. Being a member of the government 1960-72/ 1975-1980 it could be safely assumed that this would be a person who supported measures such as the 1967 Act and opposed the claims of the1975 marchers, and Bastion Point and Raglan activists.
• It was felt that his was the only way that a contrary perspective, with actions, could be asked for students at this level in this section of the topic.
• It did not prohibit the possibility of other National MPs of the time supporting the activists or opposing their own government's measures regarding Māori land. In other words, the wording indicated that this individual was not only from the group of National MPs generally, but was one from within it who also happened to be 'unsympathetic to Māori concerns'. It was not even assuming that those who supported National Māori land policies in this period were necessarily unsympathetic - it was just saying to candidates that you are given an unsympathetic character here - that's a given - now, how would this person view National Māori land policies at this time. It had to be a government politician so that candidates had examples of actions to draw on. In other words the intention was to avoid ambiguity for candidates

Justification for using National government actions as opposing Māori land aspirations

All of the above assumes of course, that the National government, which held power for 17 of the potential 20 years covered by the question, did have a contrary perspective to many Māori on issues of land (remember always, the question is about land - not other race relations issues). Why is this assumption made?

• The Māori Affairs Amendment Act 1967 was hugely unpopular with many Māori, including the NZ Māori Council which tended to have more conservative views than so-called 'activists'. This legislation was seen by Māori as another attempt to acquire Māori land. Keith Sinclair called it the 'last act of the Land Wars'.
• National opposed the formation of the Waitangi Tribunal in 1975
• National refused to make any changes to reflect Māori concerns about land alienation following the 1975 Land March
• National opposed the occupation of Bastion Point, and sent in police in 1978 to break it up.
• Although tenuous, it also cannot be denied that throughout this period most Māori continued to support Labour at the polls indicating that most Māori thought their interests were better served by Labour than National at this time.

Refer to the copies of pages from Year 11 History textbooks (the only ones known extant for this topic) - posted under separate cover - for further information. These indicate how the topic has been taught since it was introduced in 1976 and subsequently revised in 1989.

The illustration

Where an individual was representative of a group, illustrations were prepared rather than use photographs which could cause students to think that they should know the person in the photograph. Illustrations were also used for topics 1, 3 and 9.

The artist worked under directions given by the Examiner. I have not had time to find the specific instructions given to her (with more time I possibly could), but what was stated was that the illustration should reflect a common view of the person being depicted. For example, to have a young Māori woman as representative of a National Party MP at this time would have been inappropriate because no one fitted that image. Entirely of her own accord, the artist drew a middle-aged, balding Pākehā to represent a National Party MP of this time. This was not untypical of MPs at this time and would have been unlikely to confuse candidates.

At no stage or in any way was it suggested that any likeness of the current National Party leader suggested. It is astonishing that anybody could take such a view. He does not appear in any Level One History topics, and obviously was not balding, middle-aged or in Parliament during the 1960s or 1970s.

Wider issues

Examiners already operate under considerable constraints to ensure that papers are produced that are fair and valid. It would be a matter of grave concern, especially in a subject such as History, that areas of the course could not be examined for fear of offending politicians of the current generation.

If Examiners are to be beholden to sensibilities of today's politicians, how do they examine current Level One topics such as, for example:

• Alternative policies adopted to deal with the Great Depression, social security, superannuation (Social Welfare: NZ 1891-1980)
• The anti-nuclear policies of the 1970s and 1980s where political parties had - and arguably still do - have different perspectives (International Relations: NZ's Search for Security 1945-85)
• The anti-Jewish and lebensraum policies of the German government in the 1930s, and war with Japan. Do we have to harness how these are taught - and examined - to avoid of offending present day sensibilities?

It would be tragic if an outcome of this controversy is that History teachers in New Zealand become accused of 'bowing to political pressure', 'rewriting History', being forced to be 'politically correct' or other such emotive terms.