Guidelines for assuring national consistency of graduate outcomes for qualifications at levels 1-6

These guidelines are for tertiary education organisations who have graduates of a New Zealand qualification that is due for a consistency review.

You can download a pdf version (PDF, 417KB) of the guidelines or see the details below.

A key feature of New Zealand qualifications is that the qualification can be obtained through different programmes and education organisations.

NZQA quality assurance is intended to assure stakeholders that even though graduates have completed their qualifications through different programmes they are all achieving the same outcomes at an equivalent standard.

The quality assurance process used is assuring national consistency of graduate outcomes. It applies to New Zealand qualifications at levels 1-6 listed on the New Zealand Qualifications Framework (NZQF).

What is a consistency review?

Assuring national consistency of graduate outcomes is a process developed to maintain confidence in the graduates of New Zealand qualifications and the underlying knowledge, skills and attributes they gain. These guidelines outline the approach.

Consistency review is not an external moderation of assessment process, although information from moderation may be used to inform the education organisation about how well its graduates are achieving the graduate outcomes of the qualification.

The focus is on comparing graduates in relation to the qualification outcomes.

Assuring national consistency of Mātauranga Māori qualifications

A specific approach has been developed for assuring the national consistency of graduate outcomes resulting from Mātauranga Māori qualifications. The approach incorporates the principles of NZQA’s Te Hono o Te Kahurangi framework.

Please refer to the Guidelines for Te Hono o Te Kahurangi evaluative quality assurance for further information.

The New Zealand Qualifications Framework

The NZQF (PDF, 606KB) is designed to optimise the recognition of educational achievement and its contribution to New Zealand’s economic, social and cultural success.

The NZQF will list qualifications that:

  • convey the skills, knowledge and attributes a graduate has gained through completing a qualification
  • encourage the development of integrated and coherent programmes
  • enable and support the provision of high-quality education pathways
  • enhance confidence in the quality and international comparability of New Zealand qualifications
  • contribute to Māori success in education by recognising and advancing Mātauranga Māori
  • represent value for money, are sustainable and robust.

NZQA is responsible for protecting the integrity of New Zealand qualifications listed on the New Zealand Qualifications Framework and therefore is responsible for co-ordinating the overall system and processes for assuring national consistency.

Mātauranga Māori qualifications

The NZQF lists qualifications where the outcomes directly and specifically address the distinctive needs and aspirations of Māori by advancing Mātauranga Māori and the Māori world-view.

Evaluative quality assurance framework (EQAF)

The Evaluative Quality Assurance Framework uses many different quality assurance activities to reflect the quality of education and education organisations.

The NZQA quality assurance framework integrates quality assurance of entry processes, with the ongoing self-assessment activities an education organisation undertakes to assure itself of the quality of graduate outcomes.

The framework uses an evaluative approach and is underpinned by the following principles:

  • needs based
  • focused on outcomes
  • flexibility
  • quality as a dynamic concept that involves ongoing improvement
  • trust and accountability.

Diagram showing how assuring national consistency of graduate outcomes is incorporated into the Evaluative Quality Assurance Framework

1. Process of assuring consistency

NZQA uses consistency review meetings, facilitated by a consistency evaluator, to allow education organisations to demonstrate their understanding of qualification outcomes and how well the graduates of their programmes meet those outcomes.

The format and structure of these reviews is flexible to respond to the wide range of qualifications, including those meeting the distinctive needs and aspirations of Māori.

Before

Before the consistency review, each participating education organisation gathers evidence, through its self-assessment and its other processes, to show how their graduates meet the outcomes specified in the listed qualification and the approved programme.

The education organisation sends NZQA a self-assessment report that includes this evidence ten days before the scheduled meeting.

During

During the meeting a representative from the education organisation will present a verbal summary of their report.

The consistency reviewer makes an evaluative judgement that rates the quality of the self-assessment and supporting evidence each individual education organisation provides about the consistency of its graduates in relation to the graduate outcomes of the qualification.

The active and effective participation of education organisations in the consistency review meeting assists the consistency evaluator in this evaluation.

After

Each education organisation will receive a rating of ‘sufficient’ or ‘not sufficient’.

1.1. Following the review

It is expected most education organisations will provide convincing self-assessment with sufficient supporting evidence during the review meeting and receive a rating of ‘sufficient’.

Education organisations unable to do so will have an opportunity to submit further evidence that the organisation is sufficient (i.e. their graduates meet the graudate outcomes).

  • NZQA will not make the final decision on consistency of the qualification will not be made until they have considered the further evidence.
  • An interim rating of ‘not sufficient’ may be changed to ‘sufficient’ as a result of the further evidence.
  • NZQA may provide recommendations to help the organisation to improve evidence.

NZQA will take further action if organisations repeatedly cannot provide convincing self-assessment evidence.

1.2. Consistency review schedule

NZQA develops and publishes a schedule of qualifications for consistency review annually in conjunction with the relevant qualification developers.

All organisations that have awarded the New Zealand qualification scheduled for a consistency review are required to participate in the scheduled review.

NZQA will allocate appropriate reviewers to conduct the reviews.

If an education organisation has delivered an approved and accredited Te Hono o Te Kahurangi programme that leads to the relevant qualification, the allocated reviewer will have expertise in te reo Māori, tikanga Māori and applying Te Hono o Te Kahurangi evaluative principles.

2. Evaluative approach

The decisions about the quality and sufficiency of evidence, supplied by each education organisation and the consistency of the qualification are made using an evaluative approach. Confidence in the overall standard or level that has been achieved is implicit in this approach.

The evaluative methodology enables conclusions about quality, value and importance to be reached on a transparent and robust basis. The approach requires consistency reviewers to:

  • be explicit about the evidence on which judgements are made as well as the logic of their interpretation (i.e. the evidence does not speak for itself), and
  • make the outcome available in a written report that is clear and readily understood.

There are two steps in the process of reaching the decision about the consistency of the qualification.

2.1. Steps in the consistency review process

Conclusions at education organisation level

The first step is the evaluation of the self-assessment and supporting evidence that the individual education organisation provides to the consistency review to show how well its graduates meet the graduate outcomes of the qualification.

Education organisations must be able to demonstrate with high quality, convincing evidence that their graduates demonstrate the graduate profile of the qualification.

Using the evaluation question

Consistency of graduate outcomes is determined by answering the evaluation question:

How well does the self-assessment and supporting evidence provided by the education organisation demonstrate that its graduates match the graduate outcomes at the appropriate threshold?

The question applies to each individual education organisation and will be answered in the context of the graduate profile of the individual qualification that is the subject of the consistency review.

The answer to the question is informed by the following criteria:

  • the nature, quality and integrity of the self-assessment and supporting evidence presented by the education organisation
  • how well the organisation has analysed, interpreted and validated all the available evidence, and used the understanding gained to achieve actual or improved consistency
  • the extent to which the education organisation can reasonably justify and validate claims and statements relating to the consistency of graduate outcomes, including as appropriate, in relation to other providers of programmes leading to the qualification.

The conclusion, based on the answer to the evaluation question, is a judgement about the extent and validity of the self-assessment and evidence the education organisation uses to demonstrate how well its graduates meet the graduate outcomes.

Rating the answer to the evaluation question in relation to each education organisation

Rubric one sets out the expected levels of performance in relation to the evaluation question for an individual education organisation.

In the context of the rubric – ‘good evidence’ is defined as:

  • relevant to answering the evaluation question
  • obtained from more than one source (i.e. is corroborated or triangulated)
  • of more than one type (e.g. quantitative and qualitative data)
  • making sense in the context of the qualification and the education organisation programme.

The quality of the answer to the evaluation question is rated according to the rubric.

Rubric one: Performance criteria for rating answers to the evaluation question for individual education organisations

Performance criteria

Sufficient

Self-assessment and supporting evidence is sufficient to demonstrate consistency of graduate outcomes

ALL the following:
- Effective self-assessment is supported by good evidence (as defined above) to show that graduates meet the graduate profile of the qualification

- Sufficient self-assessment supporting evidence that the education organisations have taken all reasonable steps to ensure their graduates match the graduate outcomes

- The nature, quality and integrity of the evidence justifies the education organisation’s judgements about graduates meeting the graduate outcomes

- Any areas of weakness in the evidence or judgements are not serious and are being effectively managed or improved

Not sufficient

When ANY of the above are not met, the self-assessment or supporting evidence is not sufficient to demonstrate consistency of graduate outcomes

The critical outcomes are determined during the assuring consistency process, when the participants develop an accepted “threshold statement” for the qualification.

The essential evidence requirements are related to the qualification specific “threshold statement”’.

Conclusions about the qualification across participating education organisations

A consistency review reaches a conclusion about how well graduates match the graduate outcomes of the qualification, at the appropriate threshold, across all education organisations providing programmes leading to the qualification.

The conclusion is based on synthesising the evidence available from participating education organisations. It is an overall, well-informed professional judgement of the national consistency of graduate outcomes relating to an individual qualification.

The decision on the national consistency of graduates of the qualification is based on the synthesis of the answers to the evaluation question answered for each education organisation participating in the consistency review. The performance criteria for reaching a conclusion about the national consistency of the achievement of graduate outcomes are set out in rubric two.

A decision on the consistency of the graduates of the qualification will be made once:

  • there is confidence in the agreed threshold, and
  • education organisations have had the opportunity to demonstrate the quality and sufficiency of their self-assessment evidence if this wasn’t available at the time of the review.

Rubric two: Criteria for judgements about the national consistency of graduate outcomes of qualifications across all education organisations providing relevant programmes

 

Performance criteria

National consistency confirmed

Self-assessment and supporting evidence confirms graduate outcomes are being achieved at a consistent and appropriate threshold

ALL the following:

- Sufficient convincing self-assessment evidence across education organisations that graduates meet the graduate outcomes of the qualification

- Sufficient self-assessment evidence that the education organisations have taken all reasonable steps to ensure their graduates match the graduate outcomes

- The nature, quality and integrity of the evidence justifies confidence in the education organisations judgements about graduates meeting the graduate outcomes

- Areas of weakness in the evidence are not serious and are effectively managed by relevant education organisations

National consistency not confirmed

When ANY of the above are not evident, the qualification may be judged ‘National consistency not confirmed’, the self-assessment or evidence indicates significant questions about the consistency of graduate outcomes

2.2. Cost

Travel and other expenses for participating in consistency reviews are the responsibility of the participating education organisations.

NZQA annually collects a fee per graduate from education organisations. The fee is intended to cover the costs of managing the overall system.

The fee is published on the NZQA fees page and is periodically reviewed.

3. Responsibilities in the consistency review process

3.1. Qualification developers

The qualification developer is responsible for:

  • determining, in conjunction with relevant stakeholders, the examples of convincing evidence required for demonstrating consistency for the qualification (this information is included in the listing details of the qualification)
  • clarifying the evidence requirements for assuring national consistency in the qualification when requested by programme developers and other stakeholders
  • advising the consistency reviewer as a subject matter expert, on the qualification outcomes and contributing to the ‘threshold statement’ discussion
  • assisting with arrangements for consistency reviews
  • collating feedback from the assuring consistency process to inform qualification review.

3.2. Education organisations

The education organisation is responsible for:

  • participating in periodic events for assuring national consistency
  • consulting with the qualification developer (the organisation that listed the qualification) as their programme is developed
  • ensuring that all the graduates have achieved the graduate outcomes to an equivalent and appropriate standard
  • demonstrating, in a written self-assessment, with supporting evidence, how graduates meet the qualification outcomes
  • in response to the evaluation question providing an oral summary of evidence that supports their self-assessment
  • ensuring that the self-assessment and supporting evidence they provide is sufficiently representative, both of the graduates, and of the range of knowledge, skills and attributes within the graduate outcome statement
  • responding to and complying with recommendations arising from the consistency reviews.

3.3. Consistency reviewers

Consistency reviewers are responsible for:

  • working with the range of qualification developers and programme owners on behalf of NZQA
  • using an appropriate approach when working with qualifications developed to meet the needs and aspirations of Māori learners
  • using a Te Hono o Te Kahurangi kaupapa lens when working with education organisations with programmes approved and accredited through this framework
  • preparing for and facilitating the assuring consistency event
  • identifying good, quality, convincing organisational self-assessment and supporting evidence of graduates meeting the graduate outcomes
  • leading and facilitating the discussion and guiding the participants to develop the ‘threshold statement’ for the qualification
  • using an evaluative approach to decide and report on, the extent to which graduates from different programmes are achieving qualification outcomes
  • writing the consistency review report listing the ratings for each education organisation and the reasons for their decision
  • the final judgement on national consistency of the qualification, assisted as required, by subject matter experts
  • making robust decisions and reporting appropriately to NZQA
  • producing clear and credible written reports that make explicit the evidence used to reach their judgement.

3.4. NZQA

NZQA is responsible for:

  • oversight of the effectiveness of the processes for assuring national consistency of graduate outcomes
  • oversight and co-ordination of consistency reviewers by contracting, training, supporting and monitoring reviewers
  • publishing an annual schedule of qualifications for ongoing national consistency events in consultation with qualification developers
  • publishing the results of assuring consistency events
  • following up with education organisations where their evidence is judged ‘not sufficient’.

4. Sequence for the consistency review process

Here are all the steps that lead up to a consistency review and the steps during and after a consistency review:

  1. New Zealand Qualification developed and listed on NZQF.
  2. Education organisations gain accreditation for programmes that will lead to the award of the qualification.
  3. The education organisations deliver the programmes and award graduates. Education organisations self-assess programmes with a link to the graduate outcomes.
  4. NZQA develops and publishes a consistency review schedule and invites education organisations to review meetings.
  5. Education organisations complete self-assessment and supporting evidence report to answer evaluation question and submit to NZQA ten days before review date.
  6. Education organisations attend consistency review meeting and present summary of evidence.
  7. Consistency reviewer makes evaluative judgement based on the submission from the education organisations.
  8. NZQA publishes conistency review report and informs education organisation of their rating - 'sufficient' or 'not sufficient'.

5. Participating in the consistency review

This section helps applicants understand the consistency review process and prepare for participating in it.

It outlines what happens at each stage and gives targeted guidance for participants.

5.1. Preparing for the consistency review

Steps in preparing for the consistency review

  1. NZQA publishes a schedule of qualifications to undergo consistency review and allocates a consistency reviewer to each event.
  2. Consistency reviews involving organisations that have programmes approved and accredited through NZQA’s Te Hono o Te Kahurangi framework will be allocated a reviewer that has expertise in te reo Māori, tikanga Māori and applying Te Hono o Te Kahurangi evaluative principles.
  3. Education organisations who have reported graduates for the qualification involved will be sent details of the review no later than two months before the review.
  4. Each education organisation with graduates prepares and submits a self-assessment report based on evidence from its self-assessment processes. The report will demonstrate why the organisation is confident its graduates have met the graduate outcomes for the qualification.

Scheduling qualifications for assuring consistency review

Qualifications will typically be scheduled for consistency review 12-18 months before the qualification review date.

The schedule will be published and updated as required.

Arranging the consistency review meeting

The following factors are considered in arranging the details of a consistency review:

  • the number of education organisations with graduates of a programme
  • the geographical spread of the education organisations
  • identifying the approach required, in particular, using a specific approach for qualifications designed to meet the distinctive needs and aspirations of Māori (including the selection of a suitable reviewer)
  • identifying the appropriate mechanism for conducting the review (i.e. a cluster meeting at a specific venue, teleconference or videoconference or through suitable electronic media).

Education organisation report

Education organisations that have reported graduates are required to prepare and submit a self-assessment report that addresses how they know that graduates have met the qualification graduate outcomes.

The process involves:

  • reporting on the evidence collected during self-assessment activities to support conclusions and judgements
  • reviewing the self-assessment evidence to enable organisations to answer the evaluation question:

How well does the self-assessment and supporting evidence provided by the education organisation demonstrate that its graduates match the graduate outcomes at the appropriate threshold?

Education organisations should:

  • explain why they are convinced that their graduates have met the graduate outcomes
  • justify the nature, quality and integrity of the supporting evidence
  • identify any gaps in the evidence
  • identify areas of possible improvement
  • determine a self-rating.

The education organisation uses the consistency report template.

Early consistency reviews

Qualifications may be subject to a consistency review earlier than scheduled, if there are concerns about any of the following factors:

  • the qualification has had a previous judgement of ‘National consistency not confirmed’
  • an unexpectedly high volume of graduates from the qualification
  • the qualification involves high risk or high-profile activities
  • a request from qualification developer or another key stakeholder
  • an identified risk or issues or concerns about the qualification
  • a diverse range of qualification awarders
  • multiple education organisations are assessing / participating in the same approved programme.

5.2. The consistency review meeting

Steps for the consistency review meeting

  1. Review meetings will be facilitated by a reviewer allocated by NZQA.
  2. Participants make their own arrangements to attend the consistency review meeting.
  3. Participants verbally present their evidence and summary self-assessment report at the meeting.
  4. The reviewer facilitates the review meeting, assists the participants to develop the threshold statement and reports the results to NZQA.

Who needs to participate?

All education organisations with graduates of the qualification must participate. Each education organisation must be ready to send authorised representatives who have sufficient knowledge of the qualification graduates to present and discuss self-assessment and supporting evidence on behalf of their organisation.

The qualification developer is a mandatory participant in the consistency review. They can provide clarification and guidance on the standard expected by employers and industry.

Expectations of the consistency review

Successful reviews will be conducted in a spirit of co-operation and collaboration – kotahitanga, manaakitanga and whanaungatanga - between the participants.

The consistency review will take an educational and evaluative approach.

Consistency review meeting

Each participating education organisation will present a summary of their self-assessment evidence to demonstrate why they are confident that graduates have achieved the graduate outcomes for the qualification.

There are three distinct activities to be undertaken by the review:

a. Present the summary report

The first task is to systematically review, compare and discuss the self-assessment evidence presented by each education organisation. Each participating education organisation will explain how their evidence demonstrates that graduates meet the graduate outcomes.

b. Use the graduate profile and the range of evidence to agree the threshold to be achieved

The second activity in the review is to develop a “threshold statement” for the qualification under consistency review. The aim is to understand and agree the content of the graduate outcomes all stakeholders can reasonably expect education organisations to have evidence of. The intent of the meeting is to connect the graduate outcome statement with the convincing evidence the education organisations could gather.

The threshold includes a clear definition of the level of skill, knowledge and attributes required. Some parts of the threshold may be very similar to aspects of the graduate outcome statement, but the threshold is focused on the current cohort of graduates and the skills, knowledge and attributes it is reasonable to expect the education organisations to be able to have evidence of.

This part of the meeting requires the participants to consider and agree on statements that describe the outcomes of the qualification and the essential self-assessment evidence related to these. These statements are used by the reviewers when making their judgements on the nature, quality and integrity of education organisations evidence.

Depending on the size of the group this activity could be done with all participants or in small groups.

c. Reflect on their initial decision about the consistency of their graduates

At the end of the review meeting, each participant will be asked to reflect on the quality and sufficiency of their self-assessment and supporting evidence and identify:

  • what they will continue to do
  • what they will change in their self-assessment and supporting evidence in the future
  • any change to their initial rating on the evaluation question – sufficient/not sufficient.

The consistency reviewer will retain all material provided to NZQA for the review meeting. This material is confidential to NZQA.

Examples of sources of evidence

The individual education organisation’s self-assessment is the primary source of evidence. Conclusions should be supported with multiple sources of convincing evidence: for example, triangulation between what the education organisation knows about their programme, what the graduates say and what the end user says.

Self-assessment evidence should clearly link to the graduate outcomes and may reference:

  • employer feedback
  • graduate feedback
  • community feedback surveys
  • whānau, hapū, iwi feedback
  • destination or end user data (including feedback from the providers of the graduates next level of study)
  • portfolios of work or capstone event reports
  • external benchmarking activities
  • results of internal and external moderation of assessment
  • other relevant and reliable evidence.

Methods of validation of the self-assessment and supporting evidence may include:

  • comparison with formally established internal or external benchmarks or expectations, or professional, licensing or discipline standards
  • data on employment outcomes or progression to further training over time
  • comparison with similar graduates from other education organisations.

One qualification awarder

When there is only one organisation awarding the qualification, there is still a requirement to assure national consistency of the graduate outcomes.

This independent assurance is important if the qualification is offered, for example, in multiple workplaces.

An arrangement will be made that is appropriate to the context and circumstances that apply to the organisation.

Alternative mechanisms

For some qualifications, an existing panel or other body such as a registration body, has been established for the purposes of moderation or other quality assurance. In these instances, NZQA may, in discussion with the qualification developer, determine the relationship between assuring consistency and existing arrangements.

The mechanism will be agreed on a case-by-case basis.

5.3. Reporting the results of consistency review

Steps for reporting the results of a consistency review

  1. Consistency reviewer reports to NZQA at the completion of the review meeting(s) for the qualification.
  2. NZQA reviews and accepts the report then publishes either an interim or a final report on the NZQA website.
  3. NZQA follows up with individual education organisations and the qualification developer on specific issues or recommendations identified during the review.

The consistency review report

The report will address the overall quality and sufficiency of the self-assessment and supporting evidence provided for the reviewer to reach a judgement of ‘National consistency confirmed’ or ‘National consistency not confirmed’ on the qualification outcomes.

The decision on the quality and sufficiency of self-assessment and supporting evidence for each education organisation (sufficient/not sufficient) will be included in the report the reviewer provides to NZQA. The report is quality assured internally by NZQA prior to publication.

The report produced by the reviewer for NZQA will include discussion of education organisations that:

  • have participated in the process and been judged ‘sufficient’
  • those who have engaged but their self-assessment information or evidence is not sufficient
  • those not engaged in the process.

The focus for the report is on improving both the qualification and the education organisation’s understanding of the outcomes of the qualification.

The report will also include:

  • the threshold developed at the review meeting
  • an overview of the self-assessment and evidence provided by the education organisations
  • examples of the convincing and credible evidence used by the reviewer in their evaluation
  • findings and recommendations on improvements to the qualification: for example, there may be evidence that some outcomes are not fit-for-purpose based on the supplied evidence
  • examples of good practice.

Following the consistency review, NZQA will publish the names of education organisations where the decision is that the self-assessment and evidence was sufficient.

The names of education organisations that were not able to present sufficient convincing evidence at the consistency review will not be listed. These organisations will have a defined period after the consistency review meeting to present further evidence, or to demonstrate through an organisational improvement plan how they will address their self-assessment evidence issues in a prompt, agreed timeframe.

Follow-up actions

If the final decision in relation to the qualification is ‘National consistency not confirmed’ then the next consistency review or qualification review may be scheduled.

It is NZQA’s responsibility to follow up with individual education organisations where the decision on the evidence is ‘not sufficient’ or where education organisations do not engage in the consistency review process.

Possible actions include:

  • requiring a review of the accredited programme
  • NZQA monitoring of the programme with a focus on assessment practice
  • NZQA investigation of the education organisation
  • imposing conditions on education organisation accreditation, or other action, on confirmation that the organisation is not complying with the relevant Act or NZQA Rules.

Further information

If you have any questions or would like any more information, please email consistency@nzqa.govt.nz

 
Skip to main page content Accessibility page with list of access keys Home Page Site Map Contact Us newzealand.govt.nz