How NZQA evaluates educational performance in external evaluation and review

In 2014, NZQA introduced clearer reporting expectations for evaluating educational performance in external evaluation and review.

NZQA’s judgements about educational performance are primarily formed through a robust evaluation of learner achievement data and information about outcomes (key evaluation questions one and two – 'how well do learners achieve?', and 'what is the value of the outcomes for key stakeholders, including learners?').

An evaluation team must be able to demonstrate in a report how they reached their conclusions, and cite the evidence used to support the judgements that have been reached.

For NZQA to be confident in educational performance, your organisation will need to be able to credibly demonstrate that a practical focus on learner achievement, and learner and stakeholder outcomes, drives organisational self-assessment and is directly used to maintain and improve your organisation’s educational performance.

NZQA expectations for evaluating educational performance

EER is informed by available evidence, including the tertiary education organisation’s (TEO) own achievement and outcomes data, and any other relevant external data that is available. Each TEO will be asked to demonstrate how it monitors and improves its educational performance through self-assessment.

TEO claims about achievement, performance, or outcomes must be of good quality and capable of independent verification as valid.

NZQA specifically enquires into the educational achievement of both Māori and Pasifika learners as part of EER. If an organisation does not have any Māori and/or Pasifika learners enrolled, this will be stated in the report.

All EER reports will include, at minimum, direct evaluative commentary on:

• Learner achievement of relevant cohorts and/or groups within focus areas and at TEO level (evidenced through whole-of-TEO data)
• Māori learner achievement and Pasifika learner achievement, using disaggregated data alongside overall TEO results to inform conclusions
• Learner and stakeholder outcomes relevant to your organisation
• The range and quality of available performance data, including any significant gaps in that data
• Steps taken to objectively verify educational performance
• The use of achievement and outcomes data to understand educational performance in the organisation and to take informed steps to maximise achievement and outcomes
• Relevant evaluative commentary on the contribution of needs assessment, teaching quality, the quality of learner support, and the impact of governance and management in supporting and improving educational performance.

TEOs will need to have good evidence regarding:

• What counts as educational performance, learner achievement, and learner and stakeholder outcomes in the organisation
• How, and what, information or evidence is gathered on achievement and outcomes, and the quality of that information
• How this information/evidence is interpreted and understood within the TEO
• How the understanding gained is used in organisational decision-making
• The practical steps taken as a consequence of this understanding to maintain areas of good performance and to improve weaker aspects identified
• What the overall impact of these efforts has been on learner achievement and outcomes, and the evidence of that impact (i.e. evidence of improvement)
• In the case of Māori learner achievement and/or Pasifika learner achievement (or any other relevant identified group within the TEO), how the above steps have been applied in the context of that particular group to maximise achievement and outcomes.

What this means in practice for TEOs

EER teams will use the flexibility available in the EER process to modify the approach to best reflect the context of each TEO through the scoping and on-site enquiry processes.

The quality, validity and reliability of the following will be a focus in all EERs:

• Learner assessment (as critical evidence of achievement) including external data such as the Tertiary Education Commission’s Educational Performance Indicators (EPIs) (where relevant)
• Information about relevant learner and stakeholder outcomes, including any external or new data that becomes available
• The analysis and interpretation of such evidence
• The use of the understanding gained to maintain and improve achievement and outcomes

TEOs should think carefully about what data is relevant to them from sources such as the EPIs (for Student Achievement Component-funded organisations), and how they use it to understand and improve performance. TEOs will also be asked about other local data used to understand and improve achievement and outcomes.

Other sources of data

EER will not rely solely on a TEO’s own data, however, and NZQA will integrate the information on performance available to NZQA through other sources into the evaluative judgements (e.g. Risk, Approvals, Moderation business units within the Quality Assurance Division and external agencies such as the Tertiary Education Commission, Ministry of Education, Immigration New Zealand, Ministry of Social Development, etc).

NZQA’s EER reporting style will be as direct and unambiguous as possible, with clear and consistent explanation of the reasoning leading to the conclusions in the report. Any significant gaps in a TEO’s ability to demonstrate its educational performance observed by an evaluation team, including evidence of key processes, achievement and outcomes, will be stated within the EER reports.

 Māori learner achievement

TEOs should be able to discuss and answer questions relating to Māori learner achievement, where relevant:

• How many Māori students are enrolled at the TEO?
• What courses are they enrolled in?
• How does their achievement compare with that of other students?
• What is the overall quality of outcomes for Māori and how do you know?
• How effectively are Māori learners supported to be successful?

 Pasifika learner achievement

TEOs should be able to discuss and answer questions relating to Pasifika learner achievement where relevant:

• How many Pasifika students are enrolled at the TEO?
• What courses are they enrolled in?
• How does their achievement compare with that of other students?
• What is the overall quality of outcomes for Pasifika learners and how do you know?
• How effectively are Pasifika learners supported to be successful?

 Verifying/validating educational performance

NZQA is interested in how well TEOs can justify claims relating to learner achievement and outcomes. Possible means of validation may include:

• Comparison with similar providers/programmes
• Comparison with formally established internal or external benchmarks or expectations, or professional or disciplinary standards
• Monitoring and analysing individual/or group achievement data and outcome results over time, i.e. monitoring the educational value-add/progress of learners
• By time-series analysis, i.e. tracking results over years or programme cycles to identify patterns and trends
• Other locally developed approaches that may be fit for purpose in the context