NZQA's approach to programme monitoring at levels 1-6 and level 7 diplomas

The purpose of this programme monitoring information is to clarify NZQA’s approach to monitoring programmes at levels 1-6 of the NZQF, and level 7 diplomas.

NZQA’s approach to programme monitoring

After a programme is approved, NZQA may carry out a programme monitoring activity to ensure it is being delivered as approved.

To maintain accreditation, all TEOs with approved programmes need to continue meeting the requirements set out in Rule 12.1 and Rule 12.2 of the NZQF Programme Approval and Accreditation Rules 2018.

The requirements state that to maintain accreditation, TEOs must ensure that the criteria set out in Rule 4.1 and Rule 6.1 continue to be met:

Criteria to maintain programme approval Rule 4.1
Criteria to maintain accreditation Rule 6.1

To monitor programmes, NZQA gathers and evaluates evidence relating to the relevant approval and accreditation criteria.

This includes a detailed review of assessment practice for the programme.

NZQA determines an overall rating for the programme

NZQA uses the evidence it gathers to answer the key evaluation question (KEQ) for programme monitoring:

To what extent does the evidence provided demonstrate that programme delivery meets approval and accreditation criteria?

After answering the question, NZQA determines an overall rating for the programme.

How does NZQA select programmes and TEOs for monitoring?

NZQA takes an intelligence-led, risk-based approach to programme monitoring: e.g.

  • Patterns of concerning performance identified in programmes across the tertiary education sector
  • TEO-specific issues such as poor programme delivery and/or assessment practice; a change of ownership or governance
  • Programmes with specific risk factors such as high growth rates, low fees.

The 2018-19 priorities for programme monitoring

  • Business and Management diplomas level 5 and level 6
  • New Zealand Certificate in English Language (Academic) Level 4
  • Level 7 diplomas

Who monitors the programmes?

Monitors may be NZQA staff, external (contracted) monitors, or a mixture of both.

All monitors have in-depth knowledge and experience in assessment and moderation, curriculum design, and programme development and delivery.

NZQA advises the TEO who the monitors are before the monitoring.

Programmes at levels 1-6 Two NZQA monitors (Evaluators from the Monitoring and Assessment team)
Level 7 diplomas One external monitor (contracted subject expert), one NZQA monitor

The monitors are responsible for:

  • leading the programme monitoring activity
  • drafting the monitoring report
  • quality assuring moderation reports.

NZQA also contracts moderators to verify assessment decisions in relation to programme learning outcomes. Moderation findings are incorporated in the overall programme monitoring report.

How does NZQA choose moderators?

NZQA moderators are contracted based on their subject area expertise, assessment and moderation experience and educational qualifications.

NZQA has a robust conflict of interest process to ensure moderators are neutral.

How does a programme monitoring visit work?

NZQA notifies the designated contact person at the TEO

NZQA contacts the person listed as the TEO’s Quality Assurance contact on the NZQA website. The TEO can nominate another staff member to be the contact person if they wish.

This person will be NZQA’s contact for confirming the programme monitoring activity and, as required, confirming the visit agenda, supplying documents, and liaising with the monitors on the day of the visit.

Before the visit

Pre-visit documentation

Before the programme monitoring visit, NZQA requests documentation associated with programme delivery and quality management. See the Appendix to the notification letter for a list of what documentation NZQA needs.

NZQA also requests academic results so it can select marked learner work for verification of assessment decisions. NZQA might select learner work before the visit or discuss selection at the visit and select learner work after the visit.

If the programme monitoring activity is for moderation only, NZQA may request marked learner work without a visit.

NZQA selects assessments with the heaviest course weighting and assessments conducted later in the course, if possible.

The agenda for a programme monitoring visit

The monitors interview key people involved in the programme. These usually include academic staff, programme management and learners, but it is different for each visit.

NZQA sends the TEO a draft agenda before the visit. The TEO can adjust the timings or rearrange the order of interviews and class observations for minimal disruption to learners on the day.

Visits typically take one day to complete. For large TEOs, or multiple delivery sites, two days may be required.

On the day of the visit

Interviews during the programme monitoring visit

Who might be interviewed?

  • Quality and academic manager
  • Programme leaders responsible for the programme(s)
  • Academic records/administration staff
  • Key academic staff members for the programme(s) (e.g. programme delivery, teaching staff, assessors)
  • Learners currently enrolled on the programme(s).

What will NZQA ask them about?

Questions for academic manager, quality manager, programme leader, programme developer, and academic staff members may cover:

  • programme length and structure
  • changes to term/semester/block delivery
  • timetables (staff/internal and published)
  • approved learning hours
  • programme regulations
  • integration of practical and work-based components
  • rules for award of qualification
  • moderation systems
  • review of programme performance and self-assessment.

Questions for academic registrar and administration staff may cover:

  • English language requirements for international students
  • credit recognition and transfer
  • results and certification.

NZQA may select a sample of student enrolment files to check admission processes are being applied appropriately. The monitors bring this list with them on the day of the visit.

Questions for academic staff members may cover:

  • qualifications
  • professional development
  • assessment materials
  • moderation practice
  • plagiarism prevention, detection and management
  • assessment methodology.

Questions for support staff may cover their:

  • support services
  • educational resources
  • pastoral care.

Questions for current learners may cover their:

  • learning experience
  • assessment experience
  • programme requirements and expectations.

Recording the interviews

The monitors ask permission to record staff interview sessions before the discussion starts. TEOs can request copies of the recordings. The monitors do not record interviews with learners.

NZQA uses the recordings only to assist in writing the programme monitoring report.

NZQA destroys the recording six months after finalising the report.

Monitors observing TEO classes

Monitors ask to sit in on a class. The monitors are checking that the activity reflects the programme documentation and the information that the TEO publishes about the programme.

For example, that learners are attending as required, English is the medium of instruction, and qualified staff are leading the teaching activity.

After the visit

NZQA provides the TEO with a draft programme monitoring report, including a summary of moderation results, within 30 working days from receiving acceptable learner samples.

If there is any delay NZQA will notify the TEO.

TEO and NZQA follow-up actions

If NZQA has concerns with the integrity of qualifications awarded to learners, it will decide if these concerns require immediate remedial actions or specific actions in a short timeframe.

If the report identifies areas for improvement, NZQA may agree a plan of action with the TEO. The plan will be re-assessed at a future date.

NZQA may also leave the TEO to address improvements through normal self-assessment processes.

Student data and learner work held by NZQA

NZQA holds all submitted materials, including learner work samples and enrolment data, securely and only uses it for completing the programme monitoring report.

NZQA holds copies of learner materials on file for two years (i.e. to finalise the report and allow any follow-up). Then the records are destroyed.

Learner IDs (not names) are used instead of names in the moderation reports.

How does moderation for programme monitoring work?

Moderation for programme monitoring consists of verifying assessment decisions in relation to the programme learning outcomes.

  • The learning outcomes are those approved by NZQA against the graduate profile and set out in the TEO’s programme documentation.
  • The moderators seek to verify the TEO’s assessment decisions that learners have met all the learning outcomes.
  • Moderators will also comment where the assessment tasks do not provide learners with the opportunity to achieve the programme learning outcomes.

Ensuring that TEOs use effective assessment practice in their programmes is integral to assuring that any qualifications awarded are credible. For this reason, NZQA only moderates learner work that has been awarded a pass/achieved grade.

Moderation reports

NZQA moderators detail their findings in moderation reports. The moderation reports include feedback on whether there is sufficient evidence that:

  • the learner has achieved the learning outcome(s)
  • the learner work is authentic
  • assessment practice allows learners the opportunity to meet the learning outcome(s).

The programme monitoring report

The programme monitoring report shows if the delivery and assessment of the programme continues to meet the NZQA-approved documentation and the relevant criteria. The programme monitoring report also identifies any rule breaches.

The monitor(s) completes the draft programme monitoring report and asks for feedback from the TEO. After considering this feedback, NZQA finalises the report.

Focus and structure of the draft programme monitoring report

NZQA summarises programme monitoring outcomes in a report with sections relating to the following criteria.

Report sectionApproval and accreditation criteria (Met or Not Met)
1. Programme structure and delivery 4.1 Criterion 3 Delivery methods
The delivery methods are adequate and appropriate, given the stated learning outcomes for the programme.
4.1 Criterion 5 Regulations
There are clear, relevant, and appropriate regulations that specify requirements for: programme length and structure.
2. Programme regulations 4.1 Criterion 5 Regulations
There are clear, relevant, and appropriate regulations that specify requirements for:
- admission
- credit recognition and transfer
- recognition of prior learning
- integration of practical and work-based components
- normal progression within the programme.
18.1 – 18.6 English language requirements for international students
3. Assessment and moderation 4.1 Criteria 5 Regulations
There are clear, relevant, and appropriate regulations that specify requirements for: assessment procedures, including authenticity of student work.
4.1 Criteria 6 Assessment and moderation
Assessment methodology is fair, valid, consistent and appropriate given the stated learning outcomes. There is an effective system for moderation of assessment materials and decisions.
6.1 Criterion 1 Assessment and moderation
The institution has the capability and capacity to ensure assessment materials and decisions are fair, valid, consistent and appropriate for the level, given the stated learning outcomes.
4. Resources 6.1 Criterion 2 Resources
The institution has the capability and capacity to support sustained delivery of the programme through appropriate academic staffing, teaching facilities, educational and physical resources, and support services.
5. Programme review 4.1 Criterion 7 Assessment and review
The institution:
- assesses the currency and content of the programme
- has adequate and effective processes for the ongoing review of the programme, taking account of the results of any review of the qualification
- has adequate and effective processes for monitoring the quality of outcomes for learners and other stakeholders, and for reviewing programme regulations and content
- updates the programme accordingly.
6.1 Criterion 4 Assessment and review
There must be adequate and effective review of programme performance and the institution’s capability to support the programme.
There must be monitoring of improvement following review, and processes for determining whether the programme should continue to be delivered.

Section 3 of the report - Assessment and Moderation - covers the NZQA moderator’s review of assessment practice for the programme. The programme monitoring report summarises the moderation review findings, and the detailed moderation report is added as an appendix.

The programme monitoring report recognises and acknowledges effective delivery, progress and achievement. NZQA also acknowledges development during the programme’s first delivery and the TEO’s efforts in continuous improvement.

Reporting the overall rating of the programme monitoring report

Based on the report findings, NZQA proposes an overall rating to answer the programme monitoring KEQ.

There are three possible overall ratings for programme monitoring:

Programme monitoring outcomeIndicators
Meets programme criteria overall - Programme delivery continues to meet overall approval and accreditation criteria
- Education organisation is proactively managing programme quality
- Improvements have been self-identified and implemented
Meets some programme criteria - Programme delivery has some weaknesses
- Education organisation is identifying and addressing some gaps or weaknesses
- Some improvements required to meet approval and accreditation criteria
Does not meet programme criteria - Programme delivery has serious weaknesses
- Education organisation is not identifying or addressing gaps or weaknesses
- Significant improvements required to meet approval and accreditation criteria

If the overall outcome is ‘meets some’ or ‘does not meet’ programme criteria, the report will indicate how serious NZQA considers the findings to be.

Responding to the draft programme monitoring report

NZQA asks the TEO to check the programme monitoring report and point out what they believe to be any errors of fact. This may include disagreeing with the moderation findings.

Where applicable, the TEO may include in its response an outline of what it will do to address any concerns or areas for improvement raised in the programme monitoring report.

NZQA evaluates the TEO response and issues the TEO with a final programme monitoring report, including any amendments.

Who sees the programme monitoring report?

The primary audience for the programme monitoring report is the TEO’s management and programme leadership. It is not intended for the learners whose work was moderated.

The following Quality Assurance Division (QAD) business units view the programme monitoring report in both draft and final states:

  • Approvals and Accreditation
  • Consistency
  • External Evaluation and Review
  • Quality Assurance Māori
  • Risk.

Programme monitoring reports are not published on the NZQA website.

Impact on TEO applications for new approvals

Where the report identifies serious quality assurance concerns, NZQA may consider putting any new approvals on hold until the identified issues are resolved.

Further information

For further information on the programme monitoring process or ratings, contact the Monitoring and Assessment team at programmemonitoring@nzqa.govt.nz.

 
Skip to main page content Accessibility page with list of access keys Home Page Site Map Contact Us newzealand.govt.nz