NZQA programme monitoring of business programmes at Levels 5-6 and level 7 diplomas has identified a range of programme delivery issues, particularly in some providers offering programmes to the international student market.

The NZCEL Level 4 (Academic) is recognised for English proficiency entry to higher level programmes, such as those mentioned above:

Forty-four providers hold accreditation for NZCEL Level 4 (Academic) and international student numbers have been increasing steadily over 2018-2019.

In addition to the risk associated with an increase in international student numbers, in 2018 NZQA received anecdotal allegations about some providers’ delivery of the NZCEL from staff, ex-staff and other providers.

These included:

• inappropriate entry
• unqualified staff
• under-assessment
• inappropriate use of recognition of prior learning (RPL)
• inadequate resources
• compressed delivery.

NZQA intends to monitor all providers of the NZCEL Level 4 (Academic) over 2019-2020.

NZQA will give providers at least two weeks notice of a programme monitoring activity taking place.

The order in which providers are notified of programme monitoring is largely driven by operational reasons including availability of contractors. However, trends in student visas are also a factor in selection.

The first 10 providers were selected from across the PTE and ITP sectors to provide a representative group. Factors taken into consideration were:

• provider size
• increase in roll
• category
• length of time delivering the NZCEL
• National External Moderation history.

When you get your notification please note that it may be possible to change either the programme monitoring activity or the date. Please contact Monitoring and Assessment if this is the case.

There are currently two forms of programme monitoring for NZCEL Level 4 (Academic):

Site visit

This involves:

• interviews with staff and students
• QMS document review
• validation of assessor decisions.

The programme monitoring rating is against all criteria set out in Rule 4.1 and Rule 6.1 of the NZQF Programme Approval and Accreditation Rules 2018.

Moderation only

This involves validation of assessor decisions and programme document review, carried out remotely.

The programme monitoring rating is against only 6.1 criterion 1 (assessment and moderation) of the NZQF Programme Approval and Accreditation Rules 2018.

There is more information on NZQA's approach to programme monitoring at levels 1-6 and level 7 diplomas.

Follow-up actions for providers with a ‘Not Sufficient” consistency review rating may include programme monitoring, but this has not occurred with the NZCEL Level 4 (Academic) to date.

The quality assurance focus of assuring consistency and programme monitoring is ‘maintaining quality’ in the non-university tertiary education sector, but in different ways.

Consistency review applies to all NZQF qualifications at Level 1 to 6:

• The focus of consistency review is on comparing graduates in relation to the qualification outcomes.
• Consistency review seeks assurance that programmes are designed and delivered in a manner that ensures the graduates achieve the graduate outcomes of the qualification. Evidence of robust moderation activities, and the results of these, contribute to the education organisation self-assessment the consistency reviewer uses to make their judgement, as does evidence gathered from employers, study destinations and the graduates themselves.
• Consistency review is an evaluative process, considering a collection of evidence from many sources and making a judgement based on the quality and weighting of the evidence supplied.

Programme monitoring applies only to programmes with risk factors:

• Programme monitoring seeks to establish the extent to which a provider is meeting the NZQF Programme Approval and Accreditation Rules 2018.
• NZQA gathers and evaluates evidence relating to the relevant approval and accreditation criteria. This includes a detailed review of assessment and moderation practice for the programme.

Providers with a ‘Sufficient’ consistency rating could potentially have poor programme monitoring outcomes for the same programme, particularly if the evidence from the graduates’ employers, study destinations and the graduates themselves is comprehensive, robust and convincing.

Similarly, because the processes require different evidence, it is also possible that providers with a ‘Not Sufficient’ rating could have good programme monitoring outcomes, particularly if their programme related evidence is strong but evidence from the graduates’ employers, study destinations and the graduates themselves is poor.

Consistency review dates:

• October 2019 - 1881 New Zealand Certificate in English Language (Level 2)
• November 2019 - 1882 New Zealand Certificate in English Language (General) (Level 3)
• May 2020 - 1879 New Zealand Certificate in English (Foundation) (Level 1) and 1880 New Zealand Certificate in English Language (Level 1)
• November 2020 - 1883 New Zealand Certificate in English Language (Academic) (Level 4)

National external moderation and programme monitoring are separate processes.

The main difference is that programme monitoring involves all assessments leading to award of the qualification (unit standard-based and modular-based), the assessment tools as well as the learner work. The other differences are as follows.

National external moderation (NEM)

NEM is a compulsory annual requirement for providers with consent to assess NZQA-managed standards:

• Providers select the learner work for moderation.
• Three (non-graded standards) or eight (graded standards) samples of learner work for each unit standard are required.
• Moderation is carried out to ensure that assessor decisions about assessment standards are consistent nationally.
• Assessment tasks and marking schedules are not typically moderated, only the marked learner work.
• Feedback is given to the provider via moderation reports and an annual summary.
• An action plan is required if moderation reports identify major or a significant number of assessment issues (within a system or across systems in the same moderation year) or recurring or different assessment issues have been identified within or across systems for more than one year.
• NZQA may follow up with a programme monitoring visit.

Note: Providers participating in programme monitoring in 2019 are not required to submit assessment materials for English for Academic Purposes standards for national external moderation. National external moderation for these standards will resume in 2020.

Programme monitoring

Programme monitoring is programme-specific and risk-based. Moderation is usually, but not always, carried out during a programme monitoring activity:

• NZQA selects the learner work for moderation.
• A ‘learner journey’ approach is used for selection i.e. all assessments completed for the purposes of award. Typically, four graduates are selected and approximately 48 pieces of learner work are moderated.
• Moderation is a validation of assessor decisions on learner work against the approved learning outcomes for the programme.
• Detailed feedback is given to the provider on course materials, assessment design, assessor decisions, and sufficiency of assessment for qualification award.
• If serious concerns are raised in assessment practice, NZQA may consider statutory action is required to protect the integrity of qualifications awarded.

As of 1 July 2019, only two programme monitoring activities have been finalised. One was a visit, which resulted in a programme monitoring outcome of “Meets programme criteria overall” and one was a moderation-only activity which found the provider did not meet 6.1 criterion 1 (assessment and moderation).

There are eleven programme monitoring activities in process. Some are with providers for factual accuracy check and some are back with NZQA to take on board feedback on process and findings.

Some of the issues

• Versions of the unit standards have been out of step with the qualification version, e.g. qualification is version 1 and unit standards are version 4, or qualification is version 2 and unit standards are version 3. While not affecting award, the programme is not up-to-date.
• There was a lack of alignment with versions of unit standards, assessment tools, marking schedules and other documents.
• In some cases, providers have used only the five mandatory unit standards to assess all the programme’s learning outcomes (27 out of 60 credits).
• Learners were provided with too many opportunities to resubmit assessments.
• Assessment conditions were not applied appropriately.

And some good findings

• There were some well-designed assessments that aligned with programme learning outcomes and the requirements of the unit standards.
• There were examples of texts, tasks and assessor decisions that demonstrate subject expertise and encourage learning.
• There was clear evidence that some learners were at the required level.

Feedback on the format and tone of the programme monitoring reports has been that they are too negative and do not highlight where practice has met requirements or could be commended.

Monitoring and Assessment are reviewing the reports and accept the view that good practice tends not to be commented on in any detail. This is an area we will seek to improve.

Yes.

The reason NZQA selected a range of providers in the first group to be monitored was to have a range of programmes and assessments available for moderator training. We wanted the contract moderators to have a view of good practice as well as assessment issues.

As part of the NZCEL programme monitoring workstream, NZQA plan to compile some good practice assessment support material for providers.

Monitoring and Assessment will contact providers where moderation has found good assessment materials and design and ask them to share their expertise and/or materials in this area.

This project should commence later in 2019 as more of the programme monitoring is finalised.

Sector feedback from the NZCEL Providers Forum is that the EAP unit standards are not currently meeting the needs of tertiary education organisations. This topic is beyond the scope of this communication, but all NZCEL providers should be involved in the review of the NZCEL qualification suite.

Please provide detailed submissions on any issues you are finding with the unit standards or the qualifications and submit them to NZQA as the standard setter for NZCEL.

Yes. NZQA provides clarification on frequently asked questions.

See also:

• English for Academic Purposes (EAP) Assessment Support Materials (ASM)
• Best Practice Workshops - Assessment & Moderation

The expectation that providers need to include more assessments in their programmes, beyond the mandatory unit standard assessments, was not clearly communicated in the previous version of the Guiding document (PDF, 460KB).

The way it presented the sufficiency requirements was open to misinterpretation in terms of the minimum number of assessments for a programme leading to award of the qualification. It was never the intention to indicate that eight assessments would be appropriate for this programme. See the next question for more detail.

The NZCEL review panel was convened in early 2019 to work with NZQA to correct and clarify the information in the Guiding document. Recent feedback from the sector indicates that the purpose of the May 2019 changes was also not entirely clear. It is hoped that this communication addresses that.

A qualification with mandatory unit standards does impose some design issues. Here is some information that may help.

Programmes leading to the NZCEL will vary in stated learning outcomes leading to the graduate profile, but all must meet the requirements for a 60-credit qualification, deliver 15 credits of learning and assessment for each of the Speaking, Listening, Reading and Writing GPOs, as well as the requirements of all mandatory unit standards.

The mandatory unit standards specify learning outcomes and skill assessments against each of the four graduate profile outcomes of the qualification.

The mandatory unit standards can be used as a point of reference for programme design:

• For the ‘balance’ of credits required, i.e. 27 credits of learning and assessment are recognised by the requirements of the unit standards, meaning approximately half the qualification’s credits need to be covered by the provider's own description of learning and assessment.

• For the minimum number of individual skill assessments, i.e. the unit standards require six skill assessments, (1 listening, 1 speaking, 2 reading, 2 writing) meaning approximately half the skill assessments need to be covered by the provider's own specification of assessment.

The Guiding document

The Guiding document (Graduate Profile Outcomes, pp.27-31) recommends 35 credits from specific English Language unit standards for the ‘balance’ of learning and assessment:

• Speaking (2 additional assessments), Listening (2 additional assessments), Reading (2additional assessments) and Writing (1additional assessment).

The Guiding document sets out additional information on sufficiency that providers need to consider when designing an assessment schedule for the programme. The information provided in the 2018 version related to eight assessments. This information was intended to inform assessment design. It was not intended to set the total number of assessments for a programme leading to award of the qualification.

The information provided in the May 2019 version of the Guiding document clarified this by including for each skill set the expectation that more than the mandatory assessments would be required by a programme assessment schedule: i.e. an additional 2 assessments each for listening and speaking, and an additional 1 assessment each for reading and writing (a minimum of 12 assessments for the NZCEL programme).

These additional assessments are up to each provider to design. They may be similar to the methods used to meet the unit standard requirements or utilise another approach: e.g. naturally occurring evidence/portfolio.

NZQA holds the view that it is reasonable.

Programmes that recognise 60 credits of learning and assessment are often delivered full-time over half an academic year, or two semesters. Six assessments (as required by the unit standards) and another six assessments (of a similar demand) carried out by tests, graded portfolios, marked assignments, or other method, is not a heavy assessment load for a 20-week full-time course leading to a level 4 graduate profile.

The assessment schedule for the level 4 NZCEL (Academic) should prepare learners for further English language study in an undergraduate academic context.

Providers will not be penalised in programme monitoring in relation to the number of assessments. A low rate for validation of assessor decisions, when based solely or largely on the number of assessments, will not be considered a breach of 6.1 criterion 1 of the Programme Approval and Accreditation Rules.

Please apply to NZQA for a change to your programme as soon as practicable.

See also Guidelines for programme approval.

There is no centralised unit or person at NZQA who looks after all aspects of the NZCEL.

NZQA has many different functions in relation to the NZCEL - standard setting, moderation, programme approval etc - so information and services do come from different departments. We acknowledge that the different contact points, navigating the NZQA website and/or accessing all the different information available for the NZCEL can be difficult.

We have decided that Monitoring and Assessment staff will provide a single point of contact for NZCEL providers seeking information or advice from NZQA. Your liaison person will find or coordinate answers from other teams for you.

The contact person for your organisation is listed under ‘Monitoring and Assessment contact’ in your provider profile when you log into NZQA.

If you are transitioning from version 1 to version 2 of the qualification this year:

• Check that the last date for enrolment in programmes leading to version 1 of the qualification allows students to complete the programme by 31 December 2019.

Programmes leading to version 2 of the qualification must be approved by NZQA prior to student enrolment start date.