What is the quality audit process?

This section explains the stages of an audit and how the audit is conducted. Although audits will vary according to the type of organisation, a typical audit involves four stages: planning and preparation/document analysis, fieldwork, reporting, and decision-making. Each audit involves a visit to the head office and the main delivery site. Audits of multi-site organisations will include visiting some of the additional sites.

The audit process

Audit notification

NZQA notifies the organisation of the audit three months in advance, unless the organisation is on a less than one-year audit cycle. For this cycle, the audit outcome letter is also the formal notification of the audit. Each organisation needs to organise the documents to be sent to NZQA to enable the audit planning to start. The section "How to get the most out of an audit" gives more detail on these documents.

Newly registered organisations will have their first audit about six months after the date of registration. This will be advised in the letter confirming registration.

Audit planning

Scoping

Scoping means deciding what the audit will cover and whether to focus on particular areas. Tertiary education organisations (TEOs) are asked to send in their most recent quality management system (QMS), self-review documents and a Provider Update Form. These enable the auditor to begin scoping the audit.

The audit may cover all the areas of the standard or may focus on requirements that need particular attention. This may happen if the organisation has undergone significant change or has a history of non-compliance in certain areas.

The auditor will usually make contact to discuss the audit focus and, if a multi-site organisation, decide on which sites to visit. The auditor will confirm the availability of staff, students and management, and the location of key records and documents.

When scoping is completed and the date is confirmed an arrangements letter is sent six to eight weeks before the audit. The letter includes the scope of the audit.

Document analysis

The second planning task is the auditor's detailed analysis of the QMS and self-evaluation documents. The auditor checks that your QMS has adequate policies and procedures covering all requirements of the standard.

Sometimes, the auditor requests further information, such as student handbooks or prospecti, to assist with preparation for the audit. The auditor will use the self-evaluation document to gain an understanding of where the organisation believes its strengths and weaknesses lie. At the audit visit, the auditor will seek to verify the robustness of the self-review.

Confirmation of arrangements

The arrangements letter specifies:

• dates of the audit visit
• members of the audit team
• roles and responsibilities of the TEO and NZQA
• audit fees.

The letter includes a proposed agenda and list of documents that the auditor may want to look at. It also includes a confirmation of arrangements form, which must be returned by the due date.

The audit visit

The audit visit has the following broad structure.

Introduction

The auditor will need a workspace (ideally close to the key people the auditor will deal with during the visit). Although auditors typically spend some time analysing documents, they also meet with management, staff and learners.

The opening meeting

Anyone may attend the opening meeting, but it is important to have present a management representative and the key contact person. The same people who attend the opening meeting should also attend the closing meeting. The opening meeting usually takes 30-45 minutes. Topics include:

• background to the NZQA audit process
• objectives and scope of the audit
• reporting at the end of the visit
• the post-audit process, including reporting and action plans to address findings
• the audit process. This covers methods of gathering evidence, including document review, audit trails, sampling and interviews; and availability of evidence, including its type, volume and location
• the agenda, including times and location of interviews with staff, students and management or board members
• confirmation of staff to be interviewed
• confirmation of students to be interviewed. The auditor will select them from student lists provided
• confirmation of the contact person. This person will advise the location of documents and records, coordinate interviews and liaise with relevant people in the organisation.

Touring the facilities

The auditor will usually tour the site to become familiar with the layout and facilities. The auditor will ask various questions during the tour and check resources such as the student notice boards and health and safety information.

Gathering evidence

Evidence includes qualitative and quantitative information, documents and records. Other evidence includes information gathered from interviews (face-to-face and by telephone) and direct observation. The auditor will aim to gather enough information to ensure that all findings are sufficiently supported by evidence.

The auditor will seek three sources of evidence - referred to as "triangulation" - usually in the form of documents, records and interviews. The auditor will use the evidence to evaluate the effectiveness of the organisation in delivering education and training against the standard.

The QMS is the underpinning document for the audit. The auditor will have analysed the QMS before the audit, and cross-referenced policies and procedures against the standard. During the audit, the auditor seeks evidence of how well the policies and procedures are implemented.

The above example is very simple. Triangulation can be more detailed when an organisation has complex systems. For example, an organisation may have policies and procedures to manage internal moderation of unit standards across several sites.

In this case, the auditor may employ a sampling method. One such method would involve randomly selecting several unit standards. The auditor would then examine pre- and post-assessment moderation records for units from some of the sites.

Sampling tests whether systems are being followed and are working effectively for the organisation and the learners.

The auditor will track through processes to see how well procedures are being implemented. For example, the auditor may select a sample of student names from an enrolment list. The auditor may then check enrolment and assessment files for these students. This check may include:

• application of entry criteria
• assessment of prior learning
• copies of enrolment documentation
• individual learning plans
• assessments
• recording final results
• credit reporting data
• archiving final results.

The auditor records the evidence seen and heard against the requirements of the standard. As well as recording the sources of evidence, the auditor will take away samples of evidence to further support the audit observations and findings. As the audit progresses, the auditor will discuss any potential findings with the key contact person. A "finding" - i.e. a requirement not met - is an instance where the organisation's performance does not sufficiently meet the standard. Findings are made when there is evidence that the issues are systemic or more than a one-off occurrence.

Quality audit is a 'no-surprises' process. Throughout the audit, management is kept fully informed about how the assessment is progressing. There is every opportunity to present additional evidence to support actual practice.

The closing meeting

The auditor will report back and confirm the findings of the audit. This process includes areas of good performance as well as areas of non-compliance. The findings are made in relation to the requirements of the standard. The auditor will support any findings with the evidence on which they are based. The auditor will discuss the reporting and follow-up processes before leaving.

Reporting

A detailed audit report is written and received by the TEO approximately 20 working days after the audit visit. The organisation is invited to confirm the factual accuracy of the report. This involves checking statements about matters such as student numbers, courses delivered and the evidence cited to support the audit findings.

Confirming factual accuracy involves confirming that the evidence supports good practice and is factual and accurate. Agreement about the findings occurs during the audit visit. It is unlikely that findings will change after the audit visit unless there is evidence that an error was made in interpreting or recording the evidence that lead to the finding.

Any apparent inaccuracies can be discussed with the auditor before the confirmation of factual accuracy form is returned. If the TEO and the auditor cannot agree, the TEO may apply for an independent review of factual accuracy. The reviewer will review the audit file and make a recommendation to the manager.

Once factual accuracy is confirmed the report is finalised and sent with a request for an action plan, if applicable, to address any findings. A response is expected within 15 working days. Failure to do so will influence the audit cycle decision.

Decision-making

The final report will be used by NZQA to decide the organisation's ongoing registration and accreditation status. The summary report will be published on the organisation's page of the NZQA website unless NZQA has been advised that the organisation does not wish to have it published. The published report reflects the findings of the audit and does not include any actions taken since the audit to address findings.

An independent evaluation group makes an audit cycle recommendation to a senior manager based on the findings of the most recent audit and other available information. Audit cycle decisions are made against a set of published audit cycle criteria to ensure consistency.

Once the audit cycle decision is made, the TEO is notified by letter of the month and year of the next audit. The letter includes a record of the basis of the decision. If the TEO disagrees with the decision it can apply for a formal review of the audit cycle.

Page updated: 18 March 2008