Exam Centre Manager / Supervisor report

Evaluation Report 2018 (English, Classical Studies, Media Studies)

1. Introduction

The Exam Centre Manager / Supervisor Survey was designed to measure Exam Centre Manager (ECM) and/or supervisor self-reported experience of invigilating a Digital Pilot examination, including their experience of the dashboard.

Note # 1

Three respondents were both ECM and supervisor at different examination sessions. One survey respondent did not tick any role checkbox. Some ECMs or supervisors may have made more than one response to the single survey offered. We have treated each survey response as representative of the ECM/supervisor's overall examination experience, whether that was of a single examination session or multiple sessions.We do not have a conclusive response rate for this survey.

Of the ECMs/supervisors who supervised the digital examinations, 105 completed the online survey. Of the 105 respondents, 41 were ECMs and 66 were supervisors (See Note # 1). The respondents did not answer all the questions in the survey.

75% (74 of 99) of respondents had been an ECM / supervisor in 2017, and 67% (66 of 99) had managed or supervised a digital examination in 2017.

The survey consisted of 14 questions, one of which was open-ended. The first three questions were for ECMs only.

A summary of the main findings is shown below. Appendix 1 is a count of responses for each closed question. Appendix 2 is a full list of the survey questions.

2. Survey findings

2.1 Overall experience

As stated in footnote 1 below, we do not have a conclusive response rate for this survey. Respondents were asked to rate the relative difficulty of managing digital and paper-based examinations. 44% (44 of 99) of respondents found supervising digital examinations somewhat easier or much easier, which was slightly less than in 2017 when 51% (39 of 76) of respondents had the same sentiment. In 2016 33% (23 of 69) of respondents disagreed or strongly disagreed that supervising digital examinations were more difficult to manage than paper-based examinations.

Note # 2

Students sitting the L1 English Digital Pilot examination experienced a short-term loss of connectivity at 11.39am. NZQA investigated the cause and made system changes for the next and subsequent digital examination sessions so this issue did not recur.

In the open-ended question, some of the less experienced survey respondents expressed anxiety in managing the examinations, but did acknowledge that experience would increase their confidence. Survey respondents also said that their worry about potential issues that might happen during the examination made their overall experience less positive:

"As [ECM] I felt nervous about the digital exam probably because of fear of the unknown more than anything else. It ran like clockwork."

"General management is reasonably easy. It is the first exam that is the most difficult where supervisors are facing the 'unknown' - after L1 English (See Note # 2) they were much more confident."

"When the exams go well, they are great, but any hitch tends to be a big thing."

"The scope for disaster is greater therefore the stress level is higher (for supervising)."

"The more experience the better it goes."

2.2 Exam Centre Manager tasks

95% (40 of 42) of respondents agreed or strongly agreed that the support from NZQA for the management of the digital examinations was adequate. This was similar to 2017 when 97% (38 of 39) of respondents had the same sentiment. In 2016 the result was 85% (51 of 60).

While the survey respondents were generally positive about the support they received from NZQA, some commented about the lack of communication from NZQA regarding technical issues during some of the examinations:

"In Level One English when system went down couldn't get through to support phone and no direct contact made by NZQA to let us know what was happening."

"Need to be advised when [NCEA] server crashes when it happens, not after we have candidates having problems over a period of time."

90% (37 of 41) of respondents agreed or strongly agreed that the support from the schools for the management of the digital examinations was adequate. This was similar to 2017 when 92% (35 of 38) of respondents had the same sentiment. This question was not asked in 2016.

Some of the survey respondents said that their workload has increased because of digital examinations, especially when there are issues with students' digital examination entries:

"I recognised that there was a problem with L3 English students not being entered for digital exam when checking papers and L2 with only half entered for the digital exam, when checking, and alerted PN. This ended up a time-consuming issue with rolls later on."

"As an ECM my work has increased massively since digital were added. A mistake in entering a large group of boys wrongly in L2 English took me hours to sort out when making up rolls etc."

"The time taken using the current website to organise one digital exam with large numbers of candidates takes 10 times as long as any other single paper based exam."

81% (83 of 102) of respondents agreed or strongly agreed that they felt confident managing possible breaches of the rules in the digital examination room. This percentage was lower than in 2017, when 94% (81 of 86) responded with the same sentiment, but slightly higher than in 2016 when 79% (65 of 82) of the respondents had the same sentiment.

98% (39 of 40) of respondents agreed or strongly agreed that they found it easy to set up the room to minimise the risk of breaches of the examination rules. This was greater than both 2017 and 2016 when 87% (33 of 38) and 89% (50 of 56) of the respondents respectively had the same sentiment.

2.3 The dashboard

92% (94 of 102) of respondents agreed or strongly agreed that the dashboard was easy to operate. This is similar to 2017 when 94% (84 of 89) of respondents had the same sentiment, and an increase from 2016 when overall, 52% (43 of 82) of respondents had the same sentiment.

95% (97 of 102) of respondents agreed or strongly agreed that the dashboard was useful in assisting with supervision of the digital examinations. This is similar to 2017 when 99% (87 of 88) of respondents had the same sentiment, and an increase from 2016 when 70% (49 of 70) had the same sentiment.

86% (88 of 102) of respondents agreed or strongly agreed that the dashboard was useful in managing breaches of the rules. This was similar to 2017 when 86% (70 of 81) of respondents had the same sentiment, and an increase from 2016 when 68% (48 of 71) of respondents had the same sentiment.

98% (39 of 40) of ECM respondents agreed or strongly agreed that they found it easy to allocate candidates to rooms using the dashboard. This was an increase from 2017 and 2016 when 93% (64 of 69) and 93% (46 of 54) of the respondents respectively had the same sentiment.

While the dashboard was generally well received by the survey respondents in the closed questions, the comments about the dashboard were mainly negative – that it was slow, not refreshing on its own, and the difficulties with filtering:

"The dashboard was not always functioning when I was supervising (before the 19 minutes failure towards the end of the exam) so I was not always confident I could track connectivity issues students may have been having. It was fortunate there were only a few (17) students in the room."

"The method for filtering notifications for a room seemed overly complicated - and then did not work."

"The biggest problem with the dashboard was that it wasn't refreshing automatically, which worried the supervisors - they wonder what would have happened if there had been lock-outs and breaches to deal with."

2.4 Candidates switching to paper

Note # 3

45 of these 53 respondents were either ECM or supervisor for the L1 English exam, in which there was a short term loss of connectivity at 11.39am. NZQA investigated the cause and made system changes for the next and subsequent digital examination sessions so this issue did not recur.

51% (53 (See Note # 3) of 104) of respondents had candidates who switched to paper having started the examination digitally. Of these respondents, 32% (17 of 53) agreed or strongly agreed that candidates switching to paper during the examination made supervision harder.

31% (17 of 54) of the respondents agreed or strongly agreed that candidates switching to paper was disturbing to other candidates.

2.5 Instructions and training provided

95% (96 of 101) of respondents attended an NZQA run training session.

96% (98 of 102) of respondents agreed or strongly agreed that the guide notes and materials they received for managing digital examinations were useful. This was similar to 2017 and 2016 when 97% (87 of 90) and 92% (78 of 85) of the respondents respectively had the same sentiment. In the open- ended question, survey respondents said that the instructions provided should be simplified:

"The manual needs [to] have all instructions to read to the students in one place."

"The digital supervision booklet is too wordy - it needs simplifying."

"Possible breach procedures complicated."

"Pre-exam instructions for the digital exam seem excessively long."

"Having all the printed material available kind of defeats the Digital aspect of the exam somewhat."

88% (84 of 96) of respondents agreed or strongly agreed that the training they received prepared them well for managing the Digital Pilot Examinations. This was similar to 2017 and 2016 when 85% (70 of 82) and 90% (65 of 72) had the same sentiment. In the open-ended question, survey respondents suggested there is room for improvement to training provided:

"ECM digital training needs to be more 'hands on'."

"The web page looked very different from the practise site so they didn't know where it was located and we hadn't been told this in our training."

"I feel that first time supervisors of digital exams should have the opportunity to attend the training sessions to better advantage than the ECM. When the ECM attends training, the information they are passing on is second hand."

"It would have been beneficial to actually have a trial practice with some of the difficulties which may occur during exam. Lack of experience with this made me less confident during supervision."

2.6 Other comments

Some of the respondents recognised the need for digital literacy training amongst ECMs and supervisors:

"The introduction of further digital exams is going to change the nature of the supervisor recruitment process - digital competence will be a must for the majority of supervisors employed."

"A certain level of computing skills are required for at least one of the supervisors. It is difficult to train for this and assumes expertise. Even simple keyboard errors should be solvable by the supervisor rather than the need to call a technician."

Some also expressed that they are generally positive about digital examinations:

"The more times we have digital exams the better they run."

"Looking forward to seeing this initiative being rolled out at full scale."

 

For more information, download the full report (PDF, 132KB).

 
Skip to main page content Accessibility page with list of access keys Home Page Site Map Contact Us newzealand.govt.nz