Student trials report

INTRODUCTION

Note # 1

There was a total of 8,244 logins across the nine available Trials, across 7,805 distinct students.

Nine Digital Trial examinations were offered in 2018 – Levels 1, 2, and 3 for the subjects English, Media Studies, and Classical Studies. All nine of the Digital Trial examinations required essay type answers. These Trial examinations were made available to schools in September and October 2018. The setup, invigilation and marking of the Digital Trials was managed and run by schools. The Digital Trials could either be used as a practice examination completed in one sitting or over multiple sessions as a classroom activity; or used for revision. 7,8051 (see Note # 1) students participated in at least one of the Digital Trial examinations, from 82 schools.

Documentation regarding managing and administering the Trials including guidance on logging in and the use of the marking tool was provided to schools and teachers. Further advice was offered by NZQA through email and telephone support. Familiarisation exercises were made available so key elements of the digital assessment could be experienced before undertaking the Trials.

1. THE SURVEY

1.1 Information on the survey

The student survey was designed to measure their self-reported satisfaction and experience of the Digital Trial examinations, including establishing whether the students regularly used electronic devices at home and at school.

NZQA received 1,711 responses across 1,637 students. Some students sat more than one digital examination and therefore may have answered the survey more than once. Students logged into 8,244 sessions ‒ a survey response rate of 21%. Of these there were 313 survey responses from Māori and Pasifika students.

The survey was made available to students within the SoNET system, directly after they submitted their Trial examination. The survey consisted of 12 questions, including three open ended questions – see Appendix 1 for a list of the survey questions. The survey was designed to take approximately five minutes to complete. Not all the student respondents answered all the questions in the survey. See Appendix 2 for summary tables of responses to the closed questions.

The table below shows the number of responses received, listed according to Trial examination.

Table 1. Number of responses received, by Trial examination

Digital Trial Exam SessionCountPercentage
L1 Classical Studies 6 0%
L1 English 793 46%
L1 Media Studies 123 7%
L2 Classical Studies 58 3%
L2 English 467 27%
L2 Media Studies 85 5%
L3 Classical Studies 19 1%
L3 English 111 6%
L3 Media Studies 49 3%
Total 1,711  

As the survey respondents were self-selected, care must be taken when applying the findings to all the participants of the Trials.

Text mining tools have been used to attempt to gather information about the most common types of responses to open-ended questions, however such tools are much more reliable with larger datasets. The outputs for this process for the student survey have been included in the appendix where they gave reasonably reliable answers, but have not been included where they were not useful and there were insufficient commonalities for the tool to identify trends.

1.2 Comparison with previous years

In previous years the Digital Trial examinations were offered at Level 1 only, but for a larger range of subjects. In 2016 the Trial was offered in ten subjects (Economics, Business Studies, History, Health, Geography, Science, Physics, Te Reo Rangatira, Spanish, and Samoan) and in 2017 it was offered in 14 subjects (Business Studies, Classical Studies, Economics, English, French, Geography, Health, History, Media Studies, Physics, Samoan, Science, Spanish, Te Reo Rangatira).

663 students responded to the survey in 2016 – a response rate of 16%. In 2017, the response rate improved to 18% with 1,035 responses received.

2. FINDINGS

2.1 Overall satisfaction

As stated in section 1.1, this survey had a response rate of 21%. Students who responded to the survey were overwhelmingly positive about completing the Trial examinations digitally, with 95% (1,536 of 1,621) agreeing or strongly agreeing that it was a positive experience. This was higher than in 2017 when 80% (755 of 939) had the same sentiment. In particular, the percentage of respondents who strongly agreed that completing the exam digitally was a positive experience (47%, 762 of 1,621) was higher than in 2017 (24%, 226 of 939).

Of the students who responded to the survey, 87% (1,413 of 1,622) said they preferred completing the examination digitally rather than on paper. This was higher than 2017 when 61% (576 of 941) had the same sentiment. In particular, the percentage of respondents who strongly agreed that they preferred completing the examination digitally rather than on paper (52%, 848 of 1,622) was higher than in 2017 (27%, 251 of 941).

Responses were analysed separately by Digital Trial subject and level to determine whether there were any differences in satisfaction levels depending on the type of subject being examined digitally. There were higher levels of satisfaction for survey respondents who undertook Level 3 English and Level 2 Media Studies. These students may have had more experience with digital examinations than Level 1 student respondents.

Table 2.           Responses to the question: "I preferred completing the exam digitally rather than on paper", by exam session.

 

EnglishMedia StudiesClassical Studies

 

Level 1Level 2Level 3Level 1Level 2Level 3Level 1Level 2Level 3Total
Strongly Agree 359
(48%)
250
(57%)
69
(68%)
62
(52%)
52
(63%)
23
(48%)
3
(60%)
23
(43%)
7
(37%)
848
(52%)
Agree 287
(38%)
137
(31%)
29
(28%)
40
(33%)
21
(26%)
20
(42%)
1
(20%)
21
(39%)
9
(47%)
565
(35%)
Disagree/Strongly Disagree 105
(14%)
54
(12%)
4
(4%)
18
(15%)
9
(11%)
5
(10%)
1
(20%)
10
(19%)
3
(16%)
209
(13%)
Total 751 441 102 120 82 48 5 54 19 1,622

Table 3. Responses to the question: Overall, I found completing this exam digitally was a positive experience, by exam session.

EnglishMedia StudiesClassical Studies
Level 1Level 2Level 3Level 1Level 2Level 3Level 1Level 2Level 3Total
Strongly Agree 333
(44%)
219
(50%)
55
(54%)
50
(41%)
51
(62%)
19
(39%)
2
(40%)
27
(50%)
8
(42%)
764
(47%)
Agree 368
(49%)
201
(46%)
45
(45%)
62
(51%)
29
(35%)
27
(55%)
3
(60%)
27
(50%)
10
(53%)
772
(48%)
Disagree/Strongly Disagree 48
(6%)
21
(5%)
1
(1%)
9
(7%)
2
(2%)
3
(6%)
0
(0%)
0
(0%)
1
(5%)
85
(5%)
Total 749 441 101 121 82 49 5 54 19 1,621

Key themes from responses to the question ‘What did you like most about completing the exam digitally?' included that students can type faster than writing, writing hand doesn't hurt, it is easy to edit answers, and that the spell check is helpful.

"… I found typing [quicker] and less straining than hand writing which helps to get my thoughts down efficiently."

"… I liked the [word] count tool and the spell check tool."

"I liked how everything was neat, if I made a mistake I could just backspace instead of scribbling."

"…I was [able] to write more without my hands getting tired or sore."

Key themes from responses to the question ‘What did you dislike the most about completing the exam digitally?' included potential hardware and other computer issues (Wi-Fi, device, multi-plug, keyboard etc.), the noise of other students' keyboards being distracting, the red line that appears when the word count is reached, and the absence of a spell check function other than for English. A common additional response to this question was the student noting that they did not dislike anything about the Trial examination.

"I was scared my laptop or the internet was gonna die the whole time."

"The sounds of everyone's keyboard in the room got quite [irritating] after a while…"

"My eyes are a bit sore from staring at a screen for 3 hours."

"…Having to scroll up and down to read through the texts was also time-consuming and tiring…"

Some people were not aware of the existence of the spell check function.

"No spell check, but I guess that's a given. It is just when you're typing fast you're bound to make several errors without noticing."

"Didn't realise there was spell check till now."

2.2 The digital examination experience

Preparation

NZQA made familiarisation activities available to students who were participating in the Digital Trial examinations. The purpose of the familiarisation activities was to provide students with the opportunity to experience the look and feel of a digital examination, including the login and submission process that students would experience, and the different tools that are part of the digital examinations (copy and paste, drag and drop, spell check, examination lockout, and examination preview).

Most of the survey respondents (75%, 1,251 of 1,657) agreed or strongly agreed that the familiarisation activities were useful in their preparation for their digital examination. This is similar to the percentage for 2017, when 74% of respondents (712 of 972) had the same sentiment.

16% (257 of 1,657) responded that they did not know that the familiarisation activities existed. This is similar to the percentage for 2017, when 19% of respondents (184 of 972) had the same sentiment.

On the day

53% of respondents (494 of 934) reported using their own device to complete their Digital Trial examination, which was a higher percentage than the 2017 and 2016 Trials surveys (35%, 210 of 602 and 25%, 145 of 583 respectively).

80% of respondents (1,251 of 1,565) reported they used a laptop to complete their Digital Trial Examination, and 19% (301 of 1,565) reported they used a desktop computer. 1% of respondents (13 of 1,565) reported they used a tablet. Usage of laptops was greater compared with the 2017 and 2016 Trials surveys (57%, 509 of 894 and 45%, 201 of 443 respectively).

6% of respondents (96 of 1,670) reported experiencing network problems and 5% (78 of 1,670) reported experiencing device problems when accessing or completing the Digital Trial examination. This is lower than in 2017 and 2016 when 25% (247 of 981) and 30% (183 of 604) of respondents respectively reported experiencing network or device problems.

97% of the survey respondents (1,582 of 1,630) found it very easy or easy to navigate through the Digital Trial examination, which was an increase from the 2017 Trials survey (90%, 852 of 948). In particular, the percentage who found it very easy to navigate through the examination (48%, 776 of 1,632) was higher than in 2017 (36%, 343 of 948).

97% of the survey respondents (1,580 of 1,630) found it very easy or easy to enter their responses to the Digital Trial examination, which was an increase from the 2017 Trials survey (87%, 820 of 944). In particular, the percentage who found it very easy to enter their responses to the examination (49%, 798 of 1,630) was higher than in 2017 (35%, 327 of 944).

87% of the survey respondents (1,415 of 1,624) agreed or strongly agreed that completing the digital examination took less time than they would have expected had it been paper-based, which was an increase from the 2017 Trials survey (74%, 701 of 944). In particular, the percentage who strongly agreed that completing the digital examination took less time than they would have expected had it been paper-based (48%, 786 of 1,624) was higher than in 2017 (34%, 318 of 944).

2.3 Digital technology at home and at school

Most of the survey respondents reported having access to laptops and smart phones at home (93%, 1,581 of 1,698 and 91%, 1,547 of 1,698 respectively). About half of the survey respondents reported having access to desktop computers and tablets at home (48%, 807 of 1,698 and 45%, 761 of 1,698 respectively).

25% of the survey respondents (430 of 1,698) reported having access to all the four devices mentioned at home, and 35% (601 of 1,698) reported having access to three of the four devices at home. Another 30% (506 of 1,698) reported having access to two of the four devices at home.

To support their learning, students survey respondents reported that digital technology was very often or quite often used in class (86%, 1,456 of 1,699) and for homework (86%, 1,448 of 1,685). Digital technology was relatively less often used for internal assessments, where 71% (1,207 of 1,687) reported to using it very often or quite often for internal assessments. Very few respondents reported never using digital technology to support their learning in class or for homework. 4% (67 of 1,687) reported never using digital technology for internal assessments.

2018 findings show that proportionately more respondents are using digital technology to support their learning in class, for homework and for internal assessment than Trial survey respondents indicated in 2017 or 2016.

2.4 Suggestions for improvement/feedback

At the end of the survey, respondents were asked whether there were any features or functions that they thought future digital exams should include, or whether they had any other feedback.

Respondents provided a range of answers to this question including the need for a spell check function, the ability to highlight the unfamiliar text (for the English exam), a higher word count or clearer rules on word count, and improvements to the scrolling functionality. Most respondents did not have any feedback. Some also suggested having access to paper to brainstorm ideas, and other features they are already familiar with from other technology used (e.g. automatic capitalisation of the first letter of the sentence from Word and Google Docs).

"Physical [paper] to brainstorm on."

"It should scroll while you type."

"Perhaps a highlighting function when doing unfamiliar text so we could annotate the text while we were reading it."

"Automatic spell-check."

"If it is possible, I think it would be good to have automatic capitalization of the first letter of a new sentence, rather than having to do it manually each time."

"The layout should be less stiff as it looks quite intimidating.... The layout colours should be brighter."

The survey responses indicate that students' experiences of the Trial were positive. The relatively more experienced survey respondents have a higher level of satisfaction which may be due to their familiarity with the interface. Electronic devices are widely used by students, and some responses to open ended questions indicate that some digital functionality they are very familiar with elsewhere (such as Word's automatic capitalisation and spell check) is not understood by students to be available, or is actually unavailable, or requires improvement. The survey responses also indicated that some students were not sufficiently familiar with the spell check or word count features of the examination for which the same instructions were provided as in the paper examinations of the respective subjects and which were included in the familiarisation exercises that were made available prior to the examination.

 

For more information, download the full report (PDF, 177KB).

 
Skip to main page content Accessibility page with list of access keys Home Page Site Map Contact Us newzealand.govt.nz