Research, Innovation and Enhancements

NCEA Online delivers digital exams to complement the long-standing paper-based approach. We are now looking to identify and take up the new assessment opportunities offered by digitally enabled teaching and learning.

This work will help us consider how we might deliver future digital exams. 21st century assessment can contribute to the equity of NCEA outcomes by engaging students in new ways, stimulating changes in teaching and learning and providing new ways for students to demonstrate their knowledge.

NZQA’s vision (depicted below) for equitable, innovative, credible and robust digital assessment, involves us working with education leaders, teachers, students and education agencies to co-design the approach.

The following innovation and research work may result in further research, trials, pilots or enhancements to NCEA Online and the way we manage it over the next few years.  We will update this section as we progress this work.

 

Qualify Future World navigation infographic 700px300ppi

Innovation Trials in Assessment Practice

Te Reo Rangatira

What is involved: During 2020, selected schools and kura which offer Te Reo Rangatira trialled a modified digital assessment which let students listen to instructions, questions and resources, as well as reading them.

Why we are doing this work: Māori students have told us they would like to hear as well as read the content of an exam. We are using Te Reo Rangatira to first test this idea in the context of mātauranga Māori. This may enable Māori students to have a richer and equitable digital exam experience.

Progress update: Following completion of the innovation trial, we are planning the work needed to attach te reo audio to the Te Reo Rangatira 2021 digital practice and NCEA end of year exams for levels 1,2 and 3. More information is available in the summary report (PDF, 359KB).

Te ao Maori Assessment integrityAdaptability

Digital Statistics and Maths (Calculus) Assessment

What is involved: For subjects such as Statistics and Calculus which are hard to assess digitally we are trialling third-party software used by some schools. The trial involves the current assessment platform and assessment methods.

Why we are doing this: We want to provide digital assessment across all subjects and provide a good user experience for subjects difficult to do digitally. If students can use familiar software (that they have been using for their learning), it could increase their engagement and attract a broader range of candidates.

Progress update: We have completed and analysed the two maths innovation trials carried out in 2020. More information is available in the Digital Mathematics report (PDF, 851KB) and Digital Statistics report (PDF, 912KB).

Digital firstAccessibility and usabilityAssessment integrity
 

Squirm Germ Gamified Assessment

What is involved: Biology Achievement Standard 90927 has been developed into an assessment based on a game approach.  In 2020 selected schools will participate in a trial with students attempting the assessment and recording their responses.

Why we are doing this work: We want to find out whether assessments based on a game design have more applicability in the future, for example to enable candidates to better show what they know, potentially through being engaging and authentic. 

Progress update:  The innovation trial of the biology assessment based on a game design met the requirements of the standard, and there was a high level of interest and engagement from students and teachers.  More information is available in the summary report (PDF, 287KB).

Digital firstAssessment integrityAccessibility and usability

Research into Assessment Practice

Te Reo Māori (TRM) text-to-speech and spellcheck

What is involved: In 2020 we worked with Waikato University to investigate offering spellcheck and text-to-speech in te reo Māori. Their researchers worked with others to review high quality digital kupu (word) lists to support these functions.

Why we are doing this work: Providing spellcheck and text-to-speech functionality in te reo Māori can improve equity of access and outcomes for te reo Māori speakers, better reflecting how they learn.

Progress update: Waikato University’s research for NZQA is available here in the text-to-speech report (PDF, 2.1MB) and the spellcheck report (PDF, 494KB).

Te ao MaoriAssessment integrityAccessibility and usability

Text-to-speech

What is involved: The NCEA Online platform has an inbuilt text-to-speech function. In 2020 we completed a text-to-speech trial with a small group of schools and kura.

Why we are doing this work: Students who sit NCEA Online do not have the option to listen to assessment text being read aloud in digital assessments. Using this technology will improve accessibility for all students (including those who require reading support or prefer aural learning) and give them greater control over how they complete their assessments.

Progress update: In 2021 we will carry out a more in-depth two stage trial – firstly to gain student insights about this functionality and to carry out user testing.  Depending on the outcome of this work, we will look at trialling some of the text-to-speech tools using digital assessment activities.

Assessment integrityAccessibility and usability

Special Assessment Conditions software stock take

What is involved: This research will identify technology being used in New Zealand to support students with Special Assessment Conditions (SAC) so we can plan for innovation trial(s) to test the compatibility of widely used applications with the exam platform.

Why we are doing this work: Allowing students in digital exams to use the assistive technology they already use to support their learning will increase equitable access and outcomes.

Progress update: We have completed a stock take of assistive technology tools (which includes application, purpose, suppliers, platform), who is using them, and what further research and analysis is needed for NZQA to consider approving use of assistive technology in digital assessments.  More information is available in the summary report (PDF, 462KB).

Accessibility and usabilityAdaptability

More than one assessment opportunity a year

What is involved: We are researching how we can use the exam platform so NZQA and schools could offer digital exams at various times during the year.

Why we are doing this work: There is strong demand from schools to offer digital assessments, and there is potential for NZQA to stage exams at different times of the year so students can be assessed closer to the time they undertake their learning.

Progress update: In 2020 NZQA worked with the University of Waikato to complete a general literature review and synthesis of key findings from research studies. More information is available in the summary (PDF, 346KB)and full report (PDF, 1MB).

 Assessment integrityDigital firstAdaptability

Remote exam supervision

What is involved: This research aimed to identify remote supervision tools suitable for NCEA digital exams and assessments to see what opportunities these could offer for students and NZQA.

During 2020 we developed a list of the available tools, their capabilities and how they are used in assessment settings.

Why we are doing this work: This could lead to greater flexibility and resilience of exam and assessment delivery, increasing the types of places and geographical locations where students can be digitally assessed, and supporting human supervision with new tools.

Progress update:  Associated with this work, we are now preparing a remote exam supervision trial, starting with a technical proof of concept for potential solutions.

Accessibility and usabilityAdaptabilityAssessment integrity   

Automated marking

What is involved: We are exploring how well-suited automated marking of essay questions is to the NCEA context; what is involved in set up and how marking software works in practice.  During 2020 the University of Alberta – Centre for Research, Applied Measurement and Evaluation (CRAME) used some past NCEA digital essay responses to complete automated marking, with the results reviewed and analysed by NCEA markers.

Why we are doing this work: This trial will help us understand if automated marking could enable student responses to be marked in real time and for different questions to be presented to students based on their previous response. This form of adaptive testing allows the software to choose which question to present next.

Progress update: More information about the 2020 research is available in the summary (PDF, 330KB)and full report (PDF, 1.5MB).

Planning is underway for a further trial of automated marking in 2021 using a different vendor/s.  We will evaluate registrations of interest received from organisations wanting to take part in this further round of research.

Assessment integrityDigital firstAdaptability

Scoping new services

Alternative delivery methods

What is involved: In 2020 as part of an innovation trial we had planned to work with schools in New Zealand, Niue and the Cook Islands which lack access to reliable internet connections, a barrier for participating in digital exams. This work was paused in 2020 due to COVID-19.

Why we are doing this work: This could enable schools without access to reliable internet connections to participate in digital exams and may also add resilience into the system by supporting the delivery of digital exams in the event of major incidents or natural disasters.

Progress update: In 2021 we are changing the focus of this work from an innovation trial to doing research to help us understand schools’ needs and how we can best take those into account in the way we provide digital exams, as well as investigating potential delivery solutions.

 Digital firstAccessibility and usability

Externally sourced assessment – digital practice exams delivered on NZQA’s online exam platform

What is involved: We are exploring how NZQA’s online exam platform could be used in schools so teachers can develop and deliver practice exams and activities.

Why we are doing this work: If the online exam platform could be used in schools, it would help teachers and students become more familiar with digital assessment and the platform.

Progress update: In 2020 NZQA worked with three subject associations so that schools could offer digital practice exams for English, Agriculture and Horticulture Science and Classical Studies at all levels using NZQA’s online exam platform. Digital practice exams in Te Reo Māori were also offered at all levels. This was the first opportunity to make digital practice exams available to schools in a limited way. We evaluated feedback from the formal surveys sent to schools and students about their 2020 experience of participating in digital practice exams.

More information is available in the summary (PDF, 344KB), full report (PDF, 1.4MB) and summary dashboard (PDF, 242KB).

Planning is underway to provide externally sourced digital practice exams from 2 August to 1 October 2021.

Data as an assetDigital firstAccessibility and usability 

Submitted subjects

What is involved: This is looking at using NZQA’s online platform to mark externally assessed work not done by examination. We will research what solution is needed for submitting work for NCEA external standards where assessment is not administered as an exam. 

Why we are doing this work: Students are starting to generate portfolios digitally.  We want to provide a way for teachers to upload school-based content and submit it for assessment by NZQA markers, removing the need for printed materials. If the trial meets requirements, a controlled pilot could be held in 2021.

Progress update: We’re gathering information about subject specific requirements and planning service design workshops with schools and kura. Co-design Submissions workshops were held in early 2021. All information will be used to select and design the best solution.

Digital firstAdaptabilityAccessibility and usability 

Scholarship online

What is involved: During 2020 we carried out preliminary research to establish the case for a limited trial of a digital Scholarship exam.

Why we are doing this work: Increasing numbers of students have completed their NCEA external exams and course work digitally and may expect to complete their Scholarship exam in the same way.  We want to extend the digital external assessment opportunities open to senior secondary students to include Scholarship, as well as NCEA exams.

Progress update: We are trialling Scholarship Media Studies as a digital end-of-year exam in 2021. More information is available in the 2020 research summary report (PDF, 145KB).

Digital firstAdaptability Assessment integrity

Digitally-delivered Common Assessment Tasks

What is involved:

We have been investigating the feasibility of offering one or more of the Digital Technology Common Assessment Tasks (DCAT) on the NCEA Online platform in future years.

Why we are doing this work: Increasing numbers of students have completed their NCEA external exams digitally and may expect to complete other kinds of assessment in the same way.

Progress update: Our investigation is continuing, and we expect Digital Common Assessment Tasks (DCAT) will be offered on the NCEA Online platform in 2022.

Digital firstAdaptabilityAccessibility and usability  

Te Waharoa Ākonga | Student Services

What is involved: This will tailor the look and feel of the student landing page to improve the experience of students when they log into the NZQA website with their username and password.

Why we are doing this work: This will give students direct access to personalised information about the digital exams they are entered in, reduce the time to log in and access the correct exam and reduce the need for any assistance from supervisors and ECMs at the beginning of a digital exam session.

Progress update: We will continue investigation in 2021 with development of user requirements as part of NZQA’s website redesign project. Improvements to Te Waharoa Ākonga are likely to be implemented in early 2022.

Accessibility and usabilityDigital first 

Designing new functionality

Digital tools and applications

What is involved: There are subjects that are ‘hard to digitise’ such as maths, chemistry, dance and drama, which need to be managed and delivered in a different way. We are identifying suppliers who can provide digital tools and applications to ensure an equivalent assessment experience to NCEA external paper exams.  The digital tools and applications need to support a range of tasks such as: writing complex formulae and equations; creating and modifying tables; drawing maps; and identifying special requirements for music.

Why we are doing this work: We want to be able to offer a full suite of digital exams (at least at the substitution level) where possible in 2022. It’s important that using any digital tool is easy and instinctive for students so that they can focus their efforts on demonstrating their knowledge of the subject, not of the tool. Any future trials will include user experience testing to ensure the tools meet the needs of students.

Progress update: We will evaluate the registrations of interest received from vendors who might be able to deliver digital tools or applications for one or more of the identified capabilities to determine whether to progress to a Proof of Concept exercise.

Accessibility and usability Adaptability

Scanned paper responses

What is involved: During 2020 we completed a trial that scanned a selection of completed past exam paper booklets and processed them through the digital marking platform.  This allowed marking to be completed digitally.

Why we are doing this work: We are aiming to bring together paper and digital marking and reduce paper interactions within the exam cycle. If this trial leads to scanning all paper responses, they will be marked and returned to students using the digital systems.  It would eliminate the current two-week lag to return paper responses to students after the release of results and reduce the need to have separate quality assurance mechanisms for paper and digital responses.

Progress update: The 2020 trial gave NZQA confidence that scanned responses can be marked in the digital marking application RM Assessor3 and that the process is fit for purpose. Recommendations from the trial include preparing a brief for the next phase of this work and further scoping the requirements. This will be done along with benchmarking and check marking during 2021. More information is available in the summary report (PDF, 137KB).

Accessibility and usabilityDigital first AdaptabilityData as an asset

Enhancing benchmarking and check-marking

What is involved: NCEA digital exams are marked online using the RM Assessore-Marking tool. We used 2019 digital responses to test the benchmarking and check marking feature in the marking application of the digital exam platform. This enables all marking processes (for both digital and scanned paper responses) to be performed in the one digital environment. The trial may result in piloting the wider use of the digital marking tool.

Why we are doing this work: Any resulting improvements to marking information accuracy will help sustain confidence in the assessment system. Benefits would come from more streamlined delivery of candidate responses to markers and new ways of doing quality assurance.

Progress update: In 2020 panel leaders and check markers completed their trial benchmarking activities using RM Assessor3, finding that overall the functionality trialled is useful to carry out benchmarking and quality assurance processes digitally.  Recommendations from the trial include progressing digital benchmarking and check marking along with scanning during 2021. More information is available in the summary report (PDF, 110KB).

Assessment integrityData as an assetDigital first

Marker responses to students’ digital and scanned exam responses

What is involved: We completed a literature review in 2020 to investigate marker preference for, and perception of, different modes (i.e. marking scanned responses onscreen, marking paper-based responses, and marking digital responses). We have now done further research involving markers using NZQA’s marking system to mark scanned handwritten and digital responses onscreen.

Why we are doing this work: In 2021, we will scan NCEA Level 1-3 end-of-year paper exam responses so they can be marked on screen.  To support the introduction of this process, a research project was proposed to understand how markers respond to digital (typed) and scanned (handwritten) exam responses.  

Progress update: During 2021 we invited a panel of markers with experience marking both paper and digital exam responses to participate in the research.  We collected data from group and individual interviews before and after onscreen marking commenced, and from the outcome of marking.

By the end of the research, the participants’ perceptions about onscreen marking had shifted towards being strongly positive.  Participants identified key benefits as removing administration associated with paper-based exam responses, improved efficiency, greater responsiveness when checkmarking, and the ability for markers to collaborate in real time.

More information is available in the 2021 research report (PDF, 747KB).

 

Digital firstAdaptabilityAssessment integrity

 

Assessment Master planner

What is involved: This trial will assess some of the features in the digital assessment platform to support the planning and provision of digital exam centres. In addition, this trial will assess the reporting capability for digital exam management, attendance, notification and messaging features.

Why we are doing this work: Testing this new system will allow us to decide if it is ready for use in the digital NCEA exams. This will improve efficiency and the exam experience for Exam Centre Managers (ECMs) and students.

Progress update: The trial has shown that none of the tested basic functionality entirely met the needs of NZQA or Exam Centre staff.  Next steps involve better understanding the needs of the groups who benefit from the platform features trialled; and an analysis to match the AM Planner functions and needs of potential user groups. More information is available in the summary report (PDF, 111KB).

Accessibility and usabilityDigital firstData as an asset 

 

 
Skip to main page content Accessibility page with list of access keys Home Page Site Map Contact Us newzealand.govt.nz