Digital Assessment Vision: a design principles approach

The New Zealand Qualifications Authority (NZQA) has initiated work to transform the way external assessment related to the National Certificate of Educational Achievement (NCEA) is offered, including making changes to the supporting operating model by moving towards digital assessment.

While an important milestone signalling the direction and intent will be to have all NCEA examinations available online by 2020, our vision is for transformed assessment – a 21st century approach to assessment online for anyone, anytime.

NZQA’s 21st century approach to assessment will build on the pedagogical changes taking place in classrooms now and known, when implemented effectively, to improve student engagement in learning, student achievement and their readiness for higher education or employment.

A key value that underpins this move is that it supports all students to be awarded a relevant and trusted credential that will signal their readiness to succeed in higher education or the workplace. Our 21st century assessment must contribute to addressing equity of NCEA outcomes, by engaging students in new ways, stimulating changes in teaching and learning, and providing new ways in which application of knowledge can be demonstrated in an assessment situation.   

Qualify for the Future World: Kia Noho Takatū Ki Tō Āmua Ao

What is shaping our future senior secondary assessment services?

Digitally Enabled Pedagogy – In many schools, students and teachers are already working with technology that is supporting flipped classrooms[1], blended learning and project or inquiry based learning, networked learning opportunities with other providers, among other approaches. The assessment experience needs to be relevant to the way in which students have learned. For students who are increasingly learning online, it makes sense for their learning to be assessed online too.

Students also expect to work with teachers to develop personalised learning pathways (content, pace, direction) that are relevant and accessible, and which offer them the capability to achieve to their potential and make informed choices about their future. Digital assessment supports teachers and students to achieve this. A once a year, paper-based examination delivery infrastructure cannot support personalised pathways. Digital assessment has potentially greater flexibility to support future changes in the mix of formal senior secondary assessment, including the current internal assessment, external by examination, external by submitted subject and assessment of literacy and numeracy.

Digital assessment also enables richer data from the assessment experience to be re-applied back into teaching and assessment development practices, as well as assessment design.  

Future Education and Employment – Employers and tertiary institutions need students to be work-ready or prepared for higher study, and this requires that students leave secondary schools with qualifications and capabilities necessary for success in the global, connected, digitally-powered world in which we live.

Our 21st century approach to assessment is not limited to application of knowledge, but will increasingly require students to draw on a wider range of skills and capabilities, even if those capabilities are not directly assessed. Digital technologies offer new ways of presenting assessment stimuli or resources (e.g. simulations) and new ways for students to demonstrate their capabilities (e.g. storyboards).       

Quality Assurance – Those relying on the credentials that NZQA issues – students, parents and whānau, employers, domestic and overseas tertiary institutions – need to be confident these credentials continue to be of the highest integrity and fit for purpose.

From a logistical and educational perspective, digital external assessment offers a specific type of support for assurance of student identity, authenticity and authorship of response – all of which are at the heart of the credibility of the NCEA qualification.     

Why are we on this assessment transformation journey?

The current approach of once-a-year, paper-based external examinations does not afford the flexibility to respond to, and take advantage of the 21st century teaching and learning approaches that are occurring. In fact, the current assessment approach may act as a brake on further innovation in teaching and learning, and constrains students to be ready at a point in time, irrespective of their learning journey. The move to digital assessment also enables the opportunity to improve the equity of NCEA outcomes for Māori and Pasifika students. This is through, for example, access to richer data available about their experience of the summative assessment, the capability to personalise the timing of assessments and to offer diverse methods for capturing student responses.    

Our vision enables us to make good decisions about NZQA’s digital assessment services (people, processes, information and technologies) that protect our current values of robustness and credibility. It allows a responsive and cost-effective approach for moving to digital assessment in the first instance, and the creation of a platform for innovation in assessment that is enabled by digital services.

The vision is framed by six equally weighted design principles. These have enduring qualities that can be reflected at different levels of maturity along the journey from current towards a future state. Some factors that will come into play are:   

  • the extent of and timing for realising the vision
  • the sequencing of the innovations
  • interdependencies between the principles and how they present in practice

The vision is less about a specific end point and more about a direction of travel and the ability for NZQA, education leaders, teachers, students and education agencies to work together to co-design the 21st Century assessment approach we want.

Decisions about the delivery strategy will be referenced back to these design principles:

  • Assessment integrity: The integrity of external assessment of NCEA is paramount; digital assessment enables credible, reliable and valid results and allows all students to authentically and securely display their application of knowledge, skills and abilities.
  • Te Ao Māori: Digital external assessments will enable students to use Te Reo Māori and Mātauranga Māori in their assessments, maintaining validity and contributing to equitable NCEA outcomes.
  • Accessibility and Usability: The user experience for participants in digital external assessment services – students, examiners, markers, supervisors, teachers – is accessible and intuitive; for a given service or process, the effort to participate will be no greater than what is required under the current arrangements.
  • Adaptability: Digital external assessment services evolve with student and school readiness, and enable delivery modes and assessment types to evolve and develop in response to changes in standards and the prevalent teaching and learning, resulting in assessments that are always meaningful, authentic and relevant to all 21st century learners and to the community.
  • Digital first: External assessment services and their underlying functions and processes are designed for end-to-end digital-first delivery.
  • Data as an asset: Digital external assessment services enable the collection of a broad range of detailed analytics which can be used to better inform assessment development as well as teaching and learning.

Digital Assessment Vision for NCEA Online

NCEA Online is the flagship project that is delivering the digital assessment transformation. What will be happening when these six principles are in play as we move towards this future state?

Assessment integrity: The integrity of external assessment is paramount; digital assessment enables credible, reliable and valid results and allows all students to authentically and securely display their application of knowledge, skills and abilities

Psychometric analysis of results across a range of modes, subjects, assessment sessions and environmental factors will assist in the determination of transition steps, and the potential effects of moving up the SAMR model. This is critical in terms of continuing to provide confirmation of the integrity of the assessment system. The methodology will be adapted to account for characteristics of the transition, such as a non-representative distribution of students across digital and paper, and the potential effects of moving from substitution and augmentation towards modified and redefined assessments.

When external assessments become available on demand, they will require students to engage using a wider range of non-content capabilities (e.g. critical thinking, collaboration, persistence) than is possible today. That will require marking approaches and psychometric analysis of results to evolve to be able to differentiate between the application of knowledge that has been demonstrated from the other skills and abilities shown in the individual student’s response. This is so that we can continue to be sure that we are assessing what we intend to assess against the achievement standard, including assessment of collected evidence in portfolios. 

A high integrity assessment system aims to reduce inter-year variability. Intra-year variability will also become a factor to be managed. The methods used by examiners and markers, supported by artificial intelligence tools, to create and mark different assessments (e.g. Substitution or Augmentation assessments offered in the same timeframe that each assess against the standard) and ensure student responses are falling within the expected range, will be adapted to deal with the smaller numbers of students undertaking a given assessment (in a fully “anytime” model, assessments will be virtually personalised).

There will be opportunities to continuously improve assessments based on feedback loops that are shorter than the current 12 – 18 months. Fulfilling the need for assessment integrity in a digital world is likely to have wider consequences for the system. Faster feedback loops may drive the need for standards to be reviewed or exemplars to be refreshed more frequently. In turn, to be responsive to the environment, teaching approaches will need to be more agile to adapt year on year, without increasing workload.   

Authenticity and security of the digital assessments, students’ responses and markers’ judgements will be managed over reliable, well-performing networks with back-up options that ensure all students wishing to undertake an external digital assessment can do so and be confident that their digital identity and permissions, their work and their NCEA credentials are not compromised in any way.

Te Ao Māori: Digital external assessments will enable students to use Te Reo Māori and Mātauranga Māori in their assessments, maintaining validity and contributing to equitable NCEA outcomes 

Educators are prioritising the move towards personalised pathways for all students, supported by Kāhui Ako (Communities of Learning) and other initiatives. As personalised pathways become embedded in teaching and learning practices, NZQA’s NCEA external assessments, especially examinations, will need to adapt. This represents a profound opportunity for assessment systems to support Māori students to be successful and reach their potential.

Personalised learning pathways that include digital assessment have the potential to engage 21st century Māori students in new ways. They can now be assessed in a more relevant and valid manner that enables them to fully reflect cultural identity in their responses. This will require changes to the ways in which some questions are expressed to elicit the student’s response.  

Māori students and their teachers will be engaged early in the co-design of external digital assessment. Māori students will be participating and succeeding in external digital assessments at the same rate as their non-Māori counterparts.  We will deeply understand the assessment experience of Māori students.  

The value proposition for digital assessment that follows a personalised pathway for Māori students will vary, including:

  • Continuing to have Te Reo translations of assessments
  • The ability to respond in contexts that are relevant to their lived culture and experience, are reflective of Te Ao Māori, including incorporation of Mātauranga Māori, and when and where assessments are conducted
  • Assessment responses requiring applications or demonstrations of knowledge, rather than descriptions of knowledge – building on the practice already present in some externally assessed standards
  • Collaborative engagement with assessments, where the individual contributions can be identified and form part of the result  
  • Reflection of manaakitanga towards students through the digital assessment look and feel, and user experience.

Digital assessment enables this type of presentation and assessment flexibility, ensuring that the student is still being asked to apply or demonstrate their knowledge against the standard, using their other skills and abilities to do so.

Multi-lingual (including Te Reo Māori) renditions of the assessment content, the student response and the user experience (e.g. screens and templates) are readily supported by digital assessment. 

Te Ao Māori is an important principle underpinning the vision to equitably serve Māori students. There is a co-benefit for the wider assessment system. Digital assessment will successfully respond to the rigours of delivery in Te Reo Māori and the reflection of Mātauranga Māori to ensure an equitable assessment experience for Māori students. This can only occur with the systematic adoption of a 21st century approach to assessment development, delivery practices and data analytics.

Accessibility and Usability: The user experience for participants in digital external assessment services – students, examiners, markers, supervisors, teachers – is accessible and intuitive; for a given service or process the effort to participate will be no greater than what is required under the current arrangements 

All participants in the assessment system will experience an overall reduction in the effort to participate, once familiarity is achieved and processes embedded. Accessibility and usability will be locked into the early design thinking, and informed by end users co-creating the evolving digital assessment service. User feedback will be supplemented with user experience data gathered in the execution of the digital assessment services (e.g. time spent on a particular screen, number of attempts at completing a process).   

Examiners currently create examinations supported by online collaboration tools. Security of the electronic and paper forms of the examinations in the various stages of their lifecycle is closely managed. This will take new shape when examination content is managed online through all stages of its lifecycle. 

We will balance the diversity of technology used by all digital assessment participants with minimum standards that ensure the user experience is predictable, reliable and is supported by the assessment process. A degree of personalisation of the digital assessment processes will be available to all users to enhance their experience where this does not compromise the overall integrity of the service.  

Assistive technologies (e.g. text to speech) will be available to all students where this reflects their personalised learning environment. “Bring your own app” (e.g. for assistive technologies) will be supported.

Teachers using digital dashboards for tracking student progress will have a more accessible view of the students’ full capabilities and assessment performance, with the ability to extract data from the assessment results repository. This will save them time and enable them to make better informed decisions when planning the personalised learning pathways with students.

Markers will not have to worry about the physical security of written scripts, will be able to work more efficiently online with immediate access to advanced marking support tools and their work will be quality assured more efficiently.  As noted in Digital First (below), there will be a role for automated marking in the future. Because of the reduced effort to participate, more teachers from all over New Zealand may be interested in the professional development that markers have typically valued. 

Supervisors will have powerful dashboard tools available to supplement their visual supervision of examinations. Students’ identity credentials, their access to and authorisation to sit an external assessment will be managed by their digital credential, which means that supervisors will not be reliant on checking identity cards and paper examination slips. Over time, the supervision logistics will adapt where students can be supervised in non-examination centre settings or monitored remotely.      

Adaptability: Digital external assessment services evolve with student and school readiness, and enable delivery modes and assessment types to evolve and develop in response to changes in standards and the prevalent teaching and learning, resulting in assessments that are always meaningful, authentic and relevant to all 21st century learners and to the community

Just as NCEA has continued to evolve and adapt since its inception, NCEA Online will support an adaptive approach that is driven by the experience of the primary users – students and teachers. As long as external assessment (currently examinations and portfolios of externally set and marked assessments) is valued by stakeholders, NZQA will adapt its approaches to delivering assessment services. Digital assessment creates opportunities for that adaptability to occur at the pace that suits the sector, and at cohort, subject or standard level. Adaptability will be seen in many forms and will be informed by user insights and data.

An adaptable digital assessment approach will enable different techniques to be used to elicit students’ responses. These could include creation of or response to simulations, or interactions with another student or an artificial intelligence engine to solve a problem. Assessments could be designed to ensure students can use their “21st century” skills as well as their application of knowledge to respond. This could include oral responses or collaborations.     

The timetabling of assessments will be driven by student readiness, where the precursor personalised learning pathways have been implemented in schools. When we are confident that the assessment integrity can be managed with more frequent or “anytime” assessment opportunities, teaching practices may be adapted faster as students’ results become available closer to the time of the assessment. 

In time, while security, authenticity and authorship of the student response will remain important for assessment integrity, we would have the means to introduce “open book” assessments and assessments that can be supervised remotely with digital technologies.

Many of us who use digital tools expect to be able to personalise these to different degrees. Digital assessment will be adaptable enough to students to personalise their experience where that does not compromise the assessment integrity.  

Schools will be able to use the NCEA Online digital assessment service to create assessments for internal use, or to deliver purchased digital practice assessments that comply with the information standards of the assessment service.  This level of adaptability means that schools can do as much or as little online assessment as suits their pedagogical approach, and students’ opportunities to be familiar with the tools follows from that decision.         

Digital first: External assessment services and their underlying functions and processes are designed for end-to-end digital-first delivery  

External assessment services that are digital first will be designed primarily for a digital device user experience for all the participants. In the transition period, or for exceptional circumstances where paper-based processes are still required, there will be options.  

In a world of diverse digital devices, digital first will mean some compromises between features available only for “high spec” devices and those that can be used by the majority. There are two important aspects to managing the potential downsides of this. First, usability and accessibility features that are built into the design of assessments early are more likely to deliver a high-quality experience for the majority. Second, with an adaptable digital assessment service changes in what constitutes “high spec” can be incorporated more regularly, so the bar continues to rise.

Digital first will mean that time and geographical constraints will cease to be relevant. The parts of the process that still require human input will be adapted to keep pace. This could mean more continuous access to examiners and markers throughout the school year, or could mean that some assessment question development and marking could be supported with or replaced by digital tools.

A digital first approach will entail marking digital responses online. While human judgement will remain critical in the near term, machine marking will have a role in the future.  Notwithstanding the importance of the quality assurance of marking, online marking will lead to students receiving results of external assessments within a much shorter period from undertaking the assessment, enabling them to move to their next learning step, or not.  

The pace of change in a digital first assessment approach will be dependent on the pace of change in how students are prepared for digital assessment, the availability of human capital and quality of the digital solutions.  

Digital first in external assessment will be complementary to the adoption of digital first approaches to internal assessment that are being increasingly used in schools.     

Data as an asset: Digital external assessment services enable the collection of a broad range of detailed and disaggregated analytics which can be used to better inform assessment development as well as teaching and learning

Digital assessment will generate significantly more data than is currently available. The intent is to enable access (with appropriate privacy and security controls) to data about the users’ characteristics (e.g. their demographics), their experience (e.g. in real time, or to analyse later, the time taken to complete specific questions), the operations and logistics of the examinations (who is taking an examination, when, and where they are up to), their results (e.g. compared to students with similar and different characteristics) at a minimum. 

This data will support teachers to reframe less successful teaching approaches, or continue to use approaches that are supporting student achievement, or recognise new patterns in student achievement. It will support students to self-diagnose their learning strengths and gaps. It will support more efficient quality assurance of examination development, marking and finalisation of the results for credentials.

The data will follow the student through and beyond their schooling to support lifelong learning. Students will be able transform some of this data into verified achievement information that is reflected in their digital portfolios and badges.

  



[1] Flipped classrooms are a form of blended learning where activities traditionally done in classrooms, e.g. acquiring new information, are undertaken at home, often online, and those traditionally set as homework, e.g. project work and deep inquiry, are undertaken in class.

See What we're reading for a selection of articles that have helped shape our thinking about the context for digital assessment.

 
Skip to main page content Accessibility page with list of access keys Home Page Site Map Contact Us newzealand.govt.nz