New Monitoring Process: Comparison of internal and external assessment data for NCEA

16 March, 2008

This work is part of a suite of measures put in place by NZQA as part of its key responsibility to continually improve its understanding of school-based assessment.

The comprehensive package includes the appointment of 33 fulltime moderators, reporting Not Achieved results for internal assessment, the random sampling of internally assessed work, and the comparison of results for internal and external assessment data for NCEA.

NZQA is evaluating raw data gathered to compare the results of internal and external assessment at schools around the country. Work is in its preliminary stages; it is premature to draw conclusions about school assessment practices at this point.

Further research must be carried out when Not Achieved (N) results are available for internally assessed standards (from this year), as the absence of this data leaves the current analysis incomplete and potentially misleading.

NZQA Chief Executive Karen Poutasi said the research was not yet complete, and assumptions had yet to be tested with schools. No firm conclusions can be drawn on schools, or departments within schools, which appear in the data.

"NZQA needs to discuss this research with schools to ensure that it is accurate and to further investigate reasons for unexpected results. No formal report has yet been produced as it is too early to do this, and any conclusions drawn from the data about school assessment practices are at best speculative," said Dr Poutasi.

"This is part of NZQA's ongoing commitment to ensuring that internal assessment is not only robust and fair but seen to be so."

This work, involving raw data from 2006 NCEA achievement standard results, was initiated by NZQA to provide preliminary information that would assist in identifying departments and schools in which there were wider than average variations between internal and external results and to enable discussions about such outlier results with these schools. Under the statistical approach being trialled, approximately 5% of results have been categorised as outliers – these are results outside approximately two standard deviations of the mean.

Deputy Chief Executive, Qualifications, Bali Haque said the data represented a starting point for discussions with schools.

"It's important to note that the fact a given subject in a school has been statistically identified as an outlier does not of itself necessarily indicate an issue or problem. The statistical methodology used will always identify outliers – the fact these outliers exist is not the issue, it simply means that we need to discuss results with schools to examine the reasons.

"Where a subject is identified as out outlier it indicates tentatively that the difference between internal and external assessment results is greater than that found at the majority of other schools. It will prompt a dialogue with the school to explain why that school has an apparent discrepancy between internal and external assessment results, and to identify if an issue exists."

Only those students with at least one internal and at least one external result were included in the analysis; furthermore, only those schools with at least 30 such students were included.

External results were identified for all subjects in all schools. Based on these results and on demographic variables, a predicted internal assessment score was found. The actual internal results were identified by subject and school and compared to this predicted score. An average band for the relationship was established by finding the mean and standard deviation of the difference between the predicted and actual; those schools that were more than 1.96 standard deviations from the mean (i.e. 5% or very close to 5%) were identified as outliers.

On the basis of this raw data the total number of schools in this category was 102. This was made up of:

71 in 1 subject
22 in 2 subjects
7 in 3 subjects
2 in 4 subjects

"This raw data represents, at best, a starting point for discussions with a school. It should not be used in isolation to draw conclusions about the assessment practices within schools."

A further set of data was run using only the external assessment (i.e. no demographic variables). This is the first time the reseach has been conducted and NZQA will continue to refine the aproach taken.

There a range of reasons why data may appear in the outlier category:

The untested statistical approach.
Students being better prepared for the internals by teachers because teachers had greater knowledge of the internal assessment tasks.
Schools having different policies regarding the number of opportunities. allowed to successfully complete internal assessments.
Students have put more effort into the internals relative to the externals at some schools than others.
Teachers did not mark to the national standards for the internals.

Our next steps:

Refine statistical approach to include actual Not Achieved (N) statistics for internally assessed standards, to be introduced this year. This will reduce the risk of non-valid assumptions being made.
Identify "problem subjects" on national basis, if they exist.
Contact schools through our School Relationship Managers and discuss the assessment practices in the subjects for which they were outliers, and provide advice, guidance, and monitoring, if appropriate.
Collate reasons for the schools being outliers so that any trends can be identified, analysed and acted on.
Identify good assessment practices in schools and, where appropriate, suggest these methods to other schools.

For more information contact:

Dave Gillies
Communications - NZQA
DDI: 04-463-4240
Mobile : 0274-575783

Page updated: 17 March 2008