Policy and Guidelines for the Conduct of External Evaluation and Review

Stage Three: Reaching Judgements on Educational Performance and Capability in Self-Assessment

Once answers to the key evaluation questions in each focus area have been obtained, and the responses rated, the evaluation team progresses to synthesising ratings into organisational level judgements on educational performance and capability in self-assessment.

The following section identifies the three key steps and describes each sequentially. At each step, CORE, a tool for synthesis designed for external evaluation and review is used. A step-by-step guide to using the performance criteria rubrics is contained in Appendix Three.

An external evaluation and review team must be able to demonstrate how they reached their conclusions and cite the evidence used to support their judgements.

Steps to Reaching Organisational Level Judgements

  1. Rate educational performance and capability in self-assessment in each focus area.
  2. Rate educational performance and capability in self-assessment on each key evaluation question i.e. an organisational rating for the key evaluation question.
  3. Reach statements of confidence in educational performance and capability in self assessment at an organisational level.

The following are important points to consider in making ratings:

  • emerging findings and judgements should be tested by discussing and clarifying interpretations of the evidence with the TEO personnel most directly involved and knowledgeable about the focus area
  • evaluators should work collegially, meeting and discussing emerging conclusions, synthesising ratings and reaching statements of confidence as a team
  • indicative ratings may change as the answers to all key evaluation questions are synthesised into an overall rating. For example, the evidence might suggest high performance on a key evaluation question that is not particularly important in the context of the overall performance
  • evaluators should carefully distinguish between the rating levels and be prepared to explain and justify their conclusions to colleagues and the TEO on the basis of evidence and its analysis
  • judgements must demonstrate a consistent understanding and interpretation of evidence based on the tertiary evaluation indicators and the performance criteria set out in the relevant rubrics
  • the team should collectively reach all the major findings and judgements that will be included in the report
  • the meeting to conclude the on-site stage of external evaluation and review should be used to explain, test and, if necessary, modify emerging conclusions with the TEO. Any modifications are made on the basis of further analysis of substantive evidence or explanation. This is not a negotiation process.
  • clear records of judgements, reasons and evidence are maintained, these will form the basis for writing the report
  • the designated writer for the report should provide the draft report for comment by the tertiary education organisation before final confirmation.

CORE

CORE is an approach for synthesising judgements designed specifically for external evaluation and review. Although other synthesis techniques may be appropriate, using CORE in external evaluation and review provides a consistent and transparent basis for reaching judgements.

CORE (synthesis) can be used at any stage of external evaluation requiring judgements based on multiple dimensions, data sources or types of evidence. It will be used to synthesise:

  • answers from several KEQs to reach an overall judgement for a focus area
  • judgements across key evaluation questions for the organisation as a whole
  • judgements across all focus areas and from organisational level performance in answering key evaluation questions to produce statements of confidence in the educational performance or capability in self-assessment of the TEO.

Table 1: The CORE Approach

Concentration of ratingsIdentify the mode and median ratings on the range. The ratings provide the initial baseline ratings which may be modified through the subsequent CORE steps.
Outlier ratings

Identify any outlying levels of really low or high performance, and whether this would cause a shift in the rating.

High or very high levels of performance in some areas do not necessarily compensate for low levels of performance in others.

Decide the appropriate emphasis to give to the more extreme performance ratings and determine if there is sufficient reason to raise or lower the ‘concentration rating’ based on consideration of any extreme ratings.

Reflection

Reflect on whether the important findings have been given the appropriate weight. Determine if the ‘what is so?’ and ‘so what?’ questions have been answered. Consider if the emerging judgement is credible and reflects common sense.

Judgement, justification, impact – are evaluators clear about the judgement being reached? Is there sufficient evidence to reach the judgement and is the evaluative interpretation of the evidence plausible? and explainable? Why does this thing matter – what is the impact on outcomes? In context, how significant is this finding?

Consider whether there are:

  • any other reasons for adjusting the rating
  • alternative explanations for an unexpected or extreme rating.

The external evaluation team may review the earlier analysis and interpretation of information and/or adjust the final judgement. Differences in interpretations between evaluators and/or evaluators and the TEO can be discussed and alternative explanations explored.

Explanation

Identify the most critical and relevant findings that led to the judgement and which need to be reported.

The report should contain sufficient information to substantiate the judgements that have been made and explain why the findings are important.

Rating Educational Performance and Capability in Self-Assessment in Each Focus Area

The ratings for the key evaluation questions are synthesised into an overall judgement in the focus area using CORE. The descriptors for the ratings of educational performance in focus areas provided below in Rubric 2.

Rubric 2: Criteria for Rating Educational Performance in Focus Areas

PERFORMANCE CRITERIA FOR FOCUS AREAS

Excellent

  • Highly effective focus area in terms of both contributing processes and outcomes1.
  • Comprehensively meets the needs of learners/stakeholders.
  • Any gaps or weaknesses are not significant and are managed effectively.
Good
  • Effective processes and outcomes
  • Generally meets the needs of learners/stakeholders.
  • No significant gaps or weaknesses, and less significant gaps or weaknesses are mostly managed effectively.

Adequate

  • Inconsistently meets needs of learners/stakeholders i.e. meets some needs and not others.
  • Some gaps or weaknesses that are being addressed effectively.
  • Meets minimum expectations/requirements2 as far as can be determined.
Poor
  • Focus area is ineffective, or has serious weaknesses.
  • Weaknesses not being addressed effectively or still require significant improvement to meet minimum expectations/requirements.

Rating Capability in Self-Assessment in a Focus Area

Once a judgement about educational performance in a focus area has been reached, the following question is asked and the response rated using the criteria in Rubric 3:

In this focus area, how well is self-assessment information used to understand educational performance and bring about improvements?

Rubric 3: Criteria for Capability in Self-Assessment in Focus Areas and Across KEQs

PERFORMANCE CRITERIA FOR CAPABILITY IN SELF-ASSESSMENT

Excellent
  • Clear evidence of effective self-assessment3 used to gain a comprehensive (full) understanding of educational performance and bring about wide-ranging, worthwhile improvements.
  • Any gaps or weaknesses in self-assessment are not significant and are being addressed effectively.
  • In this context, self-assessment is clearly part of a coherent and comprehensive approach across the TEO.
Good
  • Good evidence of effective self-assessment used to understand educational performance and bring about many worthwhile improvements.
  • No significant gaps or weaknesses in self-assessment, and less significant gaps or weaknesses are being addressed effectively.
  • In this context, self-assessment is part of a reasonably coherent and comprehensive approach across the TEO.
Adequate
  • Self-assessment is of inconsistent quality and/or used inconsistently to understand educational performance and bring about improvements. Some improvements may be occurring but somewhat haphazard.
  • Some significant gaps or weaknesses in self-assessment but being addressed. Not yet clear if improvements are sufficiently effective.
  • In this context, self-assessment is not part of a coherent and comprehensive approach across the TEO.
Poor
  • Self-assessment is ineffective, or has serious weaknesses.
  • Weaknesses not being addressed effectively or still require significant improvement to meet minimum expectations.

Organisational Rating

Rating Performance on Key Evaluation Questions Across the Organisation

The ratings for the key evaluation questions in individual focus areas are synthesised into an overall rating, across all focus areas, sampled across the organisation (using CORE). For example, if there are four focus areas, the four individual ratings for KEQ 1 are synthesised into an overall rating for that question, repeated for KEQ 2, and so on for all six key evaluation questions. The descriptors for the ratings of performance in individual key evaluation questions are provided in Rubric 1.

Rating Capability in Self-Assessment in a Focus Area

Rubric 3 is also used to rate capability in self-assessment based on the key evaluation questions in all focus areas across the organisation. This is done by answering the methodological question:

In the context of the key evaluation question across the TEO, how well is self-assessment information used to understand educational performance and bring about improvements?

Synthesising the answers and ratings across the sample of focus areas is an important stage. It means that evaluators can discuss transparently the evidence and reasoning leading to the judgements with both the evaluation team and the TEO.

In some circumstances, during the process of synthesising performance it may become apparent that other areas of enquiry, beyond the formal scope, are needed to be made to confirm emerging findings. For example, if the rating on one key evaluation question across four focus areas is variable, evaluators might want to undertake some straightforward further enquiry into one or two other areas of the TEO’s operation to assist them reach a final judgement.

Reaching Statements of Confidence on Organisational Educational Performance and Capability on Self-Assessment

Statements of Confidence

The organisational level judgements reached through external evaluation and review are expressed as statements of confidence.

It is not possible, nor defensible, to make a judgement about the overall performance of the organisation based on the sample of focus areas. This is because the sample is unlikely to be sufficiently representative or statistically significant. However, the sample is sufficient to support the conclusion about how confident the evaluation team is in the TEO’s performance.

Each statement of confidence is derived from two lines of synthesis. One synthesises judgements across the overall ratings in the focus areas. The other synthesises the judgements for organisational ratings across key evaluation questions. One line of synthesis is used to confirm or question the other as a means of ensuring maximum validity of the results.

Once an organisational statement of confidence is reached, the evaluation team apply the reflection and explanation steps of CORE to ensure the level of confidence is credible and can be robustly explained.

Reaching a Statement of Confidence in Educational Performance

The overall statement of confidence in educational performance is derived from two lines of synthesis (using CORE and rubric 4):

  • ratings across the key evaluation questions at organisational level
  • ratings across the educational performance in focus areas.

The intention is that each line of synthesis reinforces, or calls into question, the conclusions reached from the other. Evaluators confirm or adjust the rating in light of the second rating to reach a final statement of confidence about educational performance in the TEO.

Rubric 4: Criteria for Judgements About Organisational-level Educational Performance

LEVEL OF CONFIDENCE

DESCRIPTION OF EVIDENCE ABOUT EDUCATIONAL PERFORMANCE AT AN ORGANISATIONAL LEVEL

Highly Confident

ALL of the following:

  • Clear and comprehensive evidence that the organisation is meeting the most important needs of learners and other key stakeholders.
  • Clear evidence of effective processes that contribute to learning and other important outcomes.
  • No significant gaps or weaknesses.

Confident

ALL of the following:

  • Good evidence that the organisation is meeting many of the most important needs of learners and other key stakeholders.
  • Good evidence of adequate quality in the processes that contribute to learning and other important outcomes.
  • The body of evidence may not be comprehensive and/or clear enough OR the magnitude or range of outcomes may not be sufficient to justify a rating of highly confident.
  • Areas of weakness are not serious and are effectively managed.

Not Yet Confident

ALL of the following:

  • At least some evidence that the organisation is meeting the most critical needs of learners and other key stakeholders.
  • At least some evidence of adequate quality in the processes that contribute to learning and other important outcomes.
  • EITHER, evidence of important outcomes and/or quality contributing processes is too patchy to justify a rating of confident OR evidence is sound but shows several important (but not critical) gaps or weaknesses.
  • Adequate plans are in place or are being actively developed to address areas of weakness but needed improvements in outcomes, outputs and/or their contributing processes are not yet fully apparent.

Not Confident

ANY ONE OR MORE of the following:

  • Insufficient evidence that the organisation is meeting the most important needs of its learners.
  • Evidence shows that some important needs are not being met to an acceptable level.
  • Plans to address gaps and weaknesses in educational programming, delivery or outcomes are insufficient, nonexistent or not being given high enough priority.

Reach a Statement of Confidence in TEO Capability in Self-Assessment

To reach a statement of confidence in TEO capability in self-assessment, two ratings are taken into account. The first is the ratings in the focus areas and the second is the organisation ratings on the key evaluation questions across the focus areas.

CORE is used to synthesise the ratings for capability in self-assessment on firstly the focus areas and then Rubric 5 is used to reach a statement of confidence.

Next, a statement of confidence is reached by synthesising the capability in self-assessment ratings on the key evaluation questions across the focus areas using Rubric 5.

The primary rating is confirmed or adjusted in light of the secondary rating to reach a final statement of confidence about educational performance in the TEO.

Rubric 5: Criteria for Judgements About TEO-level Capability in Self-Assessment

LEVEL OF CONFIDENCEDESCRIPTION OF EVIDENCE ABOUT ORGANISATIONAL CAPABILITY IN SELF-ASSESSMENT

Highly Confident

  • The organisation has effective self-assessment that evaluates all of its high priority programmes and activities on an ongoing basis, along with periodic reviews of other focus areas.
  • The quality and validity of the self-assessment information reviewed as part of the EER is consistently high.
  • Findings are used insightfully to make comprehensive improvements.

Confident

  • The organisation has a wide-ranging self-assessment system in place that evaluates the majority of its high priority programmes and activities on an ongoing basis, along with periodic reviews of other focus areas.
  • The quality and validity of the self-assessment information reviewed as part of the EER is generally good, although there may be some important areas where quality and validity should be strengthened in order to help justify a highly confident rating.
  • Self-assessment is purposeful and appears to be largely genuine. Findings are used to make useful improvements.

Not Yet Confident

  • The organisation has a self-assessment system in place that evaluates a reasonable proportion of its high priority programmes and activities.
  • The quality and validity of most of the self-assessment information reviewed as part of the EER is generally acceptable to good, although some important areas need to be strengthened in order to help justify a confident rating.
  • There is evidence of at least some genuine effort to use findings to make improvements.

Not Confident

  • The self-assessment system is narrow and/or covers too few of the organisation’s high priority programmes and activities; or
  • Coverage and prioritisation may be adequate but the validity or utility of evidence or conclusions are too weak to usefully inform decisions or improvements. There are critical weaknesses evident in the TEO’s capability in self-assessment.

 1 Refer tertiary evaluation indicators and the characteristics of effective self-assessment.

 2 For example, many professional or registration bodies such as those for accountancy, nursing, medical radiation technology, and social work, have expected levels of performance or professional standards that graduates are required to meet.

 3 Refer tertiary evaluation indicators and the characteristics of effective self-assessment.

 
 
Skip to main page content Accessibility page with list of access keys Home Page Site Map Contact Us newzealand.govt.nz