• The Purpose of this Resource
• Self-Assessment
• External Evaluation and Review
• Evaluation Tools
• Using Evaluation in Self-Assessment – an Overview
• Using Self-Assessment in Your Organisation
• Useful Evaluation References and Resources
• Appendix 1: Linking Key Evaluation Questions, Possible Evidence
• Appendix 2: Tertiary Evaluation Indicators – An Overview
• Appendix 3: Self-Assessment Plan of Enquiry
• Appendix 4: A checklist for Building Evaluation Capacity
• Appendix 5: Glossary

Using Self-Assessment in Your Organisation

The process below shows the key features of effective self-assessment. It is a guide only and offers a way to approach self-assessment systematically and from an organisational view.

This high-level approach to self-assessment can be tailored to suit individual TEOs and is represented in the figure below.

Figure 2: Stages of a Self-Assessment

Ultimately, self-assessment is about understanding how well your organisation is achieving its educational outcomes. You will be interested in asking questions that enable you to build up a picture of how well this is being done.

Step 1: Planning

Select an Appropriate Focus

You should develop an approach that systematically works through all aspects of your business over a defined period of time (at least every four years). Depending on the size and available resources, it may take several years to cover all aspects of your organisation.

It is recommended that you focus initially on a manageable number of important areas, to help you develop and refine your method and approach. You can broaden the scope progressively as you develop expertise and as resources allow. It is likely that you will already have some of the processes in place to gather data in the area(s) you have selected.

The area(s) of focus you choose should be of sufficient relevance or importance to learner outcomes to warrant the effort involved.

Starting with a narrow area of focus enables you to consider what data is particularly useful to you in undertaking the investigation and for your business and planning.

Examples of areas to begin with include:

• a particular learner group, e.g. international learners
• programmes and/or services where there have been complaints
• part or all of a specific programme
• an area that is strategically important to the organisation
• high priority or high-stakes programmes or processes
• important national or regional programmes
• programmes with large enrolments
• pilot programmes and new initiatives, especially those that may be expanded if successful
• areas or programmes that are experiencing problems
• language, literacy and numeracy initiatives.

The plan needs to be sufficiently flexible to enable you to adjust your priorities if new issues emerge or the original focus area is subsequently shown to have significant impacts across the organisation as a whole e.g. the adequacy of learner support.

'It’s all about knowing what your organisation is about and what the most important things are you are trying to do. Once that’s clear, your antennae become tuned and you can start to collect evidence in a whole wide variety of ways'

TEO staff member

Figure 3: Getting to the Essence of Self-Assessment

Take a Strategic View Across the Organisation

Your self-assessment activities should reflect the purpose and goals of the organisation, which will be based on the needs and aspirations of your learners and other interested parties.

They also need to make explicit the educational outcomes you are trying to achieve. This set of outcomes provides the basis for testing:

• how well you achieve your educational outcomes
• what factors impact on the achievement of these outcomes
• what other information is needed and what steps need to be taken to obtain this.

Use a Collaborative Approach

Effective self-assessment will engage a wide range of people at all stages of the process, to ensure a broad view across the organisation. That includes the initial planning phase.

Senior management need to be involved to drive change within the organisation. Teaching staff, support staff, learners, employers and communities should be involved too.

Small organisations will be able to involve all staff directly. Larger organisations may need to work through a steering group, and include self-assessment as part of a programme of work.

'You can’t be complacent. You need to be honest and look at your own situation really hard and see what you need to do to improve'

TEO staff member

Involving a wide range of people across the organisation not only helps to understand and address issues but also enhances commitment, ownership and 'buy-in'.

Allow Time

Be realistic about the time you will need for each area to be explored appropriately. It is important to anticipate the unexpected, such as a need for extra data or more investigation than initially planned.

Involving a wide range of staff also requires careful time management. For example, timetable issues can make it hard for staff from different parts of the organisation to meet.

Step 2: Gathering Data

What is known about the educational outcomes being achieved in the selected area(s)?

Having chosen an area for evaluation, you need to establish what it is you want to know about it. Appendix 1 provides key evaluation questions you could use to guide your enquiry. It shows how these can link with the relevant tertiary evaluation indicators and possible sources of evidence that you could use.

'Data may range from [the results of] recording a good conversation in the staff room to meetings with local employers – everything is valid if it helps you to provide evidence of how you meet your organisation’s own goals.'

TEO staff member

Choose questions that are relevant to the area you are looking at. For instance: 'How well do learners achieve?' or 'How well are learners guided and supported?' would apply if you were evaluating a programme or course, or could equally apply across the TEO as a whole.

It is also important to identify which outputs/outcomes are relevant here. You need to have a process that allows you to identify these. The tertiary evaluation indicators (Appendix 2) can help you do this.

Identify what qualitative and quantitative data (evidence) is already available to answer your chosen question(s). Identify any gaps and how you could fill them.

Appendix 3 provides a template for a 'Plan of Enquiry' that may be useful for this part of the exercise. Quantitative (numerical/statistical) or qualitative (descriptive) data can take many forms and can include:

• enrolments, learner completion and achievement
• moderation and monitoring reports
• notes from interviews, surveys and observations
• feedback from relevant groups − internal groups such as learners, teaching staff and management.

What is known about what causes or contributes to the achievement of the outcomes?

A mix of evidence will be needed to answer the KEQs confidently, preferably from at least two or three distinctly different sources.

Data may need additional analysis, to better understand what factors have contributed to the achievement of the outcomes.

Using enquiry questions can help you probe deeper to decide whether you need additional data or more in-depth investigation. You could use the enquiry questions listed in the tertiary evaluation indicators of your chosen KEQs. Alternatively, you may wish to develop your own lines of further enquiry.

How reliable and valid is the information used to answer the questions?

To draw reasonable conclusions about the achievement of educational outcomes your data needs to be:

• valid
• reliable
• complete
• sufficient
• up-to-date.

Ultimately your quantitative and qualitative data should be robust enough to enable reliable and consistent answers to the KEQs. Having quality data should also enable you to measure and monitor the progress made as a result of improvements over time.

How is the information benchmarked or referenced externally?

Where possible, your data should be benchmarked externally to enable you to compare your educational outcome achievement with providers of similar courses or with those teaching learners with similar attributes. The benchmarking information you chose to use will need to offer a valid comparison with your own data.

'The discussions have always happened but they are generally informal without a clear purpose and means of integrating them strategically into planning and making improvements'

TEO staff member

Benchmarking data may come from a variety of sources and, for TEC-funded providers, could include SDR (Single Data Return) and TEC benchmarking information where this is available.

What about compliance issues?

All TEOs need to be aware of any legislation or regulations that apply to them. This requires identifying relevant requirements and then regularly checking that the organisation continues to comply with them.
Although this is not an evaluation activity as such, these checks should be carried out as part of self-assessment and included in any reporting. Where non-compliance is found, action plans should include remedial work to be done.

Step 3: Analysing and Interpreting the Information

To establish your level of achievement of educational outcomes, you will need to analyse and interpret the data to determine patterns, trends, themes, linkages and relationships.

For example: Is there some pattern to the timing of learner withdrawal from a course? Does this pattern require further investigation? Is this pattern replicated in any other courses? Is it important enough to investigate further? Is there any information in the learner feedback that indicates why learners withdraw at this stage? If not, how can we get this information?

Table 2 below provides an example of where further analysis and interpretation would be important. It is an extract from a summary of pass rates by ethnicity, for a single qualification in a TEO. The unusual result highlighted for the Ethnicity D group needs to be understood. Further questioning would be aimed at uncovering the reasons for the decline in the pass rate in 2007.

Table 2: Learner Pass Rate Data by Ethnicity

How is the information and analysis used to bring about improved outcomes?

Evaluative conversations involving all levels of the organisation can be useful to interpret the information by asking questions such as:

• Is this result expected or unexpected?
• What are the probable reasons for this result and what evidence is there to
  support them?
• How important is the result and does it impact on other parts of the business?
• What else do we need to know to fully understand the significance of this result? Is it feasible to obtain this information?
• What information is available to suggest ways to improve this result?

The answers to these questions will create a picture about what is happening and why. It will then enable decisions to be made about whether more information might be needed, whether it is possible to find that, and what needs to be done to make improvements.

Step 4: Making Decisions

By this stage you should have a good understanding of how well you are achieving educational outcomes in the areas you have evaluated. You can now consider what actions you might want to take to improve performance, either in a particular area or more broadly.

Findings in a specific area may have implications for the organisation as a whole.

Good decisions will be justifiable and evidence-based.

Each TEO will need to be clear about where the responsibility for the various decisions lies.

Step 5: Reporting and Action

It is essential that your self-assessment process, its findings, and the actions taken as a result are documented and reported to senior managers, councils or business owners.

Reports need not be lengthy but must be useful. Most importantly each TEO should have a clear process for ensuring that self-assessment results are used to make real improvements.

This is important for your own internal management but also as evidence for any subsequent external evaluation and review. Such documentation will allow you to demonstrate to the evaluators, or any other internal or external groups, that you are using self-assessment successfully to make improvements.

If self-assessment is robust, and record-keeping meticulous, the external evaluation and review process can be relatively non-intrusive. Records may also include file notes, reports to Boards or owners, etc.

Step 6: Monitoring Improvements

Once you have decided what actions to take to bring about improvements, you will need a plan to help you implement these. This should also capture any opportunities for innovation you may have identified.
The plan should build on existing good practice, monitor whether the actions lead to improved educational outcomes, and, if so, integrate the actions into your overall business and strategic planning.