National External Moderation Summary Report for 2008, Semester 1

Prescription: 130/530 Organisation and Management

This report provides a national perspective on the moderation of 130/530 Organisation and Management.

Because prescription 130 and prescription 530 are substantially different, the two prescriptions are reported on separately.

It should be noted that prescription 130 will no longer be assessed after December 2008.

130 Organisation and Management

Introduction

Assessment materials from 17 Tertiary Education Organisations (TEOs) were moderated for this prescription.

Six submissions (35%) met the national standard.

Areas where the national standard was not met included the following, with the number of submissions concerned shown in parentheses:

incomplete assessment of learning outcomes (9)
learning outcomes not assessed at the appropriate level as specified by the prescription (6)
prescription weightings not adhered to (6)
lack of clear and appropriate assessment conditions and instructions (6)
marking schedules not allowing for a range of responses (3)
marking schedules with insufficient detail to ensure fair and consistent marking in terms of the assessment materials (9)
marking schedules that are not consistent with the requirements of the prescription and assessment materials (5)
inconsistent marking of candidates' work according to the marking schedules (7).

Of the 11 submissions not meeting the national standard, six did not meet four or more different requirements.

Of the six submissions that met the national standard, five met all requirements.

Factors contributing to these results are outlined in the body of this report.

Presentation of assessment materials

Most submissions were well presented, bound, clearly divided and identified according to the checklist. However, 53% of the submissions were incomplete. Missing items included assessments, marking schedules, and student scripts.

Some submissions included material not listed in the moderation submission checklist (such as additional student scripts or internal moderation forms). These additional items are not required for national external moderation.

Assessment grids

All TEOs, except two, submitted assessment grids. These grids, however, varied widely from simple indications of weightings for each topic to more detailed grids that matched specific questions and marks to the weightings for each learning outcome and assessment criteria. More detailed grids are not only helpful to moderators when trying to assess whether the required weightings are actually achieved, but are also an important assessment design tool.

In some cases the assessment grids did not match the actual weightings in the assessment materials. In a few cases, the weightings given in the course outlines did not match the weightings on the assessment instruments.

Learning outcomes

There are 12 learning outcomes and 59 assessment criteria in prescription 130. This meant that meeting the requirement for all learning outcomes to be assessed was problematic. To manage this issue, the moderator team defined the assessment criteria for prescription 130 as either "key" or "non key". Of the 59 assessment criteria, 38 were deemed "key" and 21 "non key".

The moderators initially agreed that meeting the learning outcome requirement would require that no more than four key criteria should be missed (none of which could be 2.1 management theory or 9.5 appropriate leadership for a given situation), and no more than one key criteria should be missed for any one learning outcome. After further discussion, the moderators agreed that assessing 80% of the 38 assessment criteria would allow for acceptable assessment coverage of the prescription.

Level of assessments

Some individual assessments were at too low a level. These assessments required recall and comprehension only and could often be answered by direct reference to a text. It was evident from marking schedules that these lower-level answers were deemed acceptable by some assessors. There was widespread use of multi-choice (even in non-test assignments), true/false questions and matching questions. Whilst the use of multi-choice questions may legitimately serve the purpose of eliciting basic knowledge, the prescription requires that, in general, assessment should focus on application and problem-solving. In some instances, the prescription is specific - for example, criterion 6.3 requires students to be able to "write SMART objectives suitable for an operational plan". Asking students to define SMART objectives does not appropriately assess this requirement.

Prescription weightings

Generally, any identified deviations from the prescription weightings were relatively small, and not meeting this requirement alone did not result in the national standard not being met overall. In a few cases, topics assessed were outside prescription requirements.

Assessment conditions and instructions for students

Clarity and appropriateness of assessment conditions and instructions for students was a concern. Common problems in the six submissions not meeting this requirement were:

different marks and weightings in the course outline compared to the individual assessment instruments
different marks on the coversheets of assessments to the marks allotted within the assessment instrument
general questions or questions with a large mark allocation with no breakdown in the assessment instrument of how those marks were to be awarded
poorly-worded questions where the scope/content of the answers required was not clear
an introductory statement or prologue provided to students and a following question that did not relate to the opening statement, which was confusing.

Marking schedules

Quality of marking schedules was another area of concern. The main problems were:

lack of detail in marking schedules and the resulting difficulty in determining whether work had been marked consistently
too much detail and/or prescriptive marking schedules, which did not allow for "a range of appropriate student responses"
marks given on the assessment instrument but no marks detailed in the marking schedule
insufficient breakdown of marks in the marking schedule. An example included a 50 mark question with no breakdown of how those marks are to be awarded and a question in parts, with a total mark but no breakdown of marks for each part
validity of some answers. An example was a true/false question which stated "It may be necessary in the interests of economic performance to behave less than ethically". The emphatic answer supplied was "true".

Student work samples

Student work was checked for consistency with the marking schedule. If the schedule was significantly flawed, as noted above, it was difficult to make an informed judgment on marking consistency. In a few cases, full marks were awarded for what were clearly incorrect answers.

Conclusions

The major problems identified in submissions were in relation to the learning outcomes being assessed, assessment being at an appropriate level and the validity of marking schedules.

With regard to the learning outcomes and the level of assessment, some mitigating factors have been noted in the body of this report. With regard to the marking schedules, more attention to format and content is required so that assessment can be fair, valid, and consistent.

530 Organisation and Management

Introduction

Assessment materials from 16 TEOs were moderated for this prescription.

Ten submissions (63%) met the national standard. This was a higher percentage than for prescription 130.

Areas where the national standard was not met included the following, with the number of submissions concerned shown in parentheses:

incomplete assessment of learning outcomes (3)
learning outcomes not assessed at the appropriate level as specified by the prescription (3)
prescription weightings not adhered to (6)
lack of clear and appropriate assessment conditions and instructions (5)
marking schedules not allowing for a range of responses (3)
marking schedules with insufficient detail to ensure fair and consistent marking in terms of the assessment materials (4)
inconsistent marking of candidates work according to the marking schedules (4).

Of the six submissions not meeting the national standard, four did not meet four or more different requirements.

Of the 10 submissions that met the national standard overall, eight met all requirements.

Presentation of assessment materials

Most submissions were well presented, bound, clearly divided and identified according to the checklist. Twelve submissions (out of 16) were complete. This was a much improved rate of compliance than that achieved with the prescription 130 submissions.

Assessment grids

All TEOs submitted assessment grids. These grids, however, varied widely from simple indications of weightings for each topic to more detailed grids that matched specific questions and marks to the weightings for each learning outcome and assessment criteria. More detailed grids are not only helpful to moderators when trying to assess whether the required weightings are actually achieved but are also an important assessment design tool.

Learning outcomes

Thirteen submissions (out of 16) met the requirement that all learning outcomes must be assessed. This was a much improved rate of compliance than that achieved with the prescription 130 submissions. However, in the view of the Lead Moderator, some problems associated with prescription 130 (i.e. number of learning outcomes and related assessment requirements) also apply to prescription 530. Prescription 530 has nine learning outcomes with key elements under each. NZQA's stated intention in the development of prescriptions 400-600 was that all learning outcomes should be assessed and all key elements should be evidenced. This is implicit rather than explicit in the prescriptions, which state "All learning outcomes should be evidenced".

In addition, there is a choice in what is assessed for some key elements (i.e. with the elements under learning outcomes one, two, and eight) and no choice in the others. Lack of choice can lead to the same ‘prescriptive' problems inherent in prescription 130, which produce a mechanistic coverage of individual criteria at the expense of more in-depth assessment. This matter will be considered at the time of the review of prescription 530.

The moderation team managed this situation by balancing omission of an occasional key element with evidence of appropriate depth and breadth of assessment across the prescription as a whole.

Level of assessments

Thirteen submissions (out of 16) met this requirement. This was a better rate of compliance than for prescription 130 submissions.

It was not clear that all assessors had noted the change in instruction relating to assessment levels between prescription 130 and prescription 530. Prescription 130 contains an assessment statement that "in general, assessment will be of the ability of students to apply the learning outcomes and to undertake problem-solving within the course material". Prescription 530 does not contain this instruction. In prescription 530, application and problem-solving are specifically instructed in some learning outcomes by the use of assessment verbs but not in others. Where these types of verbs are omitted, the use of recall and comprehension is legitimised. This is the case for learning outcomes one, two, four, and nine.

The learning outcomes in prescription 530 that clearly specify application and problem-solving are:

learning outcome three: "students will apply their understanding ..."
learning outcome five: "students will ... apply a range of techniques ..."
learning outcome six: "students will ... apply processes to a given situation"
learning outcome seven: "students will discuss two organisational structures, evaluate and recommend a structure for a given situation"
learning outcome eight: "students will ... apply knowledge of leadership, motivation and delegation".

Although some materials had a general focus on application and problem-solving, they did not fully meet the above specific requirements. This issue could result in assessments being determined to be not at the appropriate level.

Prescription weightings

Prescription 530 allows for a 10% aggregate variance against the prescribed weightings. Twelve submissions (out of 16) met this requirement. Generally, any identified variance from the prescription weightings was relatively small, and not meeting this requirement alone did not result in the national standard not being met overall. In a few cases, topics assessed were outside the prescription requirements.

Assessment conditions and instructions for students

Clarity and appropriateness of assessment conditions and instructions for students was a concern. Common problems in the five submissions not meeting this requirement were:

general questions or questions with a large mark allocation with no breakdown in the assessment instrument of how those marks were to be awarded
questions worded in a way that did not make the scope/content of the required answers clear.

Marking schedules

Quality of marking schedules was another area of concern. The main issues identified were:

lack of detail in marking schedules and the resulting difficulty in determining whether work had been marked consistently - for example, "the tutor to use discretion when allocating marks"
too much detail and/or prescriptive marking schedules, which did not allow for "a range of appropriate student responses"
marks given on the assessment instrument but no marks detailed in the marking schedule
insufficient breakdown of marks in the marking schedule
inconsistent allocation of marks in the assessment instrument and in the marking schedule, for example, in one submission there were 12 questions where the marks allocated did not match.

Student work samples

Student work was checked for consistency with the marking schedule. If the schedule was significantly flawed, as noted above, it was difficult to make an informed judgment on marking consistency. In some cases, full marks were awarded for incorrect answers.

Conclusions

The major issues identified were in relation to l ack of clear and appropriate assessment conditions and instructions, the validity and reliability of marking schedules, the assessment of all learning outcomes, and assessment being at an appropriate level.

However, there was a significant overall improvement in the outcomes of this moderation compared to the outcomes for prescription 130. Some excellent submissions were received. Much of this improvement can be attributed to the structure of the new prescription.

Page updated: 27 January 2009