Committee of Associate Deans (Teaching) (CADeT) Working Party on Subject
Evaluation Report
Introduction
This report is the result of a decision by the Committee of Associate
Deans (Teaching) (18/10/97) to establish a working party “to explore the
possibility of initiating a University-wide evaluation scheme . .
define elements of the evaluation strategy and explore how such a
program might be implemented.” The
full terms of reference, composition of the working party, meeting details
and supporting documentation are provided in the Appendix.
A brief history of Subject Evaluation at Monash
The development of a
‘university policy’ on the evaluation and reviewing of subjects dates
from the lead-up to the first Quality Assurance round in 1993.
One of the agenda items proposed for that first round was to
determine what procedures a university had in place to evaluate its subject
offerings. Monash realised that
in a central, university wide sense it had none; the evaluation of subjects
had until then been a faculty responsibility.
The eventual outcome was the
development of the University’s Education Policy, which contained a
section 9.1 Review of Subjects.
This section specified that student questionnaire responses should be
used as one of the elements of a process of continuous evaluation.
The questionnaires were to be designed by departments, but were to
contain a common core of 10 items, which were devised initially by the
Committee of Deans under then DVC Chubb as chair.
Responsibility for the implementation of the Subject Evaluation
questionnaires was to reside with Deans and Associate Deans (Teaching)
(Academic Board 2/92, Item 3.1). The
overall review process was to have subject coordinators preparing reports
for their Heads, those Heads in turn reporting to Deans, and Deans
incorporating this review material in their annual reports to Academic
Board.
The Teaching Evaluation Unit
within the then Professional Development Centre (now CHED) was responsible
for the processing of Subject Evaluation questionnaires.
As a result of administering this process, it became clear that the
10 core questions suffered from two basic difficulties.
First, many of the questions were global, allowing considerable
variation amongst responding students as to just what they thought the
questions were asking. Second,
the questions used an undefined response scale that left it to students to
decide individually on the specific criteria they might apply.
In response to these
difficulties it was proposed in March 1996 that the Deans and Associate
Deans (Teaching) review and revise the 10 core questions (see Subject
Evaluations ... Core Questions Document A in the Appendix).
This proposal was accepted and broadened by the Committee of
Associate Deans (Teaching) (meeting 2/96).
The review considered the identification of common fields of concern,
which might result in core ‘areas for questioning’ as an alternative to
the mandatory questions approach. The
review also considered methods of subject evaluation other than student
questionnaires, with a view to complementing questionnaire data and thus
providing a more rounded picture of the operation of subjects.
The first stage of the review of Subject
Evaluation was to analyse the University’s Education Policy.
The outcome of this analysis was a listing of areas that seemed
particularly relevant. This analysis was prepared as a report to the
Committee of Associate Deans (Teaching), July 1996.
The second stage of the review was to discuss
with Associate Deans (Teaching) individually their views on what might be a
set of ‘core issues’ around which subject evaluation procedures might be
designed. The Associate Deans
were to act as the liaisons for their individual faculties.
The results of these interview discussions were prepared as a further
report to the Committee of Associate Deans (Teaching), early September 1996.
The
Education Policy analysis and the inputs from ADs(T) were used to produce a
report and proposal in September 1996 on Evaluating
and Reviewing Subjects (Document B in the Appendix).
This proposal suggested that subjects should be periodically
evaluated and reviewed, so as to address six broad issue areas.
These areas were: content and objectives, teaching methods,
assessment procedures, students’ study behaviours in response to the
subject, evaluation mechanisms, and expertise of the teaching staff.
The proposal discussed each area in detail and identified a range of
possible information sources.
The
proposal argued that the procedures to be employed in addressing the six
broad issue areas should not be centrally mandated by the University.
Rather, it was suggested that specific data gathering procedures
should be devised as close to a subject as possible.
The role of the University would be to accredit the procedures
proposed for each subject evaluation, and to assure that subject evaluations
were carried out.
Finally,
the proposal suggested that the reporting procedures associated with subject
evaluations should include a requirement that development implications be
indicated, together with a timetable for their implementation.
The proposal was presented to the Education Committee (meeting 7/96,
November 6, Item 8.1). The
following extract from the meeting’s minutes summarises the Committee’s
decision (see Document C in the Appendix for the full extract).
“The Committee . . concluded that ..
Faculties take this paper as a basis for the development of their own review
procedures, taking care to address in their review plan the six identified
core areas . .
These procedures should then be presented
by each Faculty for the information of the Education Committee . . .
A timeline for each Faculty’s review of subjects should also be
developed and presented. After
receipt of the processes developed by each Faculty, an overarching policy
statement could be developed, to be incorporated into the Education Policy,
to update the section of that policy referring to subject review . . . ”
The extent to which this Education Committee decision has actually been
implemented varies across areas of the University.
It appears that some areas are still operating the original ‘ten
questions’ system, some are attempting to implement the Education
Committee approach (eg the Arts Faculty, Document D in the Appendix), while
others have no system for subject evaluation.
It would appear that the major problem for subject evaluation has not
been in devising an acceptable scheme, but rather in ensuring that the
scheme is implemented.
The Purpose of Subject Evaluation
The three main purposes which subject evaluations have served are as
follows:
(i)
quality improvement - to provide information that will enable
subjects to be improved;
(ii)
external quality assurance - to provide a mechanism that indicates
subjects are being monitored and reviewed;
(iii)
promotion - to provide a means for staff to document a case for
promotion.
It is the Working Party’s view that a process for subject evaluation
should have quality improvement as its primary purpose. If the process is
working, then it should be possible to demonstrate quality assurance.
In the past there has perhaps been too strong a reliance on quality
assurance (for example demonstrating that student evaluations of subjects
have been taken) with little systematic evidence that such evaluations have
been used for the purpose of improvement.
It is also the Working Party’s view that while staff should not be
prevented from using subject evaluations in a portfolio of evidence
presented for promotion, this is not the purpose of subject evaluation and
should not influence the development of subject evaluation.
Other mechanisms which allow for interpretation of results against
University and Faculty normative data (ie MonQueST) are available for this
purpose.
In summary, a scheme for subject evaluation should be developed to help
teachers improve their subjects and it is important that improvement
processes can be demonstrated, not least, to students.
Quality Improvement
Subject evaluation can be seen as an integral and normal process whereby
teachers seek information on whether their teaching is having the effect
they desire. With many more
students and far greater diversity, such information is increasingly
important. The strategic directions outlined in the Monash Plan and the
Learning and Teaching Operational Plan also have consequences for subjects.
For example, teaching for understanding (rather than reproduction),
the development of life long learning skills, engagement in community,
workplace and professional contexts, innovation and internationalisation,
are all considered important for Monash subjects.
So too is the movement toward student-centred flexible learning and
the use of technology for good educational purposes.
Subjects are the vital ‘building blocks’ of the curriculum; they are
the teaching unit with which teachers identify and an important aspect in
resource allocation. It is
therefore important that a coherent framework for subject evaluation be
developed which will assist teachers to monitor and improve their courses in
light of the changing teaching and learning environment.
A Proposed Scheme for Subject evaluation and Review
Data sources and methods
There are four major data sources for subject evaluation: students,
academic peers, educational advisers and the teacher or teachers of the
subject through self-review. Each
source has particular strengths and weaknesses.
These are briefly outlined together with an indication of possible
ways of obtaining information.
Students (St)
Students are best placed to
comment on the classroom experience. They
are less well placed to comment on the level of the content material and the
academic integrity of the subject. Both
present and past student may be asked to comment and the most common methods
for obtaining comment are by questionnaires, focus group interviews and
class representative panels.
Academic Peers (AP)
Academic peers are well
placed to comment on the level and academic integrity of the subject.
Unless specifically trained, they are less well placed to comment on
the classroom experience. Academic
peers from similar areas within the University, other institutions or
professional bodies, are normally asked to comment on subject documentation
and materials.
Educational Advisers (EA)
Educational advisers (eg CHED)
are well placed to comment on educational design and educational processes
but are not well placed to comment on disciplinary or content knowledge.
They are normally asked to comment on subject documentation and
materials but may also be asked
to give formative feedback on classroom processes.
Self Review (SR)
Teachers are well placed to
evaluate many aspects of the teaching of a subject through systematic
self-review. However for a more
complete picture other sources are also necessary. Systematic self-review
through the keeping of a teaching journal, notebook or other recording
device can be used for most aspects of teaching.
Possible data sources are
identified in the following sections by the labels: St, AP, EA, SR. Self-review
is included in all categories but would normally need to be augmented by
other sources.
Key areas for subject Evaluation
Following the scheme
previously endorsed by the Education Committee, it is recommended that
subject evaluation be structured around the six key areas of:
These key areas have been
‘redeveloped’ by the working party and are elaborated below, together
with data sources.
1. Content and objectives
This section includes a
general consideration of the learning outcomes that a subject hopes to
achieve in its students and the appropriateness of the curricular objectives
and content in developing these outcomes.
1.1
Does the curricular content of the subject have academic
integrity and coherence; is it at an appropriate level in the course or
program; does it meet relevant professional requirements?
(AP, SR)
1.2 Does
the subject clearly complement rather than overlap or duplicate others; does
it build on previous subjects; does it provide a basis for related subjects?
(St, AP, SR)
1.3
Do the various components that make up the subject integrate with or
reinforce one another? (St,
AP, EA, SR)
1.4
Are the subject’s learning objectives clear; are the
objectives described in student performance terms?
(St, AP, EA, SR)
1.5
Are full descriptions of the subject’s curricular content
and its learning objectives provided to students; do students properly
understand the descriptions? (St,
SR)
1.6
Does the subject include major curricular objectives identified in
the Monash Plan and Learning and Teaching Operational Plan?
(AP, EA, SR)
2.
Teaching methods
This section seeks to ensure
that teaching methods are selected to best support the learning processes
through which students will achieve the designated learning outcomes
identified in the objectives.
2.1 Are the
teaching methods used consistent with the learning processes which are
necessary to meet the learning objectives; have alternative approaches been
considered? (AP,
EA, SR)
2.2
Is there a generally supportive climate in the subject; do the
subject’s teaching staff exhibit an enthusiasm for the subject and its
teaching; do the students feel comfortable in contributing, questioning and
commenting? (St,
SR)
2.3 Has the
rationale behind the use of teaching methods been explained to students?
(St, SR)
3. Assessment procedures
This section seeks to ensure
that assessment tasks provide direct evidence of students’ progress
towards attainment of the learning objectives. It includes the development
and systematic use of marking and grading procedures and the criteria upon
which these are based (summative assessment).
It also includes the provision of feedback during the learning
process in order to guide and support learning (formative assessment).
There is a need to clearly differentiate between the summative and
formative aspects of assessment.
3.1
Do the assessment tasks clearly relate to the learning
objectives? (St, AP, EA, SR)
3.2
Does formative assessment comprise part of the assessment
strategy; is it clearly differentiated from summative assessment; is
feedback timely; does feedback provide clear indications of how improvements
can be made? (St, AP, EA, SR)
3.3
Have the criteria for assessment been adequately described;
are there descriptions of the performance required for each level of mark
awarded? (St, EA, SR)
3.4
Have the criteria for assessment and the processes for
submission, marking and appeal, been made clear to students?
(St, SR)
3.5
Are grade and mark distributions regularly compared with those of
similar subjects? (AP,
SR)
4. Student workload and study behaviour
This section concerns
attempts to ensure that workload is reasonable, reflects the primary aims of
the subject and takes account of student background.
The provision of support to students is also identified as a concern.
4.1 Has
information been gathered on workload for the subject; does workload
correspond to the weighting of the subject; is workload reasonably spread?
(St, AP, SR)
4.2 Does the
time spent by students reflect the emphases of the learning objectives,
teaching methods and assessment tasks? (St, AP, EA, SR)
4.3 Has
information been gathered on the background of students; their knowledge,
experience and reasons for taking the subject?
(AP, SR)
4.4
Are there procedures for monitoring student learning behaviour and
for providing individual advice, learning support and counselling?
(St, AP, SR)
5. Routine
evaluation
This section considers the
process of routine evaluation. This represents the ongoing and less formal
aspect of evaluation that forms part of normal teaching activity.
It may comprise one or two of the data sources discussed and may be
undertaken quite frequently. While
it differs from the multi-sourced and systematic evaluation of subjects
outlined in this document, ongoing information may usefully feed into formal
subject evaluation.
5.1
Are routine processes in place to gather student and other
feedback on the organisation and teaching of the subject?
(St, AP, SR)
5.2
How is feedback used to monitor and improve the subject?
(AP, SR)
5.3
How are students informed of the use made of feedback
information? (St, AP, SR)
6. Expertise of the teaching staff
This section is concerned
with ensuring that staff teaching the subject have appropriate subject
matter and teaching expertise. It is recognised that there may be variation
among Faculties in how they satisfy themselves that staff are properly
equipped to teach a subject.
6.1
Do the teaching staff have the appropriate level of subject
matter expertise and familiarity with the content? (AP, SR)
6.2
Do the teaching staff have appropriate expertise in the
teaching and assessment methods used? (AP,
EA, SR)
Review of Subjects
Collection of data from the various data sources will
require considerable effort. It
is the recommendation of the working party, therefore, that the level of
subject evaluation outlined should not be undertaken each year, but should
normally be undertaken every five years as part of an overall review
process.
It is recommended that subject evaluations feed into
a schedule of departmental, faculty or course review.
They should also run in conjunction with accreditation and other
external professional reviews. For
example, following a course review, a staged process of evaluations for each
subject in the course could be planned for the 5 years leading up to the
next course review. Such a
rolling process of review would minimise repetition and the over-burdening
of students in a particular course in a particular year.
It is essential to ensure that the information gained
in subject evaluations is used to improve
the subject. Subject evaluation
can easily become a compliance issue, rather than a mechanism for
improvement. It is therefore
recommended that demonstration of improvement as a consequence of subject
evaluation be written into the terms of reference for course (or similar)
reviews. The monitoring of
improvements made to subjects as they are evaluated could also be required
on a yearly basis, following up on the evaluation of subjects evaluated the
previous year. Responsibility
for this task would need to be assigned.
The establishment of a program of review along the
lines suggested will require considerable consultation between the central
administration and faculties. It
is possible that Education Committee could be the approval body for faculty
based programs of review. It is important that a central monitoring role be
defined in order to ensure implementation of the scheme proposed in this
document.
|
